7

I am trying to load a queue of interval processes. In other words I have a queue and I want each item in the queue to run on an individual interval.

My problem is that I can't seem to get more than 25 threads to run at one time. I'm using .Net 4.5 on a 64bit machine which has a default max thread count of 32768.

How do I make my app run as many concurrent threads as my machine can handle?

Here is an example app that replicates the actual problem in my production code:

class Program { static void Main(string[] args) { System.Threading.ThreadPool.SetMaxThreads(200, 200); test t = new test(); t.LoadUrls("http://www.google.com"); while (1 == 1) { System.Threading.Thread.Sleep(1000);//refresh every 5 seconds Console.WriteLine(System.Diagnostics.Process.GetCurrentProcess().Threads.Count); } } public class test { public void LoadUrls(string url) { for (int i = 0; i < 100; i++) { System.Threading.Timer t = new System.Threading.Timer(new System.Threading.TimerCallback(RunInterval), url, 0, 1000); Console.WriteLine("Loaded {0} feeds.", i); } } private static void RunInterval(object state) { string url = state as string; string data = ""; using (System.Net.WebClient cl = new System.Net.WebClient()) { Console.WriteLine("getting data for " + url); data = cl.DownloadString(url); } //do something with the data } } }

This code should theoretically run 198 threads after 2 seconds or so.

By the way, this worked beautifully in my prototype app; it was written in node. But, now I can't get it to work correctly in c#...

ANSWER: The problem was actually with garbage collection and wasn't a threadpool issue at all; the pool is more than capable of spooling all the threads I'm throwing at it. The trick is to use the single parameter constructor of the System.Threading.Timer; this will make the timer use itself as a semaphore, thus avoiding gc.

class Program { static void Main(string[] args) { for (int i = 0; i < 100; i++) { test t = new test(); t.url = "http://www.google.com?" + i; System.Threading.Timer ti = new System.Threading.Timer(new System.Threading.TimerCallback(t.RunInterval)); ti.Change(0, 1000); } while (1 == 1) System.Threading.Thread.Sleep(int.MaxValue); } public class test { public string url { get; set; } public void RunInterval(object state) { Console.WriteLine("getting data for " + this.url); string data = ""; using (System.Net.WebClient cl = new System.Net.WebClient()) { data = cl.DownloadString(this.url); } } } }

I'm not for sure why you would ever want a timer to be collected by the gc, but hey what do I know.

Improve this question
3
  • 1
    Why do you want so many threads running at the same time? Commented Jan 10, 2013 at 19:10
  • This is for a polling service. It has to poll n feeds at specified intervals. I could make a fancy queue system and run n individual services that only poll 1 feed per service. But, I would like to make this meta; I would like the program to start each individual process. Commented Jan 10, 2013 at 19:48
  • Your answer is still wrong. Everything may be garbage-collected, nothing special to exclude timers. You should keep a reference to all the timers and then dispose of them when done. Otherwise, they may still be disposed of when garbage-collected. Commented Oct 20, 2021 at 23:34

3 Answers 3

Reset to default
5

How do I make my app run as many concurrent threads as my machine can handle?

That's the wrong approach. Every thread is costly, create a few hundred and your system will seriously degrade.

Your code uses the ThreadPool. The pool has an algorithm to limit the number of threads. When you increase the Sleep() times you might see a few more threads.

A more direct way would be to set ThreadPool.MinThreads. But expect less performance, not more.

Improve this answer
Sign up to request clarification or add additional context in comments.

12 Comments

It isn't that costly... I have 64GB of ram to work with, but it never gets over 10MB. I want it to use all 64GB. What would your suggestion be?
Also, it never uses more than 1% of my available CPU.
I've added a few suggestions but it is still the wrong approach.
Curious how you measure memory, every thread should use 1 MB. And the Sleep() and I/O calls don't use CPU so that's why you see 1%. A good indicator that more threads won't help.
What would be the correct approach? I basically want to load a queue to retrieve data from n rest services. Each service has it's own refresh limits, so they all have to have individual timers.
|
3

According to System.Threading.Timer

Use a TimerCallback delegate to specify the method you want the Timer to execute. The timer delegate is specified when the timer is constructed, and cannot be changed. The method does not execute on the thread that created the timer; it executes on a ThreadPool thread supplied by the system.

Then in System.Threading.Threadpool

There is one thread pool per process. Beginning with the .NET Framework 4, the default size of the thread pool for a process depends on several factors, such as the size of the virtual address space. A process can call the GetMaxThreads method to determine the number of threads. The number of threads in the thread pool can be changed by using the SetMaxThreads method. Each thread uses the default stack size and runs at the default priority.

Improve this answer

Comments

1

you want to use property called MaxDegreeOfParallelism, but you will need to modify your code a bit. and then test it and figure out the optimal number of threads you want to run in parallel. here is the link: http://msdn.microsoft.com/en-us/library/system.threading.tasks.paralleloptions.maxdegreeofparallelism.aspx

Improve this answer

1 Comment

MaxDegree imposes a Ceiling. Not a minimum.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.