2

I have a simply foreach loop that limits itself based on while loops and a static int. If I dont limit it, my CPU stays under 10% if i limit it my CPU goes up to 99/100%. How do I safely limit the number of calls to a class within a Paralell.Foreach?

static int ActiveThreads { get; set; } static int TotalThreads { get; set; } var options = new ParallelOptions(); options.MaxDegreeOfParallelism = 1; Parallel.ForEach(urlTable.AsEnumerable(),options,drow => { using (var WCC = new MasterCrawlerClass()) { while (TotalThreads <= urlTable.Rows.Count) { if (ActiveThreads <= 9) { Console.WriteLine("Active Thread #: " + ActiveThreads); ActiveThreads++; WCC.MasterCrawlBegin(drow); TotalThreads++; Console.WriteLine("Done Crawling a datarow"); ActiveThreads--; } } } });

I need to limit it, and yes I understand Max Parallelism has it's own limit, however, my switch gets bogged down before the CPU in the server hits that limit.

Improve this question
4
  • 2
    You don't seem to be passing the ParallelOptions (options variable) to Parallel.ForEach (it should slip in between the first and second argument). How else will you be limiting the parallelism? Commented Apr 11, 2012 at 23:34
  • @yamen Even if I pass the option to limit it, the threshold is hit before it stops creating threads Commented Apr 11, 2012 at 23:42
  • @yamen And i'm trying to limit the number of times it can enter the class at any one time Commented Apr 11, 2012 at 23:43
  • 1
    If you want to put a limit on entry into a particular class, you can gate that class using a semaphore. It's not really 100% clear what you're trying to achieve here. Perhaps provide a more complete code sample? Commented Apr 11, 2012 at 23:45

2 Answers 2

Reset to default
2

Two things :

1) You don't seem to be using your ParallelOptions() that you created in this example.

2) You can use a Semaphore if for some reason you don't want to use the ParallelOptions.

 Semaphore sm = new Semaphore(0, 9); // increment semaphore or block if = 9 // this will block gracefully without constantly checking for `ActiveThreads <= 9` sm.WaitOne(); // decrement semaphore sm.Release();
Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

I cant use the ParallelOptions, i took out the options to test, the threshold of the ParalellOption =1 is never hit before the switch blows up, so I have to limit the number of times accesses the class
A Semaphore seems to be an answer, by definition it does just that - it protects the critical section
@xavier Maybe a silly question but where do I place the sm.WaitOne() is it within the ForeachLoop?
@Mike if I get your problem correctly, then you create a Semaphore before foreach and before the call to your class you place sm.WaitOne(). After the call you put sm.Release() and you're done.
@xavier Thanks, could you provide an example using what I have above, I tried placing the sm.WaitOne() and placed it before the class call but it never actually got past sm.WaitOne()
|
1

I have a simply foreach loop that limits itself based on while loops and a static int. If I dont limit it, my CPU stays under 10% if i limit it my CPU goes up to 99/100%.

That is pretty odd. It may be a result of the way you have limited the concurrency with the loop which, by the way, appears to cause each drow to be crawled many times. I doubt that is what you want. You are getting low CPU utilization because the crawl operation is IO bound.

If you really want to limit the number of concurrent calls to MasterCrawlBegin to 9 then set MaxDegreesOfParallelism = 9. The while loop and maintanence of TotalThreads and ActiveThreads is not going to work. As a side note you are incrementing and decrementing the counters in a manner that is not thread-safe.

Change your code to look like this.

int ActiveThreads = 0; var options = new ParallelOptions(); options.MaxDegreeOfParallelism = 9; Parallel.ForEach(urlTable.AsEnumerable(),options,drow => { int x = Interlocked.Increment(ref ActiveThreads); Console.WriteLine("Active Thread #: " + x); try { using (var WCC = new MasterCrawlerClass()) { WCC.MasterCrawlBegin(drow); } } finally { Interlocked.Decrement(ref ActiveThreads); Console.WriteLine("Done Crawling a datarow"); } });
Improve this answer

Comments