Parallel.ForEach and foreach loops for BlockingCollection.GetConsumingEnumerable() collection in TaskFactory.Tasks

Question

I have experimented with both of these loops and came to notice that even though regular foreach loop in Task's Action delegate supposed to perform in parallel, it doesn't process elements in parallel. However if i replace it with Parallel.ForEach i see data is being processed in parallel across multiple threads.

Code 1:

Task loadingTask1 = Factory.StartNew(() => { foreach (MyOneClass dg in Queue.GetConsumingEnumerable()) { MyOtherClass vl = new MyOtherClass(); vl.Id = dg.Id; vl.PerformTimeConsumingAction(); OutputQueue.Add(vl); } });

Code 2:

Task loadingTask2 = Factory.StartNew(() => { Parallel.ForEach(Queue.GetConsumingEnumerable(), (dg) => { MyOtherClass vl = new MyOtherClass(); vl.Id = dg.Id; vl.PerformTimeConsumingAction(); OutputQueue.Add(vl); }); });

Code 1 when run with Console.Write statement on each iteration seems to be waiting for the previous cycle to complete till it grabs next one, but Code 2 does process multiple elements in parallel.

Am i not understanding regular foreach in Task.Action correctly? I thought .NET would start as many threads for the task as load warrants and each iteration of foreach would be processed in parallel.

I have also tried passing PLINQ result to both of the above codes and observer same behavior: regular foreach seemed to wait for the previous iteration to complete to start the next one, even though i have used .AsParallel() and .WithExecutionMode(ParallelExecutionMode.ForceParallelism) directives.

Any insight would be highly appreciated. I am aware of OrderingPartitioner class and may try using it

"even though regular foreach loop in Task's Action delegate supposed to perform in parallel" Where did you get this idea? A regular foreach is a regular foreach. No fancy parallel stuff in it. That's why we have Parallel.ForEach(). — R. Martinho Fernandes
– R. Martinho Fernandes, Commented Apr 17, 2011 at 16:10
i have compared regular foreach performance in Task's Action delegate and linear process. Inside Task's action delegate it was performing faster, so i assumed threadpool was deciding how many parallel threads to spawn for the task to perform optimally. — Dimitri
– Dimitri, Commented Apr 17, 2011 at 16:15
how fast? I bet it was either pure luck or bad benchmarking. Running a foreach inside a task still runs it sequentially, because it's still a foreach. There is no magic. What you can do is run stuff in parallel with that sequential foreach. — R. Martinho Fernandes
– R. Martinho Fernandes, Commented Apr 17, 2011 at 16:16
it was not double the speed, but it was noticeably faster. I'd say 10 seconds vs 7. You may be correct, it could have been either of those reasons you mentioned. Just wanted to make sure, no matter how you implement foreach, it will still wait for the previous iteration, even if you cycle through the BlockingCollection (the collection being parallel-ready) — Dimitri
– Dimitri, Commented Apr 17, 2011 at 16:18
Update: Using PLINQ to get Parallel query from BlockingCollection and the using regular foreach loop on the resulting Parallel Query gave the best performance and it kept records in order — Dimitri
– Dimitri, Commented Apr 17, 2011 at 21:42

R. Martinho Fernandes · Accepted Answer · 2011-04-17 16:43:24Z

A regular foreach always runs its iterations sequentially. There is no magic that turns it into a parallel construct in some situations. That would be like throwing you into the pit of despair, because then it would be difficult to assert the correctness of something as simple as a foreach loop. Fortunately, one of the goals of C# is to throw you into the pit of success:

C# throwing you into the pit of success

If you put a foreach loop running on a separate task, you get all the iterations running sequentially, but you can run other code in parallel with the entire foreach.

The flow of execution of a regular foreach on a separate task looks like this:

 | __v_ / \ other code | | foreach iteration 1 other code | | foreach iteration 2 other code | | foreach iteration 3 ...... other code | | foreach iteration n-1 other code | | foreach iteration n v v

And the flow of execution of a Parallel.Foreach looks like this:

 | _________________v________________ / / / / \ \ \ \ |1 |2 |3 |....| |n-2 |n-1 |n \____\____\____\____/____/____/____/ | v

Hope that helps understand what's happening.

@Dimitri: It's from Jon Skeet's blog: msmvps.com/blogs/jon_skeet/archive/2010/09/02/….
Now, is there a way to guarantee First In First Out processing of BlockingCollection using Parallel.ForEach? I guess i'll have to implement Custom Partitioner?
@DimitriIf you want FIFO behavior, use a ConcurrentQueue instead.
I use BlockingCollection that wraps around ConcurrentQueue by default, doesn't it?

Collectives™ on Stack Overflow

Parallel.ForEach and foreach loops for BlockingCollection.GetConsumingEnumerable() collection in TaskFactory.Tasks

1 Answer 1

7 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Linked

Related