How can I gather results from parallel async tasks in C#?

Question

I'm actually facing a problem with parallelism on asynchronous tasks.

My goal is to have an async Task which connects to multiple endpoints to retrieve the same kind of data. A list with all "connection strings" is provided. The data retrieval shall happen in parallel with all connections and the retrieved data of each connection shall be merged into a single container which will be returned, once all parallel async tasks finished.

My first idea was using AsParallel.ForAll() - passing the configurations, and an async Task to fill a (Dataflow) BufferBlock with SendAsync. Now I got stuck on the problem, that Parallel queries seem to only take Action<T> as parameter and not Task<T>. This will lead to an undesired behavior if an exception occurs.

Does anyone have a better approach to my problem?

Here is a snippet for better visualization:

configurations.AsParallel().ForAll(async config => await FetchAllData(config, resultBufferBlock, cancellationToken));

After some comments: I basically need Parallel.ForEachAsync which is available in .NET 6. we will move to .NET 6 in a couple of months. This is already planned. Although I would appreciate a solution for .NET 5. Actually it looks like I will use the bad way using the task as Action and fix that later…

Is this what you are looking for ? Parallel.ForEachAsync learn.microsoft.com/en-us/dotnet/api/… — maraaaaaaaa
– maraaaaaaaa, Commented Feb 2, 2022 at 18:32
Is this helpful? How to limit the amount of concurrent async I/O operations? Or this? Parallel foreach with asynchronous lambda — Theodor Zoulias
– Theodor Zoulias, Commented Feb 2, 2022 at 18:32
@maksymiuk I recently learned the Parallel.ForEachAsync is .net 6 (it is tagged .net 5) — Jeroen van Langen
– Jeroen van Langen, Commented Feb 2, 2022 at 18:35
@maksymiuk yes! Didn't come accross this because it's .Net 6 and currently we are on .Net 5. I'm already happy this problem is solved in .Net 6. Anyway I need a solution before we move to .NET 6 (we will probably do it in a couple of months). — Otto V.
– Otto V., Commented Feb 2, 2022 at 18:36
There's also the actual source code for the .NET 6 implementation of ForEachAsync: github.com/dotnet/runtime/blob/main/src/libraries/… — Gabriel Luci
– Gabriel Luci, Commented Feb 2, 2022 at 19:24

Stephen Cleary · Accepted Answer · 2022-02-03 05:04:28Z

I'm actually facing a problem with parallelism on asynchronous tasks.

The problem you describe is concurrency (doing more than one thing at a time) and AFAICT does not need parallelism (multiple threads).

Does anyone have a better approach to my problem?

The way to do asynchronous concurrency is to start all the tasks (commonly using a LINQ select with an asynchronous lambda), and then use Task.WhenAll to join them.

var tasks = configurations.Select(async config => await FetchAllData(config, resultBufferBlock, cancellationToken)); await Task.WhenAll(tasks);

After some comments: I basically need Parallel.ForEachAsync which is available in .NET 6.

I disagree. Parallel.ForEachAsync is for when you have a complex mixture of CPU-bound and I/O-bound code and need to do both parallelism and asynchronous concurrency. In this case parallelism isn't necessary.

Collectives™ on Stack Overflow

How can I gather results from parallel async tasks in C#?

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related