System.AggregateException in await Task.WhenAll()

Question

I have a process which retrieves from the DB a list of rows, and per each row must execute a list of long-running operations:

Upload a file to a FTP server
Check every 1 minute if a particular file (that is created based on the uploaded in 1) is available in such FTP server
Download the file found in step 2, once available
Save content of the file downloaded in the DB

I've implemented a method which does all these steps following the sequence mentioned above: I want to call that method multiple times in parallel (asynchronously): The amount of times to call it will depend of the number of rows retrieved at the beginning of the workflow.

An extraction of the main method follows:

List<Model> ftpModels = _service.CreateFtpModel(rowsFromDB); List<Task> asyncTasks = new List<Task>(); ftpModels.AsParallel().ForAll(p => { asyncTasks.Add(DoStepsAsync(p)); }); // Wait for all the tasks to finish. await Task.WhenAll(asyncTasks.Where(p => p != null));

First, a list of Models is created based on the rows retrieved from the DB, then an empty list of Tasks is created and then iterates through the models created to execute the async method "DoStepsAsync" which retrieves a task to insert into the asyncTasks list. Finally, it calls Tasks.WhellAll for all the tasks.

This approach is working properly usually, but I happened to find that sometimes (randomly so thread related issue) the next exception is thrown on the WhenAll method:

System.AggregateException: One or more errors occurred. ---> System.ArgumentException: The tasks argument included a null value.

Which clearly indicates me that any of the tasks above returned null (instead of a task) while they were being executed. Then I read the logs I'm writing but I did not find any exception anywhere, just the flow interrupted out of the blue falling next into the System.AggregateException

I was googling and reading a lot, and this issue happens when:

At least one of the Task instances was canceled. If a task was canceled, the AggregateException exception contains an OperationCanceledException exception in its AggregateException.InnerExceptions collection. -or- An exception was thrown during the execution of at least one of the Task instances.

But no exception was thrown as I have try-catch blocks implemented accordingly. Therefore, i don't know what the hell is happening.

Any suggestion?

Just to give more info about what happens within the DoStepsAsync() method, there are asynchronous calls to the FTP client (thread safe), writing logs and stuff like that. What really surprises me is that no exception is caught anywhere even with all the catch blocks mentioned above.

Any suggestion really appreciated.

did you try to enable debbuger to break when exception is thrown? — Krzysztof Skowronek
– Krzysztof Skowronek, Commented Nov 28, 2017 at 12:50
yes I did, but it does not happen in debug mode. Only in the server where the application is deployed, and as I said, not always. — Carlos
– Carlos, Commented Nov 28, 2017 at 13:04
asyncTasks.Where(p => p != null)) is not the same p as that above it, it's a Task which is likely never null. Did you you mean to filter the null p before the ForEach? — JSteward
– JSteward, Commented Nov 28, 2017 at 15:03
On a side note you'd have cleaner syntax if you used a Select to produce the asyncTasks collection instead of the mutating ForEach since the only command running inside the ForEach is a List<T>.Add — JSteward
– JSteward, Commented Nov 28, 2017 at 15:06
The exception comes from a method which have the "tasks" argument. Seemingly your code snippet cannot produce that exception. Try looking for another place in your program where you call something like WhenAll, WhenAny, etc. Async code does not provide good callstack to identify a particular call, unfortunately. — dvorn
– dvorn, Commented Nov 28, 2017 at 16:46

Carlos · Accepted Answer · 2017-12-21 12:45:36Z

Found the issue: As soon as I iterate through the models list via AsParallel().ForAll , it might happen (depends on the CPU workload) that the task to create (async) has not been yet created but is being accessed by the WhenAll method (async too), therefore throws the exception mentioned above. This happens sometimes, as I said, according to the CPU workload: In order to easily reproduce the issue , I've created a unit test which calls the problematic method up to 50 times, and usually fails within the first 10 runs. To solve the issue, I I've replaced the useless (because in this scenario does not bring anything relevant) function AsParallel().ForAll by ForEach and the issue does not happen anymore because the tasks are always created following a sequence (not executed though, they are executed asynchronously), no matter the CPU workload. Code will be as follows, to solve the issue:

List<Model> ftpModels = _service.CreateFtpModel(rowsFromDB); List<Task> asyncTasks = new List<Task>(); ftpModels.ForEach(p => { asyncTasks.Add(DoStepsAsync(p)); }); // Wait for all the tasks to finish. await Task.WhenAll(asyncTasks.Where(p => p != null));

I hope it helps in the future to other people.

Collectives™ on Stack Overflow

System.AggregateException in await Task.WhenAll()

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related