What you are observing is the behavior of the await operator, not the behavior of the Task.WhenAll method. If you are interested in why the await behaves this way, you could read this article from the early days of async/await:

Having the choice of always throwing the first or always throwing an aggregate, for await we opt to always throw the first. This doesn’t mean, though, that you don’t have access to the same details. In all cases, the Task’s Exception property still returns an AggregateException that contains all of the exceptions, so you can catch whichever is thrown and go back to consult Task.Exception when needed. Yes, this leads to a discrepancy between exception behavior when switching between task.Wait() and await task, but we’ve viewed that as the significant lesser of two evils.

In case you would like to implement a method similar in behavior to Task.WhenAll, but without losing the convenience of the async/await machinery, it is tricky but there are workarounds available here.

What you are observing is the behavior of the await operator, not the behavior of the Task.WhenAll method. If you are interested in why the await behaves this way, you could read this article from the early days of async/await:

Having the choice of always throwing the first or always throwing an aggregate, for await we opt to always throw the first. This doesn’t mean, though, that you don’t have access to the same details. In all cases, the Task’s Exception property still returns an AggregateException that contains all of the exceptions, so you can catch whichever is thrown and go back to consult Task.Exception when needed. Yes, this leads to a discrepancy between exception behavior when switching between task.Wait() and await task, but we’ve viewed that as the significant lesser of two evils.

Modifying the behavior of await to throw an AggregateException instead of the first exception is not very difficult. See this question for ideas.