What you are observing is the behavior of the await operator, not the behavior of the Task.WhenAll method. If you are interested in why the await behaves this way, you could read this article from the early days of async/await:
Having the choice of always throwing the first or always throwing an aggregate, for
awaitwe opt to always throw the first. This doesn’t mean, though, that you don’t have access to the same details. In all cases, the Task’s Exception property still returns anAggregateExceptionthat contains all of the exceptions, so you can catch whichever is thrown and go back to consultTask.Exceptionwhen needed. Yes, this leads to a discrepancy between exception behavior when switching betweentask.Wait()andawait task, but we’ve viewed that as the significant lesser of two evils.
Modifying the behavior of await to throw an AggregateException instead of the first exception is not very difficult. See this question for ideas.