Make worker async cache minimum thread configurable#17179
Make worker async cache minimum thread configurable#17179humengyu2012 wants to merge 2 commits intoAlluxio:master-2.xfrom
Conversation
| @humengyu2012 Thanks for the contribution and your insightful findings! After reviewing the PR, I'm not sure when the worker async cache minimum thread is very small (e.g. =1) or large (>>4), whether there'll be an impact on other read/write operations. @beinan When you're available, would you mind taking a quick review of this PR? This might be helpful for our Dora worker and local cache development. |
There was a problem hiding this comment.
It LGTM. @ChunxuTang The default behavior is not changed if we merge this PR, and we can specific the core pool size other than 4.
| This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions. |
github-actions bot added the stale 5 participants
What changes are proposed in this pull request?
Make worker async cache minimum thread configurable.
Why are the changes needed?
We want to concurrently cache a file of 30G size(250 blocks), but we found that the concurrency of the file cache is always 4 due to 250 < 512(the default value of queue size). If the number of blocks we cache at the same time is always less than 512, then we will never get a concurrency greater than 4. This is too slow, we should make the minimum threads configurable.
Does this PR introduce any user facing changes?
No