Skip to main content
Software Engineering

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

7
  • My gut instinct in this situation is that you cannot do any optimization unless you work on OS or even HW layer. Maybe try to find OS APIs that could be used to call that would return the size of the whole directory, instead of doing it yourself? This would give OS way to use it's own optimizations. Commented Apr 12, 2020 at 7:43
  • @DocBrown I added test results Commented Apr 12, 2020 at 8:17
  • @tera_789: the test results don't seem to support what you wrote in your question - for 2 of them, ProcessPoolExecutor is faster, but for 3 of them, ThreadPoolExecutor. Commented Apr 12, 2020 at 9:11
  • 1
    @DocBrown yeah I see that...it fluctuates a lot...hard to make a decision thus. At this point, I am starting to think that either network or storage device's OS is playing a big role here...sometimes it is ThreadPoolExecutor, which is faster in most tests, and, sometimes it is ProcessPoolExecutor... Commented Apr 12, 2020 at 9:16
  • 1
    @DocBrown The thing is that CPU usage never goes really high when I run these scans (it is mostly 20-30%), sometimes even less. Commented Apr 12, 2020 at 9:36