Why isn't this Parallel.ForEach loop improving performance?

Question

I have the following code:

 if (!this.writeDataStore.Exists(mat)) { BlockingCollection<ImageFile> imageFiles = new BlockingCollection<ImageFile>(); Parallel.ForEach(fileGrouping, fi => DecompressAndReadGzFile(fi, imageFiles)); this.PushIntoDb(mat, imageFiles.ToList()); }

DecompressAndReadGzFile is a static method in the same class that this method is contained in. As per the method name I am decompressing and reading gz files, lots of them, i.e. up to 1000, so the overhead of parallelisation is worth it for the benefits. However, I'm not seeing the benefits. When I use ANTS performance profiler I see that they are running at exactly the same times as if no parallelisation is occuring. I also check the CPU cores with process explorer and it looks like there is possibly work being done on two cores but one core seems to be doing most of the work. What am I not understanding as far as getting Parallel.ForEach to decompress and read files in parallel?

UPDATED QUESTION: What is the fastest way to read information in from a list of files?

The Problem (simplified):

There is a large list of .gz files (1200).
Each file has a line containing "DATA: ", the location and line number are not static and can vary from file to file.
We need to retrieve the first number after "DATA: " (just for simplicity's sake) and store it in an object in memory (e.g. a List)

In the initial question, I was using the Parallel.ForEach loop but I didn't seem to be CPU bound on more than 1 core.

Is there any synchronization done in DecompressAndReadGzFile? — SimonC
– SimonC, Commented Nov 10, 2011 at 7:06
Not that I am aware of. Although there is a call to imageFiles.Add which automatically adds a lock from what I understand. — Seth
– Seth, Commented Nov 10, 2011 at 7:39

Jon Skeet · Accepted Answer · 2011-11-10 07:04:32Z

Is it possible that the threads are spending most of their time waiting for IO? By reading multiple files at a time, you may be making the disk thrash more than it would with a single operation. It's possible that you could improve performance by using a single thread reading sequentially, but then doling out the CPU-bound decompression to separate threads... but you may actually find that you only really need one thread performing the decompression anyway, if the disk is slower than the decompression process itself.

One way to test this would be to copy the files requiring decompression onto a ramdisk first and still use your current code. I suspect you'll then find you're CPU-bound, and all the processors are busy almost all the time.

(You should also consider what you're doing with the decompressed files. Are you writing those back to disk? If so, again there's the possibility that you're basically waiting for a thrashing disk.)

I am not writing the decompressed files to disk. I am reading the compress files into memory using GZipStream, creating a TextREader to extract what I need putting it into that imageFiles collection in the code above. I might try your RAM disk suggestion though.
@Seth: Note that the ramdisk suggestion is really just to verify that you're IO-bound rather than CPU-bound. If that's the case, then you'll just end up moving the cost to the "copying the data onto the ramdisk" stage.
Remember those hard disk drive turbo booster software drivers years ago? They made disk I/O faster by compressing the data in memory, and then writing the compressed data to disk.

ratneshsinghparihar · Accepted Answer · 2011-11-11 08:18:54Z

is there any chance your static method is sharing any global resource among its calls. Because in that case this static method will be called sequentially and no parallel benefit. Can you put your fileGrouping class code ?

Collectives™ on Stack Overflow

Why isn't this Parallel.ForEach loop improving performance?

2 Answers 2

3 Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Related