5

I just wonder is parallel File.Read using PLINQ/Parallel can be faster? My code is as follows ( .Net 4.0):

public static void ReadFileParallel(List<string> fileName) { Parallel.Foreach(fileName, file=>File.Read(file)); } public static void ReadFilePLINQ(List<string> fileName) { fileName.AsParallel().foreach(file=>File.Read(file)); }

The reason I ask this is because I thought that file reading is IO bound, so doing parallel won't help, am I right?

Improve this question

8 Answers 8

Reset to default
7

It depends.

If your files were in different locations, on different network shares, or on different physical hard drives, then yes, parallel loading will probably help. If they're on a single spinning hard drive, reading the files in parallel will probably hurt your performance significantly due to the extra seek time that you will likely incur from these parallel reads.

If your files are on an SSD, you will probably get slightly less performance, but it would depend on how many files you're reading in parallel and what their sizes are. I imagine that at a certain file size threshold and number of parallel reads, performance will drop significantly. Hard to tell on that one without some experimentation.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Those are reasonable criteria. In practice, I'd say measure it rather than guessing, though.
1

You'd think so, but that's not what measurements show. When file I/O has significant latency, particularly over networks, doing it in parallel can keep the pipe filled.

Improve this answer

Comments

0

There is an excellent PDF from MSFT which goes into detail regarding Parallel and threading possibilities.

It might help.

http://www.microsoft.com/downloads/details.aspx?FamilyID=86b3d32b-ad26-4bb8-a3ae-c1637026c3ee&displaylang=en

Improve this answer

Comments

0

To a first approximation, it will help if the files are on different disks and make it slower otherwise (due to increased time spent seeking).

It might be slightly faster if all the files are cached (since you can use multiple cores).

Your best bet, is of course, to run some benchmarks.

Improve this answer

Comments

0

You are not exactly doing a parallel File.Read, you are doing multiple File.Reads in parallel. If the files are in different spindles you will experience improved throughput just by utilizing multiple spindles at once.

You can also experience improved performance even if you use a single spindle, if each Read is followed by CPU-bound processing, although in this case it would be much better to and schedule Tasks objects. In this case you can have some tasks loading data from files while others use already loaded data to execute some heavy processing.

Improve this answer

2 Comments

Yeah, but if his files are on the same HDD, he'll hit the head search time, and the throughput will decrease much worse then 2 times. Remember the average seek time for an 3.5" 7200 RPM drive is 13-15 milliseconds. And unlike capacity and linear read/write rate, this figure is consistent over the last several years.
That's why I said "each read followed by CPU-bound processing". While one thread is reading the file, another is doing processing, thus keeping both of them working.
0

I think that you've pretty much hit the nail on the head here.

Parallel Operations in general are always throttled by the point at which you run out of resources to run the operations in parallel on, but even then you still have diminishing returns on an increasing amount of parallel threads.

Jeff Atwood tweeted up an interesting graph which I will add to this later showing the diminishing returns of mutli-core processors with a multi-threading environment. Granted this is not exactly the same. But let's look at this from the thought that even if you had 100 files on 100 hard drives, somewhere that IO is getting pipped back down a single channel which will cause some diminishing of the read increase.

What I'm basically trying to say is just running something in parallel doesn't mean that it will be sped up dramatically, it's important to consider how the parallel processes are actually being executed.

Improve this answer

Comments

0

It is tricky business. If you do it wrong, the disk head is moving back and forth trying to read two files at the same time. This is especcially a problem on larger files.

However, if you read a lot of small files in parallel, you may gain a little because the disk subsystem can choose to read the files in a different order than you asked. I have not seen this effect in real life, however.

Also processing you do on the content can be done in parallel with the reading of the files. So you need to profile and benchmark before you ship.

Improve this answer

Comments

0

This an old post, but I was curious about this question with a current version of C# with files on a local SSD drive. I used BenchmarkDotNet to measure performance of reading files in parallel + async.

From the results of my test with, parallel code is ~3.4 times as fast as serial. I suspect this likely has to due more-so with utilization of multiple cores and keeping the IO pipeline full.

Benchmark code

[SimpleJob(RuntimeMoniker.Net60, baseline: true)] [RPlotExporter] public class FileReadBenchmark { private List<string> _files = new List<string>(); [Params(10, 100)] public int Count; [GlobalSetup] public void Setup() { _files = Directory.GetFiles(@"C:\LocalDirectory\") .Take(Count) .ToList(); } [Benchmark] public void ParallelFileRead() { Parallel.ForEach(this._files, (f) => { _ = File.ReadAllLines(f); }); } [Benchmark] public async Task ParallelFileReadAsync() { await Parallel.ForEachAsync(this._files, async (f, c) => { await File.ReadAllLinesAsync(f, c); }); } [Benchmark] public void SequentialFileRead() { this._files.ForEach((f) => { _ = File.ReadAllLines(f); }); } [Benchmark] public async Task SequentialFileReadAsync() { foreach (var f in _files) { await File.ReadAllLinesAsync(f); } } } 
BenchmarkDotNet=v0.13.3, OS=Windows 10 (10.0.19044.2364/21H2/November2021Update) 11th Gen Intel Core i7-1185G7 3.00GHz, 1 CPU, 8 logical and 4 physical cores .NET SDK=7.0.101 [Host] : .NET 6.0.12 (6.0.1222.56807), X64 RyuJIT AVX2 .NET 6.0 : .NET 6.0.12 (6.0.1222.56807), X64 RyuJIT AVX2 Job=.NET 6.0 Runtime=.NET 6.0 | Method | Count | Mean | Error | StdDev | Median | Ratio | |------------------------ |------ |----------:|----------:|----------:|----------:|------:| | ParallelFileRead | 10 | 1.166 ms | 0.0232 ms | 0.0595 ms | 1.167 ms | 1.00 | | | | | | | | | | ParallelFileReadAsync | 10 | 1.604 ms | 0.0385 ms | 0.1130 ms | 1.563 ms | 1.00 | | | | | | | | | | SequentialFileRead | 10 | 3.970 ms | 0.0785 ms | 0.0934 ms | 3.943 ms | 1.00 | | | | | | | | | | SequentialFileReadAsync | 10 | 4.043 ms | 0.0750 ms | 0.1692 ms | 3.996 ms | 1.00 | | | | | | | | | | ParallelFileRead | 100 | 10.981 ms | 0.2060 ms | 0.1826 ms | 11.007 ms | 1.00 | | | | | | | | | | ParallelFileReadAsync | 100 | 12.573 ms | 0.1678 ms | 0.1570 ms | 12.501 ms | 1.00 | | | | | | | | | | SequentialFileRead | 100 | 38.171 ms | 1.2692 ms | 3.7423 ms | 36.319 ms | 1.00 | | | | | | | | | | SequentialFileReadAsync | 100 | 43.606 ms | 0.8559 ms | 1.1130 ms | 43.589 ms | 1.00 |
Improve this answer

2 Comments

Isn't this just testing performance of windows' filesystem cache, i.e. no IO involved? learn.microsoft.com/en-us/windows/win32/fileio/file-caching
Good point - there may be a way to write the test so that it can properly test that. I think my hypothesis of keeping the IO pipeline full will still yield performance benefits, but would need to update accordingly.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.