18

In my function, I need to read some data from a file into a buffer, manipulate the data and write it back to another file. The file is of unknown size and may be very large.

If I use a small buffer, there will be a long read/write cycle and it would take much time. In contrast, long buffer means I need to consume more memory. What is the optimal buffer size I should use? Is this case dependent?

I saw some application like 'Tera copy' in windows that manages huge files efficiently. Is there any other technique or mechanism I should be aware of?

Note: This program will be running under Windows.

Improve this question
4
  • 1
    How about a dynamic buffer size? Commented Mar 21, 2013 at 6:20
  • 2
    I say run some benchmarks to see. Also do not worry about allocating 1MB or 2 on a modern PC. Commented Mar 21, 2013 at 6:20
  • 2
    Perhaps using mmap could help. And Linux also have a readahead syscall. Commented Mar 21, 2013 at 6:21
  • See also stackoverflow.com/questions/8803515/… Commented Apr 24, 2014 at 10:28

6 Answers 6

Reset to default
27
+100

See what Microsoft has to say about IO size: http://technet.microsoft.com/en-us/library/cc938632.aspx. Basically, they say you should probably do IO in 64K blocks.

On *NIX platforms, struct stat has a st_blksize member which says what should be a minimal IO block size.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Note: the underlying reason of such observed behavior, and recommendation, is a hardware one. TL;DR: the best size is the maximum your hardware can provide, which appear to be 64K on modern "desktop" computers.
6

It is, indeed, highly case dependent, and you should probably just write your program to be able to handle a flexible buffer size, and then try out what size is optimal.

If you start small and then increase your buffer size, you will probably reach a certain size after which you'll see no or extremely small performance gains, since the CPU is spending most of its time running your code, and the overhead from the I/O has become negligible.

Improve this answer

Comments

3

First rule for these things is to benchmark. My guess would be that you prematurely optimizing. If you are doing real file IO, the bandwidth of your disk (or whatever) will usually be the bottleneck. As long as you write your data in chunks of several pages the performance shouldn't change too much.

What you could hope to is to do your computation of parts of the data in parallel to your write operation. For this you would have to keep two buffers, one which is currently written, and one on which you do the processing. Then you would use asynchronous IO funcions (aio_write on POSIX systems, probably something like that exists for Windows, too) and switch buffers for each iteration.

Improve this answer

Comments

1

Memory management is always case dependent and particularly when combined with file I/O.

There are two possible suggestions from my side.

1) Use fixed I/O buffer size, e.g. 64K, 256K, 512KB or 1MB. But in this case when there is I/O more than this fixed buffer size, you have to consider offsets to complete I/O in multiple iterations.

2) Use variable I/O buffer size using malloc(), but this also depends on certain factors. Such as available RAM in your system and maximum dynamic memory allocation limit for process in your OS.

Improve this answer

Comments

1

I will suggest you to use buffer size of page size. For example is page size is 4K then you can use 4K Byte buffer size to minimize context switches.

Improve this answer

Comments

-3

While I cant speak for the algorithm... Memory usage versus processor usage is a classic dilemma in programming and you should probably choose on a case by case basis... So if the system has 4GB Available RAM you can obviously consume quite a bit, whereas if you only have 512MB you should consume very little at the cost of exercising the CPU. Best way would be to check and change your size pro grammatically :)

Improve this answer

1 Comment

This is not a CPU vs memory question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.