Java―good size for a char buffer

Question

I am coding a little java based tool to process mysqldump files, which can become quite large (up to a gigabyte for now). I am using this code to read and process the file:

BufferedReader reader = getReader(); BufferedWriter writer = getWriter(); char[] charBuffer = new char[CHAR_BUFFER_SIZE]; int readCharCout; StringBuffer buffer = new StringBuffer(); while( ( readCharCout = reader.read( charBuffer ) ) > 0 ) { buffer.append( charBuffer, 0, readCharCout ); //processing goes here }

What is a good size for the charBuffer? At the moment it is set to 1000, but my code will run with an arbitrary size, so what is best practice or can this size be calculated depending on the file size?

Thanks in ahead, greetings philipp

Oracle's BufferedReader already uses a default buffer of 8192. — Sotirios Delimanolis
– Sotirios Delimanolis, Commented Sep 25, 2013 at 14:31
I don't know if there's a standard for this as it'll depend on your available memory. I would recommend experimenting with it at different sizes to see how it affects your performance — StormeHawke
– StormeHawke, Commented Sep 25, 2013 at 14:31
@StormeHawke AFAIK the best values are 4096 and 8192, which value to use depends entirely on your hard drive speed. — Luiggi Mendoza
– Luiggi Mendoza, Commented Sep 25, 2013 at 14:32
@SotiriosDelimanolis but then you get the wrong idea. Nor OpenJDK nor HotSpot source code are the codes. If the size is not specified in the javadoc, then you must not assume it will always be 8192, note that JRockit or IBM JVM can change this. — Luiggi Mendoza
– Luiggi Mendoza, Commented Sep 25, 2013 at 14:34
@NFE Just like Luiggi Mendoza stated, it depends on the implementation. I was referring to Oracle JDK 7's implementation. The OpenJDK also seems to use that size. — Sotirios Delimanolis
– Sotirios Delimanolis, Commented Sep 25, 2013 at 14:43

Community · Accepted Answer · 2017-05-23 10:29:12Z

2

It should always be a power of 2. The optimal value is based on the OS and disk format. In code I've seen 4096 is often used, but the bigger the better.

Also, there are better ways to load a file into memory.

edited May 23, 2017 at 10:29

CommunityBot

11 silver badge

answered Sep 25, 2013 at 14:33

mikeslattery

4,1191 gold badge21 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Luiggi Mendoza Over a year ago

More than a power of 2, it should be a power of 1024.

philipp Over a year ago

I tries a bunch of values which were a power of 2, all ran fine. But except for very small values I could not see any extraordinary performance gains, but I guess that is due to my implementation

mikeslattery Over a year ago

I could be due to a lot of factors. Many OSes and disk controllers are smart enough to read blocks ahead of time. When that occurs, your buffer size doesn't matter much, except for the cost of round-tripping to the OS API.

Collectives™ on Stack Overflow

Java―good size for a char buffer

1 Answer 1

3 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Linked

Related