Storing file contents to StringBuilder apparently takes too much memory

Question

I have this code:

public static void main(String[] args) { System.out.println("Reading file..."); String content = readFile(args[0]); System.out.println("Done reading file."); } private static String readFile(String file) throws IOException { BufferedReader reader = new BufferedReader( new FileReader (file)); String line = null; StringBuilder stringBuilder = new StringBuilder(); while( ( line = reader.readLine() ) != null ) { stringBuilder.append( line ); } return stringBuilder.toString(); }

The readFile method works fine, well, for small files.

The thing I noticed is that it takes too much memory.

If I open the System Monitor on windows (CTRL-SHIFT-ESC), I see the java process taking up to 1,8GB RAM, while the size of my file is just 550MB.

Yes, I know, loading a file entirely into memory isn't a good idea, I'm doing this just for curiosity.

The program gets stuck at Reading file... when the newly created java process starts, it takes a bunch of MB of RAM and goes up to 1,8GB.

I also tried using String concatenation instead of using StringBuilder, but I have the exact same result.

Why does it take so much memory? Is the final stringBuilder.toString causing this?

I also tried using String concatenation instead of using StringBuilder, but I have the exact same result. - I am afraid the result would have been worse in case of String — TheLostMind
– TheLostMind, Commented Aug 18, 2014 at 14:24
then just user a scanner to read in the file if your problem is memory — rush2sk8
– rush2sk8, Commented Aug 18, 2014 at 14:32

Peter Lawrey · Accepted Answer · 2014-08-18 14:29:29Z

You have to remember how these libraries work.

One byte on disk can turn into 2 byte char. The StringBuilder grows by doubling in capacity so it can be up to twice as large as you really need, and you need both the StringBuilder and String in memory at the same time.

So take your example. 550 MB can turn into 1100 MB as char alone. However, the size doubles in size so the it will be approximately the next power of two i.e. it could be 2 GB, and this is on top of a the String which would be 550 MB.

Note: the reason it is not using this much memory is that you have a bug. You are discarding all the new lines \r\n which means you have less characters.

When processing a large file where you don't have enough memory to load it into memory at once, you are better off processing the data as your read it.

BTW If you have plenty of memory, you can read the file faster, with less memory this way.

static String readFile(String file) throws IOException { try(FileInputStream fis = new FileInputStream(file)) { byte[] bytes = new byte[(int) fis.available()]; fis.read(bytes); return new String(bytes); } }

the reason it is not using this much memory is that you have a bug. You are discarding all the new lines Well, it's not a bug, I'm doing it because I need to discard them :) Anyway, thank you, everything is clear now :) And yes, your method is much faster and takes less memory, it's great.
@BackSlash to reduce it further I would use a memory mapped file. This will use almost no heap.
Uh! I didn't know about memory mapped files... I'm taking a look to MappedByteBuffer, thank you again! :)
@BackSlash This allows you to use a file as if it was memory, without bringing it into the heap. I have mapped in a 100 TB file using memory mapping.
But if I store the content into a String (either the full content, or just a small part of it) I think it would allocate the String into the heap... Am I right?

Collectives™ on Stack Overflow

Storing file contents to StringBuilder apparently takes too much memory

1 Answer 1

6 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Related