How to write large result into a file in Java

Question

I want to generate some data and write them into a file. The size of these data is larger than the capacity of memory. Which way should I use to write these data?

Now I am using this way:

BufferedWriter output = new BufferedWriter(new OutputStreamWriter( new FileOutputStream(fileName), "utf-8" ));

I don't know if it works when the data is very large, and if there is other way with better performance.

Also, the encoding must be "utf-8".

How is the data being generated? All at once, or in smaller chunks? (If in smaller chunks, then your current solution is fine. ) — Louis Wasserman
– Louis Wasserman, Commented May 17, 2014 at 20:03
Or you can use FileWriter directly, as mentioned in stackoverflow.com/questions/1062113/… — conFusl
– conFusl, Commented May 17, 2014 at 20:09
Perfectly allright. Tip: instead of a String "utf-8" you could use a Charset constant StandardCharsets.UTF_8. — Joop Eggen
– Joop Eggen, Commented May 17, 2014 at 21:08

MCHAppy · Accepted Answer · 2014-05-17 20:20:25Z

You can handle very large files using this method

Open input stream Open output stream create byte buffer while (read stuff into byte buffer) { write byte buffer to output stream }

That's how you do this. You use a buffer (say 8k)

 byte[] b = new byte[8192];

Read into that, write it out. Repeat until you run out of things to read in.

Calculate the buffer size after calculating VM heap size. You may want to store constant values on a file e.g. XML and then transfer via FIleChannels which is 250% faster
Ummmm ...i didn't understand you well , can you post this solution as an answer ? it appears to be useful for me

Community · Accepted Answer · 2017-05-23 10:24:15Z

When you start the JVM you can either specify heap (RAM) size via the -Xmx and -Xms flags or just run it by java -jar *.jar and have it use default heap size see How is the default java heap size determined? You can use launch4j to make a native excecutable for most os' or you can make the jar automatically restart itself with more RAM (All these need to be done with caution in order to avoid stupidly using RAM that do not exists).

If you have constant values , store them somewhere before you make the file programmatically (database/files/blah).

I believe most of the people would do this

Open input stream Open output stream create byte buffer while (read stuff into byte buffer) { write byte buffer to output stream }

PREFER FILECHANNELS THOUGH , they are better and faster. Perform some googling by yourself , I am sure you will solve your problem. Be sure to flush any buffers before you use them and close streams etc... at the end!

P.S. Strings do not take THAT much to be written to a file. IDK if one second matters THAT MUCH!

Collectives™ on Stack Overflow

How to write large result into a file in Java

2 Answers 2

2 Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Linked

Related