7

I have a database with 150k records. I want to write this to file as fast as possible. I've tried many approaches, but all seem slow. How do I make this faster?

I read these records in blocks of 40k. So first I read 40k then another 40k and so on.

After reading the records, this process returns a StringBuilder which contains 40k lines. Then we write this StringBuilder to a file. 

private static void write(StringBuilder sb, Boolean append) throws Exception { File file = File.createTempFile("foo", ".txt"); FileWriter writer = new FileWriter(file.getAbsoluteFile(), append); PrintWriter out = new PrintWriter(writer); try { out.print(sb); out.flush(); writer.flush(); } finally { writer.close(); out.close(); } }

I read this other example but it is equally slow: Fastest way to write huge data in text file Java

I also tried it with NIO api:

private static void write(StringBuilder sb, Boolean append)) throws Exception { FileChannel rwChannel = new FileOutputStream("textfile.txt", true).getChannel(); ByteBuffer bb = ByteBuffer.wrap(sb.toString().getBytes("UTF-8")); rwChannel.write(bb); rwChannel.close(); }

Which is the best method to write/append huge data into file?

Improve this question
2
  • 2
    I believe the problem is with out.print(sb). This forces toString() on the builder, and that in turn involves copying the underlying character array. If you need speed, try simple rewriting using a char array buffer and raw byte streams. Commented Sep 10, 2013 at 12:17
  • It's the network communication part that is slowing you down, find a way to avoid transmitting raw data across a network. Perhaps consider file compression to speed up the transmission phase. Commented Mar 21, 2015 at 22:39

5 Answers 5

Reset to default
11

You don’t need a PrintWriter here. If you have whatever kind of Writer (e.g. a FileWriter) you can simply invoke append(sb) on it. And you don’t need to flush, close implies flushing.

private static void write(StringBuilder sb, Boolean append) throws Exception { File file = File.createTempFile("foo", ".txt"); try(FileWriter writer = new FileWriter(file.getAbsoluteFile(), append)) { writer.append(sb); } }

On my system I encountered a small performance improvement using a Channel rather than an OutputStream:

private static void write0a(StringBuilder sb, Boolean append) throws Exception { File file = File.createTempFile("foo", ".txt"); try(Writer writer = Channels.newWriter(new FileOutputStream( file.getAbsoluteFile(), append).getChannel(), "UTF-8")) { writer.append(sb); } }

However these are only slight improvements. I don’t see much possibilities here as all the code ends up calling the same routines. What could really improve your performance is keeping the Writer alive during the invocations and not flushing every record.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

6

If you have a huge amount of data, it's better that you don't store it to StringBuilder and then write it to file at once.

This is the best scenario:

1) Before you start process on the data create FileInputStream

FileOutputStream fos = new FileOutputStream("/path/of/your/file");

2) Create and OutputStreamWriter from this file

OutputStreamWriter w = new OutputStreamWriter(fos, "UTF-8");

3) Create BufferedWriter (Improve file writing performance)

BufferedWriter bw = new BufferedWriter(w);

4) Pass bw to your process function and then flush/close

bw.flush(); bw.close();

The functionality of StringBuilder and BufferedWriter is almost same, So you do not need to change your code so much. The only negative point of this scenario is that, your process will involve all the time that the data are writing to file, but if you don't process the data in different thread, it is not an issue.

In this way, it doesn't matter how large data is it

Improve this answer

Comments

1

You are using a FileWriter (or a FileOutputStream in the second example). These are not buffered! So they write single chars resp. bytes to the disk.

That means, you should wrap the FileWriter in a BufferedWriter (or the FileOutputSystem in a BufferedOutputSystem).

private static void write(StringBuilder sb, Boolean append) throws Exception { File file = File.createTempFile("foo", ".txt"); Writer writer = new BufferedWriter(new FileWriter(file.getAbsoluteFile(), append)); PrintWriter out = new PrintWriter(writer); try { out.print(sb); out.flush(); writer.flush(); } finally { writer.close(); out.close(); } }
Improve this answer

2 Comments

Nope, from FileWriter's javadoc: The constructors of this class assume that the default character encoding and the default byte-buffer size are acceptable
Oh, yes. You are right. I had FileOutputStream in mind (and wrongly assumed, that FileWriter behaves the same). Hmm ...
0

You are opening the file, writing one line, then closing it. It's the opening and closing that takes the time here. Find a way to keep the output file open.

Improve this answer

Comments

-1

Did you try Apache IO, is the performance still the same?

Improve this answer

1 Comment

Try it from here[commons.apache.org/proper/commons-io/download_io.cgi] and tell me

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.