how to read from a huge file and write to a new file by java

Question

What I am doing is to read one file line by line, format every line, then write to a new file. But the problem is that the file is huge, nearly 178 MB. But always getting error message: IO console updater error, java heap space. Here is my code:

public class fileFormat { public static void main(String[] args) throws IOException{ String strLine; FileInputStream fstream = new FileInputStream("train_final.txt"); BufferedReader reader = new BufferedReader(new InputStreamReader(fstream)); BufferedWriter writer = new BufferedWriter(new FileWriter("newOUTPUT.txt")); while((strLine = reader.readLine()) != null){ List<String> numberBox = new ArrayList<String>(); StringTokenizer st = new StringTokenizer(strLine); while(st.hasMoreTokens()){ numberBox.add(st.nextToken()); } for (int i=1; i< numberBox.size(); i++){ String head = numberBox.get(0); String tail = numberBox.get(i); String line = head + " "+tail ; System.out.println(line); writer.write(line); writer.newLine(); } numberBox.clear(); } reader.close(); writer.close(); } }

How can I avoid this error message? Moreover, I have set the VM preference: -xms1024m

@MadProgrammer: That should not cause OOM. The buffer is not that huge. — Thilo
– Thilo, Commented Sep 5, 2013 at 7:41
Looks like it is an eclipse problem. Have a look at this thread : stackoverflow.com/questions/11001252/running-out-of-heap-space — Marc
– Marc, Commented Sep 5, 2013 at 7:42
Probably caused by the fact that you write to system.out the entire file... — Marc
– Marc, Commented Sep 5, 2013 at 7:43

Daniel S. · Accepted Answer · 2013-09-05 07:45:43Z

3

Remove the line

System.out.println(line);

This is a workaround the fialing console updater, which otherwise runs out of memory.

answered Sep 5, 2013 at 7:45

Daniel S.

6,7004 gold badges42 silver badges85 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Marc Over a year ago

I reproduced the problem and removing the line solves it. This is an eclipse problem. the code works like a charm without writing to the sysout.

Thilo · Accepted Answer · 2013-09-05 07:45:45Z

The program looks okay. I suspect the problem is that you run this inside of Eclipse, and System.out is collected by Eclipse in memory (to be displayed in that Console window).

 System.out.println(line);

Try to run it outside of Eclipse, change Eclipse settings to pipe System.out somewhere, or remove the line.

Claudiu · Accepted Answer · 2013-09-05 07:58:36Z

This part of the code:

 for (int i=1; i< numberBox.size(); i++){ String head = numberBox.get(0); String tail = numberBox.get(i); String line = head + " "+tail ; System.out.println(line); writer.write(line); writer.newLine(); }

Can be translated to:

 String head = numberBox.get(0); for (int i=1; i< numberBox.size(); i++){ String tail = numberBox.get(i); System.out.print(head); System.out.print(" "); System.out.println(tail); writer.write(head); writer.write(" "); writer.write(tail); writer.newLine(); }

This may add a little code duplication but it avoids creating a lot of objects.

Also there if you merge this for loop with the loop contructing the numberBox, you won't need numberBox structure at all.

Ashish Pancholi · Accepted Answer · 2013-09-05 08:01:16Z

If you read whole file the heap memory will occupy so better option in to read the file in chuck. See my below code. It will start reading from the offset given in argument and will return the end offset . You need to pass number of lines to be read.

Please remember: You can use any collection to store these read lines and clear the collection before calling this method to read next chunk.

FileInputStream fis = new FileInputStream(file); InputStreamReader streamReader = new InputStreamReader(fis, "UTF-8"); LineNumberReader reader = new LineNumberReader(streamReader);

//call this below method recursively until the file does not reaches to the end

public int getParsedLines(LineNumberReader reader, int iLineNumber_Start, int iNumberOfLinesToBeRead) { int iLineNumber_End = 0; int iReadUptoLines = iLineNumber_Start + iNumberOfLinesToBeRead; try { reader.mark(iLineNumber_Start); reader.setLineNumber(iLineNumber_Start); do { String str = reader.readLine(); if (str == null) { break; } // your code iLineNumber_End = reader.getLineNumber(); } while (iLineNumber_End != iReadUptoLines); } catch (Exception ex) { // exception handling } return iLineNumber_End; }

Collectives™ on Stack Overflow

how to read from a huge file and write to a new file by java

4 Answers 4

1 Comment

Comments

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

Comments

Comments

Comments

Linked

Related