6

Question

How can you read files > 1.1 GB into memory under node.js?

Example

I'm trying to use topojson under node.js to convert > 1.1 GB GeoJSON files to TopoJSON format.

$ topojson -o outfile.json larger_than_one_point_one_GB_input_file.json

(the above has worked for files up to 517 MB)

Results in the following error

buffer.js:242 this.parent = new SlowBuffer(this.length); ^ RangeError: length > kMaxLength at new Buffer (buffer.js:242:21) at Object.fs.readFileSync (fs.js:200:14) at /usr/local/share/npm/lib/node_modules/topojson/bin/topojson:61:26 at Array.forEach (native) at Object.<anonymous> (/usr/local/share/npm/lib/node_modules/topojson/bin/topojson:60:8) at Module._compile (module.js:449:26) at Object.Module._extensions..js (module.js:467:10) at Module.load (module.js:356:32) at Function.Module._load (module.js:312:12) at Module.runMain (module.js:492:10)

What I've Tried so Far

  • Extensive searching
  • Command line memory settings
    • --max-stack-size=2147000000
    • --max_executable_size=2000
    • --max_new_space_size=2097152
    • --max_old_space_size=2097152
  • Custom compile most recent v8 version into custom node.js install

Versions

  • node.js: v0.8.15
  • v8: 3.11.10.25
Improve this question
0

1 Answer 1

Reset to default
2

The problem is because topojson uses fs.readFileSync to read the entire file. What that does is open a buffer of size (length of file), then fill it up. But node buffers have a maximum size of 0x3FFFFFFF bytes, or 1GB - 1 byte. So you get that exception.

Solution? Open the topojson source and replace readFileSync with streaming methods which wouldn't read the entire file as one block. Or if you're feeling really hackish, maybe recompile node with a larger kMaxLength constant...

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

Fascinating. I'll try replacing readFileSync with a streaming method. If it works, I'll gladly mark your answer as correct. Thanks!
Good luck - I don't know how easy this will be since you may have to change some other things as well, like the JSON.stringify call later on - that would probably fail as well with the same error.
Unfortunately, I have not been successful so far. Probably because my javascript and node.js fu are weak. I read the file as a stream, but can't write it to memory (same issue). There's no point writing back to disk. Perhaps some intelligent way of splitting it across multiple buffers...
You could try recompiling Node for this specific one-time operation. Like I said, it's hacky, but for a one-time run should work. The only change you'd have to make is the kMaxLength constant. From my cursory glance it looks like everything else should just work.
Oh, and some of the constants in deps/v8/src/objects.h might need to change too.
|

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.