read file stream line by line synchronously

Question

I'm looking at nodejs readline module documentation for a task where I've to read a very large file line by line and it looks good. But for my particular task, I need it to read lines synchronously ie. no matter what, line 5 must not be read before line 4, and due to nature of node, I just want to confirm that is this code safe for that usage -

const readline = require('readline'); const fs = require('fs'); const rl = readline.createInterface({ input: fs.createReadStream('sample.txt') }); rl.on('line', (line) => { console.log(`Line from file: ${line}`); });

If not, what should I use/do? Currently it is working for me but I don't know if it'll work with large lines where next line could be parsed faster than previous one etc..

I tried this and everything seems to work out nicely. Try the same thing on your end and see if you can use var sleep = require('sleep'); and replace the sleep function with sleep.msleep(500 * Math.random());. Perhaps pipe the output of your program to a second file and diff them. — Winestone
– Winestone, Commented Apr 21, 2017 at 10:08

Lazyexpert · Accepted Answer · 2017-04-21 10:39:06Z

I doubt very much that it is possible, that the callback fired later can be executed earlier than another one. Basically, it refers to the event loop and stack of the process.

Still, to guarantee I can suggest to implement something similar to async/queue, but with ability to dynamically push callbacks.

Assuming you will have something like this:

const Queue = require('./my-queue') const queue = new Queue() function addLineToQueue(line) { queue.push(function() { // do some job with line console.log(`Line: "${line}" was successfully processed!`) }) }

You will modify your code:

rl.on('line', (line) => { addLineToQueue(line) console.log(`Added line to queue: ${line}`) })

And sure your queue implementation should start as far as it has any tasks to execute. This way the order of callbacks will be guaranteed. But as for me it looks like a little overhead.

but if line 4 was extracted before line 3, the order will still be 1 2 4 3 inside your queue.
No its impossible, the lines will be extracted directly one by one. The only way they can be shuffled is - some reason very long synchronous callback execution, and next line pumping. This is what for I suggest pushing them to queue. Inside queue implementation can launch tasks in child process, so we never get blocking moment and shuffling.

Collectives™ on Stack Overflow

read file stream line by line synchronously

1 Answer 1

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Related