Synchronous execution for node.js program using 'readline'

Question

I'm having some problems getting the asynchronous nature of node to co-operate with me, and after hours of callbacks and googling; I finally turn to you guys.

I have a program that needs to read in lines from a file using the readline module of node. This file contains data that is passed to some asynchronous functions defined within my node program. Once all the data is successfully read and processed, this data needs to be parsed into JSON format, and then outputted.

My problem here is that when I call: readLine.on('close', function() { ...... }, this is run before the asynchronous functions finish running, and therefore I am left with an output of nothing, but the program keeps running the asynchronous functions.

I've created a simple skeleton of functions that should explain my situation more clearly:

function firstAsyncFunc(dataFromFile) { //do something asynchronously return processedData; } function secondAsyncFunc(dataFromFile) { //do something else asynchronously return processedData; } //create readline var lineReader = require('readline').createInterface({ input: require('fs').createReadStream('data.txt') }); //array to hold all the data processed var totalDataStorage; //read file lineReader.on('line', function(line) { var processedData = firstAsyncFunction(line); var moreProcessedData = secondAsyncFunction(line); //store processed data and concatenate into one array var tempDataStorage = [{ 'first': processedData, 'second': moreProcessedData }] totalDataStorage = totalDataStorage.concat(tempDataStorage); }).on('close', function() { var JSONString = JSON.stringify(... //create JSON for totalDataStorage ...); console.log(JSONString); //DOESN'T OUTPUT ANYTHING! });

I have tried to add a callback to the first/secondAsynFunction, I have tried to make the reading and parsing bit of the program seperate functions, and create callbacks so that parsing is only called when reading finished, but none of those solutions seemed to be working and i'm really struggling - so any help would be appreciated.

Thanks!

EDIT: The data.txt file is of the form

IPData1 DataCenter1 IPData2 DataCenter2 ... IPDataN DataCenterN

I use str.split(" ") to get the respective values, and then pass them appropriately. IPData is a number, and DataCenter is a string

@JaromandaX See again OP's code. It is an array, not string. — user6586783
– user6586783, Commented Aug 31, 2016 at 23:55
The data.txt file is of the form: IPData, DataCentre. In the actual program, I use str.split(" "), to split the two values, and then pass them into the necessary function. The IPData is a number, and the DataCentre value is a string. Hope this helps — OliverOstach
– OliverOstach, Commented Aug 31, 2016 at 23:56
@OliverOstach You have a lot of mistakes in your code. You need to realize that every time you call an asynchronous function, it doesn't return a value and you need to pass a callback function too. Your line var processedData = firstAsyncFunction(line); makes no sense at all. Also, it can be done very easily in a few rows using fs.readFile. — user6586783
– user6586783, Commented Sep 1, 2016 at 0:05
Ah I forgot to mention that the async functions return values! String values to be precise! I'll edit that in now — OliverOstach
– OliverOstach, Commented Sep 1, 2016 at 0:10

score 1 · Accepted Answer · 2016-09-01 00:21:48Z

Asynchronous functions do not return a value, but you must pass a callback function to it instead. Your line

var processedData = firstAsyncFunction(line);

doesn't make sense at all. If your data.txt file looks like this

IPData1 DataCenter1 IPData2 DataCenter2 IPData3 DataCenter3

you can read data as following

var fs = require('fs'); var rl = require('readline').createInterface({ input: fs.createReadStream('data.txt') }); var arr = []; rl.on('line', a => { a = a.split(' '); arr.push({ first: a[0], second: a[1] }); }).on('close', () => { console.log(JSON.stringify(arr, null, 2)); });

It will log

[ { "first": "IPData1", "second": "DataCenter1" }, { "first": "IPData2", "second": "DataCenter2" }, { "first": "IPData3", "second": "DataCenter3" } ]

Thank you for your help, but unfortunately this doesn't work. I've registered a callback function to 'return' results from the async functions, and then called them appropriately from within the lineReader.on('line', ...), but I am still having the same issue of .on('close'), running BEFORE the async functions have finished and returned the result.. (this was my issue, I've been able to verify that i'm doing everything else correct!)
@OliverOstach You simply can't understand how asynchronous functions work. By definition, asynchronous function CANNOT return a value, so stop calling it asynchronous. If it returns a value, then it MUST run before onClose event is fired, because it is the way v8 engine works. How do you think my answer doesn't work? Did you try it? Copy-paste it to ensure it works properly.

Bamieh · Accepted Answer · 2016-09-01 00:30:24Z

I changed the following and its working locally.

use promises to make your life easier.
remove .close since you dont have an output defined in the interface.

The 'close' event is emitted when one of the following occur:

The rl.close() method is called and the readline.Interface instance has relinquished control over the input and output streams;

The input stream receives its 'end' event;

The input stream receives -D to signal end-of-transmission (EOT);

The input stream receives -C to signal SIGINT and there is no SIGINT event listener registered on the readline.Interface instance.

function firstAsyncFunc(dataFromFile) { return new Promise(function(resolve, reject) { //do something asynchronously resolve(result); }) } function secondAsyncFunc(dataFromFile) { return new Promise(function(resolve, reject) { //do something asynchronously resolve(result); }) } //create readline var lineReader = require('readline').createInterface({ input: require('fs').createReadStream('data.txt') }); //array to hold all the data processed var totalDataStorage; //read file lineReader.on('line', function(line) { Promise.all([ firstAsyncFunc(line), secondAsyncFunc(line) ]) .then(function(results) { var tempDataStorage = [{ 'first': results[0], 'second': results[1] }]; // i'd use push instead of concat totalDataStorage = totalDataStorage.concat(tempDataStorage); }); })

This seems like it could work, but I just can't seem to figure out where I would log the data such that all the asynchnorous functions would have been executed by the logging stage

Collectives™ on Stack Overflow

Synchronous execution for node.js program using 'readline'

2 Answers 2

2 Comments

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Related