how to iterate two arrays one after other using async forEach

Question

I have two arrays. I would like to iterate the arrays using async.foreach. But when I do so, only the second array is getting executed.How to execute both. Below is my code:

var _sizesArray1 = [_2000px, _1400px] var _sizesArray2 = [_800px, _400px, _200px, _100px] async.forEachOf(_sizesArray1, function(value, key, callback) { async.waterfall([ function download(next){ //code }, function convert(response, next) { //code }, function process(response, next) { gm(response).command('convert') .resize(_sizesArray[key].width,_sizesArray[key].width) .gravity('Center') .extent(_sizesArray[key].width,_sizesArray[key].width) .quality('20') .toBuffer( 'JPG', function(err, buffer) { if (err) { next(err); } else { console.timeEnd( "processImage array1" ); next(null, buffer, key); } }); } }); async.forEachOf(_sizesArray2, function(value, key, callback) { async.waterfall([ function download1(next){ //code }, function convert2(response, next) { //code }, function process3(response, next) { //code } });

In my code, only array2 is getting invoked.Why don't first one get executed. Is there any mistake in my code. Can somebody help resolve this.

Are you using _sizesArray2 in both loops or is it copying mistake? — Sagar Chaudhary
– Sagar Chaudhary, Commented Oct 11, 2018 at 15:05
I wrongly updated code here, i am using _sizesArray1 & _sizesArray2. — merla
– merla, Commented Oct 11, 2018 at 15:33
I did get this working upto an extent by changing the function names within both the arrays. but the application is sometimes processing all the dimensions in both arrays, but sometimes it is just picking few dimensions randomly. Is this due to buffer limit. how can we clean up the buffer after iterating through first array. — merla
– merla, Commented Oct 11, 2018 at 17:46

Gabriel C. Troia · Accepted Answer · 2018-10-11 18:47:43Z

How about this simple technique:

var allSizes = _sizesArray1.concat(_sizesArray2); async.foreach(allSizes, function(value, key, next) { // do whatever you like with each item in the concatenated arrays // once you are done move to the next item next() })

Updated based on comments

Version 1, based on callbacks (welcome callback hell):

function asyncIterate(array, onEach, onDone) { async.forEach(array, (val, index, onNext) => { onEach(val, key, function onDecoratedNext() { // tell the async.forEach it's ready for the next item onNext(); // but if it got to the end, // then mark the completion of the whole iteration if (index === array.length) { onDone(); } }); }); }

and implement it like:

function doAsyncStuffWithEachItem(val, index, cb) { // do async stuff with each item // make sure to mark the completion of the async operation cb(); } asyncIterate( _sizesArray1, doAsyncStuffWithEachItem, function onDone() { // once the 1st iteration finished, time to move to the 2nd array asyncIterate( _sizesArray2, doAsyncStuffWithEachItem, function onDone2() { // great we are done // What if we have _sizesArray3, 4, 5 ... ? // Well, we can continue to nest more callback here // but in that way we'll soon end up with callback hell // and that's a big NoNo } ) } );

Version 2, based on Promises:

To avoid callback hell, luckily we can use Promises. Something like this should do it:

const promisifiedAsyncIterate = (array, onEach) => new Promise((resolve) => { asyncIterate(array, onEach, resolve); });

and use it like:

promisifiedAsyncIterate(_sizeArray1, doAsyncStuffWithEachItem) .then(() => promisifiedAsyncIterate(_sizeArray2, doAsyncStuffWithEachItem)) .then(() => promisifiedAsyncIterate(_sizeArray3, doAsyncStuffWithEachItem)) .then(() => promisifiedAsyncIterate(_sizeArray4, doAsyncStuffWithEachItem))

It could be abstracted and cleaned up even more, or even made completely dynamic – say you have an array of sizeArrays that you pass in to your function, but I think this is enough for now :). Hope it helps.

My issue was initially to overcome the buffer limit error in aws lambda. If I have all the dimensions in single array while processing large files, it is exceeding memory limit. hence I divided it into arrays and iterating them one after the other. so now I need to clear/dispose the buffer after processing first array
Interesting. Hmm, then the only way I see this working is by deferring the iteration of the second array until the first one finishes. Let me think of a clean solution and I'll update the answer asap.
@merla see the updated version, and let me know if you need any clarification or if you have any problems with it.

Collectives™ on Stack Overflow

how to iterate two arrays one after other using async forEach

1 Answer 1

3 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Linked

Related