Linked Questions
29 questions linked to/from Make several requests to an API that can only handle 20 request a minute
9 votes
2 answers
9k views
Is there a limit to how many promises can or should run concurrently?
Surprisingly google had trouble returning the result for this question. I'm wondering how many promises can or should be ran in parallel before queuing them and waiting for the next one to finish. I ...
7 votes
1 answer
9k views
Promise.all consumes all my RAM
I've got a rate limiter for an API I am using which allows 20 requests per second. All requests are promise based and the promise will be resolved with the API data once there is a response. The ...
1 vote
1 answer
4k views
unable to complete promises due to out of memory
I have a script to scrape ~1000 webpages. I'm using Promise.all to fire them together, and it returns when all pages are done: Promise.all(urls.map(url => scrap(url))) .then(results => ...
0 votes
1 answer
8k views
Error: connect ETIMEDOUT when scraping
I have a function that: 1. gets an array of 3000 'id' properties from mongoDB documents from collection foo. 2. Creates a GET request for each ID to get 'resp' obj for id, and stores it in another ...
1 vote
1 answer
6k views
How to download files with node-fetch
I need help implementing a file downloader in nodejs. So i need to download over 25'000 files from a server. Im using node-fetch but i don't exactly know how to implement this. I tried using Promise....
2 votes
1 answer
1k views
Properly batch nested promises in Node
I'm running a knex seed in Node and need to batch an additional query to my database due to restrictions on my server. I'm starting to get the hang of promises and async/await, but I'm having trouble ...
1 vote
2 answers
4k views
NodeJS Heap out of memory with long process with database access
I'm building a NodeJs App using Express 4 + Sequelize + a Postgresql database. I'm using Node v8.11.3. I wrote a script to load data into my database from a JSON file. I tested the script with a ...
0 votes
1 answer
2k views
Fastest way to make a million POST requests to a cloud function?
I have an array with a length of one million. Each element is a string. I have a cloud function that takes a string and processes it. What is the fastest way to POST all million strings in my array to ...
2 votes
1 answer
2k views
Running a promise.all of promise.alls
I have am making 400ish requests to a server - and putting each inside promises. When running all 400 requests in a single promise.all - the system falls over. I've split my requests into batches of ...
1 vote
1 answer
2k views
Javascript Promise.allSettled with large amount number of requests
I have a bulk create participants function that using Promise.allSettled to send 100 axios POST request. The backend is Express and frontend is React. That request is call a single add new participant ...
2 votes
2 answers
1k views
How to loop many http requests with axios in node.js
I have an array of users where each user has an IP address. I have an API that I send an IP as a request and it returns a county code that belongs to this IP. In order to get a country code to each ...
1 vote
1 answer
3k views
Who nodejs http request fails when there are too much tasks?
I should login and call http some api points by nodejs, var mins = 150; var url = require('url'); const domainim= 'example.com'; const request = require('requestretry').defaults({ jar : true, ...
1 vote
2 answers
2k views
Loop through an api get request with variable URL
I am trying to call CompaniesHouse API and fetch companies registered between November and February. The approach I took is to pick a starting index(a company registered in November) and a stop index(...
0 votes
4 answers
984 views
How to download huge list of remote files using Node JS http get method without running into errors
I'm trying to download a list of files generated by an internal processing system via HTTP get method in node js. For a single files or for a few files it works fine and there is an answer for that ...
0 votes
2 answers
1k views
node and socket.io multiple API calls hang after a while
I have a node express/ socket application where my express server makes several API calls with node-rest-client looping through elements in var jobs and when each finishes, it sends the data via ...