How can the AWS Lambda concurrent execution limit be reached?

UPDATE

The original test code below is largely correct, but in NodeJS the various AWS services should be setup a bit differently as per the SDK link provided by @Michael-sqlbot

// manager const AWS = require("aws-sdk") const https = require('https'); const agent = new https.Agent({ maxSockets: 498 // workers hit this level; expect plus 1 for the manager instance }); const lambda = new AWS.Lambda({ apiVersion: '2015-03-31', region: 'us-east-2', // Initial concurrency burst limit = 500 httpOptions: { // <--- replace the default of 50 (https) by agent: agent // <--- plugging the modified Agent into the service } }) // NOW begin the manager handler code 

In planning for a new service, I am doing some preliminary stress testing. After reading about the 1,000 concurrent execution limit per account and the initial burst rate (which in us-east-2 is 500), I was expecting to achieve at least the 500 burst concurrent executions right away. The screenshot below of CloudWatch's Lambda metric shows otherwise. I cannot get past 51 concurrent executions no matter what mix of parameters I try. Here's the test code:

// worker exports.handler = async (event) => { // declare sleep promise const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); // return after one second let nStart = new Date().getTime() await sleep(1000) return new Date().getTime() - nStart; // report the exact ms the sleep actually took }; // manager exports.handler = async(event) => { const invokeWorker = async() => { try { let lambda = new AWS.Lambda() // NO! DO NOT DO THIS, SEE UPDATE ABOVE var params = { FunctionName: "worker-function", InvocationType: "RequestResponse", LogType: "None" }; return await lambda.invoke(params).promise() } catch (error) { console.log(error) } }; try { let nStart = new Date().getTime() let aPromises = [] // invoke workers for (var i = 1; i <= 3000; i++) { aPromises.push(invokeWorker()) } // record time to complete spawning let nSpawnMs = new Date().getTime() - nStart // wait for the workers to ALL return let aResponses = await Promise.all(aPromises) // sum all the actual sleep times const reducer = (accumulator, response) => { return accumulator + parseInt(response.Payload) }; let nTotalWorkMs = aResponses.reduce(reducer, 0) // show me let nTotalET = new Date().getTime() - nStart return { jobsCount: aResponses.length, spawnCompletionMs: nSpawnMs, spawnCompletionPct: `${Math.floor(nSpawnMs / nTotalET * 10000) / 100}%`, totalElapsedMs: nTotalET, totalWorkMs: nTotalWorkMs, parallelRatio: Math.floor(nTotalET / nTotalWorkMs * 1000) / 1000 } } catch (error) { console.log(error) } }; Response: { "jobsCount": 3000, "spawnCompletionMs": 1879, "spawnCompletionPct": "2.91%", "totalElapsedMs": 64546, "totalWorkMs": 3004205, "parallelRatio": 0.021 } Request ID: "43f31584-238e-4af9-9c5d-95ccab22ae84" 

Am I hitting a different limit that I have not mentioned? Is there a flaw in my test code? I was attempting to hit the limit here with 3,000 workers, but there was NO throttling encountered, which I guess is due to the Asynchronous invocation retry behaviour.

Edit: There is no VPC involved on either Lambda; the setting in the select input is "No VPC".

Edit: Showing Cloudwatch before and after the fix