Send a lot of SQS messages in a lambda without getting timed out

Question

I have an array, lets say with a length of 9999. Now I want to send a message to SQS with the contents of each object in the array.

Here is how I want to send the messages:

const promises = tokens.map((token) => { const params = { Body: JSON.stringify(data), }; return sqs.sendMessage({ MessageBody: JSON.stringify(params), QueueUrl: 'my-queue-url', }).promise(); }); await Promise.all(promises);

Now, the problem is that I trigger this function via API Gateway, which times out after 30 seconds. How can I avoid this? Is there a better way to do this?

Do you need a synchronous response, or are you okay if you know the system is working on it? — Maurice
– Maurice, Commented Mar 17, 2022 at 15:12
docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/… — Mark B
– Mark B, Commented Mar 17, 2022 at 15:13
@Maurice Nope. Don‘t need a response. Just need to trigger it — Ic3m4n
– Ic3m4n, Commented Mar 17, 2022 at 15:19

Maurice · Accepted Answer · 2022-03-17 15:26:04Z

Based on your response in the comments I suggest you do two things:

Use the SendMessageBatch API to reduce the number of API calls and speed up the process as @Marc B suggests
Set up asynchronous invocation of the backend Lambda function. This means the API Gateway will just send the Event to your Lambda and not wait for the response. That means the Lambda functions is free to run up to 15 minutes. If you have more messages than can be handled in that period of time, you may want to look into StepFunctions.

Collectives™ on Stack Overflow

Send a lot of SQS messages in a lambda without getting timed out

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related