Get all message from aws sqs

Question

I have a ScheduledEvent on my lamda function for every 24 hours and then inside function, I am calling SQS to get my messages.

export class EmailNotificationProcessor { public static async run(): Promise<void> { console.log('event'); await this.getNotificationFromSqs(); } private static async getNotificationFromSqs(): Promise<void> { const messagesToDelete: DeleteMessageBatchRequestEntryList = []; const messageRequest: ReceiveMessageRequest = { QueueUrl: process.env.DID_NOTIFICATION_SQS_QUEUE, MaxNumberOfMessages:10, WaitTimeSeconds:20 } const { Messages }: ReceiveMessageResult = await receiveMessage(messageRequest); console.log('Messages', Messages); console.log('Total Messages ', Messages.length); if (Messages && Messages.length > 0) { for (const message of Messages) { console.log('body is ', message.Body); messagesToDelete.push({ Id: message.MessageId, ReceiptHandle: message.ReceiptHandle, } as DeleteMessageBatchRequestEntry); } } await deleteMessages(messagesToDelete); } }

I am expecting 1 to 30 messages inside my queue and want to process all messages before sending an email which consists of the content that I will parse from sqs body.

My function for receiving messages

export const receiveMessage = async (request: SQS.ReceiveMessageRequest): Promise<PromiseResult<SQS.ReceiveMessageResult, AWSError>> =>{ console.log('inside receive'); return sqs.receiveMessage(request).promise(); }

Now I am not able to receive all messages at once and only getting 3 messages or sometimes 1 message at a time.

I know limit for API call is 10 in one single request but is there any way to wait and get all your message.

You haven't included a question, or a description of a problem. Your post is just a statement about what you are currently doing, without a description of what you need help with. — Mark B
– Mark B, Commented Dec 20, 2020 at 16:09
@MarkB please read it again i just updated the question and thanks for pointing it out. — Mike
– Mike, Commented Dec 20, 2020 at 16:20
There's no way to get more than 10 messages in a single API call, and no guarantee you will even get 10. You need to wrap the call to SQS in a loop and keep requesting more messages until the queue is empty. — Mark B
– Mark B, Commented Dec 20, 2020 at 16:23

Abdul Moeez · Accepted Answer · 2020-12-21 21:42:43Z

First of all, There is no configuration to get more then 10 messages from queue.

ReceiveMessage: Retrieves one or more messages (up to 10), from the specified queue

For your other problems: I think you are using Short poll ReceiveMessage call.If the number of messages in the queue is extremely small, you might not receive any messages in a particular ReceiveMessage response.

Try Long Polling: Long polling helps reduce the cost of using Amazon SQS by eliminating the number of empty responses (when there are no messages available for a ReceiveMessage request) and false empty responses (when messages are available but aren't included in a response).

Note* For getting more messages you need to wrap the call to SQS in a loop and keep requesting more messages until the queue is empty, which can lead you to get duplicate messages as well so try VisibilityTimeout in that problem.

Try VisibilityTimeout: The duration (in seconds) that the received messages are hidden from subsequent retrieve requests after being retrieved by a ReceiveMessage request.

Sample Wrap up SQS call code:

function getMessages(params, count = 0, callback) { let allMessages = []; sqs.receiveMessage(params, function (err, data) { if (err || (data && !data.Messages || data.Messages.length <= 0)) { if(++count >= config.SQSRetries ){ return callback(null, allMessages); } return setTimeout(() => { return getMessages(params, count, callback); }, 500); } else if (++count !== config.SQSRetries ){ allMessages.push(data); return setTimeout(() => { return getMessages(params, count, callback); }, 500); } else { allMessages.push(data); callback(null, allMessages); } });

In config.SQSRetries, we have set the value according to our requirement but as your SQS have 1 to 30 messages, Then '7' will be good for you!

Links: RecieveMessage UserGuide

it would be great to add some code in your answer to show how i can wrap call to sqs and i am using long pooling please look again at my question
The answer is updated with sample code as you mentioned and Sorry but I have no grip on promises that's why i have to give you overview in callback's.

Collectives™ on Stack Overflow

Get all message from aws sqs

1 Answer 1

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Related