4

I have an API endpoint on AWS API Gateway with AWS Lambda (Python & Flask) to store some data from a JSON file. e.g) curl -X POST http://www.xxx.yyy/store -d @zzz.json

However, when I tried executing the API with a bigger JSON file, I encountered a timeout error. Through my investigation, the maximum timeout setting for Lambda is 300 seconds, and API Gateway is 29 seconds. The maximum timeout for Lambda 300 sec sounds fine, but 29 seconds sounds too short. What kind of things could be a solution? The JSON data can be split by id, but it needs to be sent as one file.

EDIT: Sure I can't change the number. Any suggestion to solve this problem using another technology/system design pattern? I can't change the input, though.

EDIT 2: Currently, the Lambda function has validation based on JSON scheme, parse into models, and save into database. Any suggestions?

Update - June 4, 2024

AWS has removed the 29 second API Gateway limit:
Amazon API Gateway integration timeout limit increase beyond 29 seconds

Improve this question
1
  • For anyone else that is facing this issue. I've written a blog post about how you can overcome API Gateway integration timeouts by switching to an asynchronous setup. It lists a few potential solutions and includes some sample code using typescript lambda's and WebSocket. You can find it over here. Commented May 1, 2021 at 14:15

3 Answers 3

Reset to default
2

Is there anyway you can update your Lambda function to hand off to another process?

By decoupling you could for example do the following:

API Gateway -> Lambda (Perform any mandatory action, then store in S3 as a blob) -> S3 -> Another Lambda to process.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Interesting. I was actually thinking of an async approach (I am not sure how I can achieve with AWS Lambda as I am an AWS beginner) but it looks much easier to implement. Thanks.
I am a massive advocate for decoupled architectures, so if you can make it work that would be awesome. Theres many good articles and videos that discuss how people use services :). Good luck
2

Uploading files with lambdas can be tricky and a direct upload is not recommended unless the file size is under the limits.

Warning currently:

  • API Gateway has a payload limit of 10 MB
  • API Gateway has Maximum timeout of 30 s
  • Lambda has an invocation payload (request and response) limit of 6 MB

The best approach is a basically a two step process:

  1. The client app makes an HTTP request to the lambda to get an upload URL. The lambda returns a pre-signed POST URL to S3
  2. The client post the file using the pre-signed URL

API Gateway limits : https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html Lambda limits: https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html

Improve this answer

2 Comments

Thank you for the information. Does the payload include the size of an object in S3 if another lambda function for reading the JSON file and processing it is triggered by S3 when the file is saved?
I found it stackoverflow.com/questions/48536024/…
1

The timeout value cannot be increased:

Resource or operation: Integration timeout

Default quota: 50 milliseconds - 29 seconds for all integration types, including Lambda, Lambda proxy, HTTP, HTTP proxy, and AWS integrations.

Can be increased: Not for the lower or upper bounds.

Source: https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.