Skip to main content

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

4
  • Thank you so much for the detailed response. The timing really also depends on which machine I ran. I was planning to use other AWS service to do data pre-processing - we are looking at maybe EMR, Glue, or another lambda function. After the pre-processing, the data will be sent to S3, the clean data itself can still be around 1gb. I know it is supposed to be serverless, but Is it possible to change the computing power for lambda? Or tune the settings? Would you say the step functions approach will encounter multiple "cold-start" issue? Thanks again!! Commented Apr 8, 2020 at 22:58
  • 1. Compute power: yes. The docs have more info, but you can adjust the memory allocated to the lambda. This has the effect of also increasing how much CPU is given to the lambda, as well. Once you max out that, though, the way to scale is to parallelize between multiple lambdas. Keep in mind: if you can, having a ton of lambdas with each processing a tiny amount is ok. For example, a common pattern is to trigger one lambda per file write, got many files. Commented Apr 9, 2020 at 0:21
  • 2. Step functions / cold start: no. Cold start is a concern when lambdas back a user-facing api, so the lambda execution delay can be noticeable to the user. This is not your use case. Even if the cold start were relevant, it will be dominated by a lambda that runs even a moderate workload Commented Apr 9, 2020 at 0:23
  • Great!! Thanks again for your help! I'll start looking into the step-function and try it out. :) Commented Apr 9, 2020 at 1:15