Download to read offline
Node Summit 2018 - Optimize your Lambda functions
- 1. Optimize your Lambda functions Fastto execute and to debug Matt Lavin @mdlavin LifeOmic @LifeOmicfrom
- 2. Our goals Icons madeby Freepik from www.flaticon.com licensed by CC 3.0 BY 1. Make worst latency better 1. Make average latency great 1. Find performance bugs quickly 1. Make regular debugging easy
- 3.
- 4.
- 5. Node starts 100xfaster than Java and C# Node 8 does not start faster than Node 6
- 6. 10x faster start-upby avoiding VPCs
- 7.
- 8.
- 9. Move code outof handler function expensiveComputation() { // Something that takes a lot of time } // Code here executes once for each Lambda runtime exports.handler = function (event, context, callback) { // Code here executes on every request const value = expensiveComputation(); callback(null, value); }
- 10. Reuse what youcan ● Cached data ● Configuration data ● Database connection pools ● AWS SDK Clients
- 11. Make DynamoDB useHTTP Keep-Alive const AWS = require('aws-sdk'); const https = require('https'); const sslAgent = new https.Agent({ keepAlive: true, // This is new maxSockets: 50, // From the aws-sdk source code rejectUnauthorized: true // From the aws-sdk source code }); sslAgent.setMaxListeners(0); // From the aws-sdk source code
- 12. Photo by FrancescoCrippa / CC BY The service is fast because of developers
- 13. Visualize performance withAWS X-Ray
- 14. Enable X-Ray: UpdateLambda configuration ● Enable ‘active tracing’ mode for your functions ● Update IAM Role for xray:* write actions
- 15. Enable X-Ray: Hookfunctions const AWSXRay = require('aws-xray-sdk-core'); AWSXRay.captureHTTPsGlobal(require('http')); AWSXRay.captureAWS(require('aws-sdk')); AWSXRay.capturePromise(); AWSXRay.setLogger(logger.child({'xray': true}));
- 16. Power-up X-Ray withcustom segments
- 17. Adding custom segmentsare easy const AWSXRay = require('aws-xray-sdk-core'); AWSXRay.captureAsyncFunc('Your custom segment', function (subsegment) { someCodeToExecute() .then(function () { subsegment.close(); }, function (error) { subsegment.close(error); }); });
- 18. Power-up X-Ray withannotations
- 19. Annotations are easyto add const AWSXRay = require('aws-xray-sdk-core'); const segment = AWSXRay.getSegment(); segment.addAnnotation('account', account); segment.addAnnotation('hostname', os.hostname());
- 20. The service isfunctioning because of developers Photo by Francesco Crippa / CC BY
- 21.
- 22. Use the requestIDs in your logs exports.handler = function (event, context, callback) { // Get the Request ID const requestId = context.awsRequestId; // Use it in your application logs const logger = bunyon.createLogger({"requestId": requestId}); logger.info("Some interesting output"); callback(null, value); }
- 23. Add a correlationID Request ID: RID-A Correlation ID: CID-1 Service A Request ID: RID-B Correlation ID: CID-1 Service B Request ID: RID-C Correlation ID: CID-1 Service C Correlation-ID: CID-1 Correlation-ID: CID-1
- 24. Size doesn’t matter(much) Smaller does use less memory Smaller does not start faster
- 26. Find surprises withwebpack-bundle-analyzer
- 27. Quick tips forWebpacking ● Exclude built-in package with externals ● Shrink your package with MinifyPlugin ● Enable source-maps
- 28. Icons made byFreepik from www.flaticon.com licensed by CC 3.0 BY
Editor's Notes
- #2 Hi. I’m Matt Lavin and I’ve been a part of a team that has spent the last year and a half building a nearly serverless backend for the APIs at LifeOmic. LifeOmic is building a service to help doctors and researchers understand their patient data, we’re building a mobile app to help people build healthy habits, and we’re building a security product to help other companies maintain compliance. But, what you really care about is that we’ve been doing it all almost exclusively with Node.js and AWS Lambda functions.
- #3 I’d like to share what we’ve learned with the goal of making two types of people happy. First, I’ll show you some tips to make your users happy. The improvements I’ll go through are based around understanding how the Lambda execution environment works and changing your app to fit into the framework nicely. Since the AWS Lambda service takes care of scaling your application automatically, we can focus on improving the response time of individual requests. After your users are happy, we can spend some time trying to make our lives as developers easier. Again, the advice is to understand the AWS service ecosystem and play nicely with it so that you can leverage the tools that the AWS team is constantly working to improve.
- #4 Before digging into the details, it’s important to understand the basic execution lifecycle of a function. When you create your Lambda function you tell AWS what runtime you want along with IAM and networking requirements. So, when a request to your deployed Lambda arrives AWS needs to setup up a container with those requirements. When the container is up, the first thing it does is fetch your function code. Next, your Lambda function code is loaded into memory. In Node.js functions your module is loaded using the standard `require` function just like any other module. Finally, your Lambda function handler is executed. This can happen many times for a single Lambda container instance, but only a single request is processed at a time. With this understanding, especially the part about multiple invocations, there are tricks to improve your execution performance in each of these three steps.
- #5 If you look at a graph of a lightly used AWS function you’ll immediately notice something like this. Most of your execution time is fast, but every once and a while you’ll see a spike in request latency. This often happens because AWS is launching a new instance. This spike is commonly referred to as “cold start” time and it’s possible to reduce it with some changes. Getting rid of these spikes means getting rid of user frustration when the app is slow to respond. One common trick that I’ll mention with caution is a warmer. A warmer is a scheduled API execution that hits your service often enough to keep AWS from shutting down your instance. This trick does give a nice improvement for lightly used functions, but that’s just avoiding the problem. In a highly used service, AWS will scale your number of instances up and down based on load so you will not be immune to the cost of Lambda ‘cold starts’. Better to understand and improve it as much as possible.
- #6 I’ll start with the easiest piece of advice for this crowd and also the best way to create fast startup. Use Node! If you are here at NodeSummit to learn if Node is a good choice for your next application you can leave knowing the answer is ‘yes’. If I’m being honest, Python also starts equally fast in AWS. But, we’re here at NodeSummit so let’s keep it simple and say ‘Use Node’ I was going to put a chart here showing the difference between Node and other languages but it’s not a very interesting picture. Java and C# have a huge amount of startup time, in the range of seconds, and Node and Python both start very fast, in the range of milliseconds. Since the recent release of Node 8 support in Lambda I was curious if Node 8 started faster than Node 6. I ran some tests and I did not detect any noticable difference between the two runtime versions in startup. It’s still a good idea to use Node 8, and Node 8 might be faster after startup, but it wasn’t dramatically faster to start than Node 6 in my tests. The data on startup times comes from an awesome blog post and open source project from Yan Cui titled ‘How does language, memory and package size affect cold starts of AWS Lambda’. Google it for more details and to reproduce the results https://read.acloud.guru/does-coding-language-memory-or-package-size-affect-cold-starts-of-aws-lambda-a15e26d12c76
- #7 The second piece of advice for fast startups is by avoiding VPC network connectivity. This is an optimization in the ‘Container initialization’ phase from my lifecycle slide and it was not obvious that it would have such a big impact. If you’re like me and used AWS for a while, you’ve developed a habit of putting everything inside a VPC. Since Lambda configurations allow for a VPC to be specified it was an automatic response to fill in the VPC values. Do not do that without pausing to think! Setting a VPC configuration does not change the location of where your Lambda code is executed, it only adds an additional network link to your VPC. That additional network link is more expensive to setup than you might expect, consistently in the range of seconds and often as high as ten seconds. The cause seems to be the cost of attaching a new ENI to your Lambda instance. I have to imagine that AWS is working to reduce this time, but for now it’s best to avoid if possible. If you are using the common Lambda / DynamoDB pair then you should not need a VPC connection since DyanmoDB is publicly accessible. If you are using RDS or ElastiCache, having a VPC connection is the easiest approach, but you could consider exposing your DB/Cache instance to the public (with appropriate security in place) if you are desperate for dropping the VPC connection. Showed ~8-10 seconds to setup networking https://medium.freecodecamp.org/lambda-vpc-cold-starts-a-latency-killer-5408323278dd
- #8 Increasing the amount of memory for your Lambda function is another configuration tweak that is not immediately obvious, but it can have a huge impact on your startup time. Lambda functions are billed in Gigabyte-seconds, which means that less memory requested will mean less cost. Since everybody wants to reduce cost, the default behavior seems to be for people to reduce the memory to the lowest needed. However, reducing memory also reduces the CPU speed that you get . The reduction in performance can mean increasing execution times that almost cancel out your cost savings. From the results I’ve seen, it seems like the sweet spot is between 1G to 1.5G. Less memory than that slows down the function enough to keep the cost about the same and more memory than has diminishing returns and your cost will increase. Like everything performance related, your mileage will vary and you should play with different memory sizes with an eye on how it changes your execution time.
- #9 Ok, the last couple of slides were almost exclusively configuration tweaks and not particularly node-y. We’ll switch to lowering the average response time, which is controlled mostly by the time it takes for a already warm Lambda function to execute. Here we get into specific coding advice rather than just configuration.
- #10 As I mentioned in the Lambda lifecycle slide, Lambda function handlers are executed more than once on each instance. The next couple slides take advantage of that fact to keep the average execution time fast. If you are following good functional coding practices, you’ll default to keeping your variables scoped inside functions and avoid state and execution in the module scope. But, for improving performance, it’s time to relax that habit a bit. If you have some computation or initialization that can happen outside the handler, it is best to move it out so that it will only execute once. The example here is simple since the function is not an async function. The approach works well for async functions too. You can lazily initialize your long computations and the `.then` executions will be nearly immediate for every execution after the first.
- #11 The last slide showed caching a value of a long computation, but you should consider initializing as much as you can just once. This single initialization approach can be used to cache application and configuration data to avoid reloading between requests. Be careful to protect your caches with access control if they are shared between different user requests. The single initialization approach should also be used for database and AWS clients too. Since the node module stays loaded in memory, connection pools can be used for RDS connections if you share your client between handler executions. It wasn’t obvious to me that connection pools would be possible when initially thinking about the Lambda programming model. As you would expect, avoiding the network connection initialization can lead to a big time savings. One quirk of caching network clients is there is not currently a hook to gracefully terminate your client connections when the Lambda instance is being shut down. I imagine that AWS will address this with some sort of hook eventually, but you’re already designing your application so that it works with random instance terminations (right????) so this is not much different from that.
- #12 Talking about connection pools is the perfect segway into another trick to reduce latency. The AWS SDK does not enable HTTP keep-alive for its network requests. For a low latency service like DynamoDB, the network overhead can become larger than the actual service execution time. What I’ve shown here is the code we use to make sure that Keep-Alive gets enabled for connections in our Lambda functions. The line of importance here is the keepAlive line and the rest of the code is to mirror existing behavior and hook up the custom agent. This change, combined with sharing your AWS client between requests can make a big improvement in request latency.
- #13 When my daughter saw this photo she said “I like it because it’s a bunch of people working together to make a fast car”. That’s a nice reminder that there are people involved. I’ve shown some ways to build a fast Lambda function. Your functions should start fast and run faster than before. But, the ideas I’ve shown were found by humans, implemented by humans and validated by humans. Since we have to admit that there are humans involved in running a successful (fast) service we should spend some time to make sure that developers can be productive when working with performance. Even a fast car, or Lambda function, needs to be constantly worked on to keep it fast and I’m going to show you a couple of things that will make that ongoing maintenance easier
- #14 When it comes to performance improvements, the first step should always be collecting data on where your slowness is coming from. There are many tools that you could use to collect and inspect the performance of your Lambda functions, but I’m going to focus on integration with AWS X-Ray because I know you are using AWS already and I’ve had good luck using it for my own work. There’s a bonus! It’s free and I know that the AWS team is investing in new improvements to make it even better. This is a screenshot of an X-Ray trace for a some function in our environment. At first glance it looks like it might be a trace for the user-service, but it’s really showing that the user-service was called three times. The groups endpoint is called twice, hitting DynamoDB multiple times for each call. This stands out to me as a case where a cache or code refactoring could be explored to reduce the number of calls down to a single fetching of groups. Being able to visualize what’s happening can make finding problems or areas for improvement much easier and getting this level of detail out of your services it not hard.
- #15 The first step to using X-Ray for your functions is to enable it in the Lambda configuration. The AWS Console calls X-Ray ‘active tracing’. After enabling ‘active tracing’, you need to update the IAM role that your Function is using to have access to the X-Ray write operations. While both things can be done with the AWS Console, you know that you should be automating the setup and both actions are easy to do through the AWS CLI, CloudFormation or tools like Terraform.
- #16 Now that your function has X-Ray enabled and the ability to write data, you need to start generating data. The simplest way to generate data is to use the aws-xray-sdk-core module to instrument common patterns in Node.js code. Using captureHTTPsGlobal means that segments will be added to show requests to external services Using captureAWS means that segments will be added for AWS API executions and traces can span across services. This was seen in my screenshot before with one Function calling another and seeing the details inside the downstream call. Using capturePromise means that the X-Ray transaction does not get lost when Promises are used. Setting up the logger makes X-Ray logging consistent with the rest of your application NOTE: For best results when running on Node 8 continue to transpile async/await back to Promises until full node 8 support is available. The X-Ray team is looking at the best way to instrument async methods, but it’s not supported yet.
- #17 The out of the box support for creating segments is useful, but pretty high level. The segments are granual, usually only at the entry and exit points for your functions. That’s helpful to get a big picture view, but sometimes you want to see why a downstream service is being called. Here is a screenshot showing how custom segments can be added for more details. In this case, I’ve added a segment for each GraphQL resolver so that I could see which Dynamo and Lambda requests were associated with each resolver. I’ve open sourced the support for this on GitHub. If you search for “xray graphql resolver” the top hits will point you at the project. GraphQL details are what helped me, but you’ll want to instrument your code to see the details that matter to you.
- #18 Here is an example of how a custom resolver can be added. You’ll use aws-xray-sdk-core again and there are a handful of helper functions in their documentation about capturing segments. I’ve shown how to add a segment that tracks an async function because that’s where most interesting (and time consuming) execution is going to be. The basic pattern is to create a new function scope, execute the function you want to trace inside the scope and let X-Ray know when it’s done. If you’ve enabled Promise tracing with the capture function from the previous slides then X-Ray will pick up the details automatically. You should notice that when the Promise rejects, I close the segment with an Error. Once you start using X-Ray you’ll find that you want to filter on things like “traces with failures” or some other features of your request which leads me to ...
- #19 … filtering all the X-ray data down to what you want. In the AWS Console, you can filter X-Ray traces with a simple query language that includes matching things like request time, errors, timeouts, URLs, users and also custom annotations. Custom annotations can be really helpful in narrowing down traces related to a specific customer complaint. If a single customer complains about slow performance but your system otherwise seems fast, you could use an annotation to filter on an account attribute like I’m showing here. Maybe your customer has generated data with different qualities than others and your code was making some poorly performing assumptions.
- #20 Adding annotations is even easier than adding custom segments. All you need to do is get the current Segment and add the annotations you want to track. If you just created your subsegment from the previous slide then you don’t need to call getSegment at all. Like other data collection systems, you’ll want to find experiment to find what’s useful for your debugging.
- #21 This whole presentation has been working under the assumption that your service is in a state that is worth making faster. If it’s not functioning as you want then it doesn’t matter how fast it is. Just as it’s worth making debugging performance problems easy, it’s also worth making normal debugging easier. We’ll continue the pattern of integrating with the AWS ecosystem, this time to improve our application logs.
- #22 To make application logs more helpful you should consume and take advantage of the AWS Request ID. For API Gateway requests and for Lambda function executions, AWS will automatically create a request ID that will be included for its own request tracking. In the screenshot above you can see a couple headers that are useful to capture and use in your application. If you are collecting data to include in client error reports, these are some very helpful values to capture. The bottom two are easiest to explain. The x-amzn-requestid is the Request ID used for the API Gateway request. If you have API gateway logs, then you can use that for searching for the request. The x-amzn-trace-id value is the X-Ray trace ID that can be used to lookup the associated X-Ray trace for the request. That’s even faster than using annotations to get the trace details from your customers. The top header is interesting when API Gateway is used in front of a Lambda function. The API Gateway will take any x-amzn headers coming from the downstream Lambda function and will append the remapped prefix to avoid collisions with its own headers. This remapped request ID value is the request ID of the Lambda function that was executed and would let you jump straight to the logs for that execution.
- #23 To make finding all application log messages associated with a single Lambda request execution, you can take the approach shown to capture the request ID from the context and add it to each log in your handler processing. I used the Bunyon logger here, but any logging library that supports child loggers with context values would work. The advantage of building on top of the AWS provided request ID instead of creating your own is that the AWS request ID will also be in system messages about the lambda framework start, end, and report messages along with any other system messages. This approach means that filtering on the request id, which I showed was easily available in client headers, will immediately show you both your application and AWS system logs together.
- #24 One trouble with the Request ID is that it’s only scoped to a single service execution. In the slide showing headers you might have already noticed the problem, since there were separate request IDs for both the API Gateway execution and for Lambda function execution. Sometimes filtering on a single service execution is exactly what you want. You don’t need the noise of every service in your request processing to be included. Other times you do want to see the logs for all services together and in that case I’d recommend adding your own Correlation ID header that is a fairly common practice. You’ll have to make sure the correlation ID is created at the start and make sure to pass it through headers in all outgoing service requests and include it in log entries. The combination of both the single service request ID and the full execution correlation ID means you can quickly filter the logs depending on what you need.
- #25 In the summary for the talk it said that I’d talk about improving size, but I’m going to warn you that if your goal is performance then you time is probably spent in other places. When reducing the size of our Lambda functions we did see dramatically less memory used, and we could run the functions with smaller memory requirements. The thought was that it would keep costs lower, but as I mentioned before the loss of CPU at the lower memory settings made it not useful for us. I also hoped that smaller code sizes would keep cold starts faster. I have not seen any improvements in start times. I would have expected less code means faster code downloads or less reading from the disk, but it doesn’t doesn’t make a noticeable impact to cold start times.
- #26 I’ll be honest, even though it doesn’t help performance there is just something that feels good about reducing code size. Webpack has been used reduce code size on the web for years and can use it to reduce the size of your Lambda code too. There are a million plugins to help reduce code size and a massive amount of time put into the project. There is good babel integration to allow using the latest Javascript features before AWS releases support for the latest Node version. My favorite advantage of using webpack is the ability to use webpack-bundle-analyzer
- #27 One of those things that makes reducing code size feel good is getting rid of a dependency that you don’t need. If you have to justify the effort to others, and performance justification is a weak one, you can say that you’re reducing your source of security problems. The Webpack bundle analyzer plugin is really fun tool to use when looking at understanding why your code package is so large. This is the size equivalent of the X-Ray traces that let you visualize performance problems. This is a real picture from a service that have not optimized for size. Being able to see the breakdown of size so clearly made a couple of things jump out. I should really stop using moment. This service needs very simple date parsing, but the seemingly harmless dependency ends up being bigger than GraphQL itself. I should track down why mime-db (that big db.json blob) is being pulled in. I don’t know of a reason why mime types would be needed.
- #28 One thing you didn’t see which you often will at first is AWS-SDK and that’s because our default webpack config file will exclude that package. Since the AWS runtime already includes a copy of the AWS module, you can require it without packaging it in your application bundle. Another easy win is to use one of the minification plugins available with webpack. We use the MinifyPlugin and it’s worked well for us. My last piece of advice is to remember source maps. If you minify or you use babel to transpile, then you’ll want to enable the source-maps option in your webpack config and ship the source maps in your package.
- #29 Ok. I hope you’ve learned something to make your users and developers happy. If there is time I’m happy to answer any questions now. If there isn't enough time, or you want to dig into some idea in detail, find me after this or stop me when you see me later in the conference.