Download to read offline
Uploaded byHeitor Lessa
re:Invent 2023 - The Pragmatic Serverless Python Developer
The document outlines best practices and use cases for developing serverless applications using Python on AWS, with an emphasis on pragmatism and integration strategies. It discusses the importance of project structure, infrastructure as code, testing methodologies, and handling integration with services like Amazon DynamoDB and EventBridge. Additionally, it provides insights into optimal handler designs, testing approaches, and scenarios for building effective product APIs and event processors.
re:Invent 2023 - The Pragmatic Serverless Python Developer
- 1. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved.
- 2. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. The pragmatic serverless Python developer Heitor Lessa O P N 3 0 5 - R 1 (he/him) Chief Architect, Powertools for AWS AWS Ran Isenberg (he/him) Principal Software Architect CyberArk
- 3. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Pragmatism Use case Product API Stream processor Wrap-up Readiness
- 4. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Pragmatism: Analysis paralysis Infra as code framework decisions & trade-offs situational awareness is key Project structure Domain-driven design? Micro or mono functions? Open source tools? Sync or async? Asynchronous testing? App integrations Observability? Leading practices? Metrics? Alarms? Documentation?
- 5. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Pragmatism: Open source tools mkdocs-material pytest Pydantic Powertools for AWS Lambda pre-commit YAPF Tuna Radon Xenon AWS CDK mkdocstrings isort flake8 poetry cdk-monitoring-constructs codecov actionlint gitleaks GitPython typing-extensions ` mypy-boto3 cdk-nag cachetools aws-lambda-env-modeler mypy pytest-socket markdownlint dependabot Semantic PR
- 6. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Pragmatism: Leading practices (CI) Project setup pre-commit check Merge conflicts Valid Python code JSON/YAML/TOML Secrets linting Code linting pre-pull request +pre-commit checks Test (unit/contract) AppSec baseline Complexity baseline Docs linting Pull request checks Governance checks Test (integ/E2E) SAST +pre-PR checks Dependency CVE code checkout virtual env dependencies pre-commit hooks
- 7. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Pragmatism Use case Product API Stream processor Wrap-up Readiness
- 8. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Use case: Simple yet practical Product API Product database Product events
- 9. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Use case: Product API Product database Product events Route handlers CREATE Product LIST Products GET Product DELETE Product Amazon API Gateway
- 10. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Use case: Product database Route handlers CREATE Product LIST Products GET Product DELETE Product Amazon DynamoDB Amazon API Gateway Product events
- 11. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Use case: Product events AWS Lambda stream poller AWS Lambda Stream processor Amazon EventBridge Route handlers CREATE Product LIST Products GET Product DELETE Product Amazon DynamoDB Amazon API Gateway
- 12. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Use case: Synchronous path AWS Lambda stream poller AWS Lambda Stream processor Amazon EventBridge Route handlers CREATE Product LIST Products GET Product DELETE Product Amazon DynamoDB Amazon API Gateway Synchronous
- 13. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Use case: Asynchronous path AWS Lambda stream poller AWS Lambda Stream processor Amazon EventBridge Route handlers CREATE Product LIST Products GET Product DELETE Product Amazon DynamoDB Amazon API Gateway Synchronous Asynchronous
- 14. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Pragmatism Use case Product API Stream processor Wrap-up Readiness
- 15. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Product API: Practices Practices Project structure, infra as code Lambda handler practices Business domain Integration logic, AWS services, and more Unit, integration, end-to-end testing Amazon DynamoDB Route handlers GET /products GET /{product} PUT /{product} DELETE /{product} Amazon API Gateway
- 16. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Synergy of elements DevOps culture DEV -> Production Faster development IaC Domain code Tests Readiness Project structure Project: Overview
- 17. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Project repository structure reflects elements: • Business domain: Lambda handlers code, business logic, and integrations code • Tests: unit, integration, end to end • Production readiness: spread across all folders • pyproject.toml: one file to rule them all • IaC: CDK, AWS SAM, Terraform, and more Project: Folder structure
- 18. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • We chose AWS CDK as IaC framework • One CDK app and one stack • Domain-driven approach: • One construct for product REST API • One constructs builds: • API GW • Lambda functions • DynamoDB table • All roles Project: Infrastructure as code
- 19. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Start with one Lambda function • All business domain code and logic • What is the ”best” way to write a Lambda function? Amazon DynamoDB Route handlers PUT /{product} Amazon API Gateway Handler: Overview
- 20. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Start with a handler that does it all • Single file • Gets input Handler: Event source parsing
- 21. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Start with a handler that does it all • Single file • Gets input • Domain logic/DB access Handler: Integration
- 22. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Start with a handler that does it all • Single file • Gets input • Domain logic/DB access • Returns response Handler: Response
- 23. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Start with a handler that does it all • Single file • Gets input • Domain logic/DB access • Returns response • Context matters • Cron job handler vs. REST API with logic Handler: One handler to rule them all?
- 24. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • As the service expands: • Code duplication • Testing becomes harder • Code readability issues – 300+ lines • This does not scale! • We need a better way to develop complex services Amazon DynamoDB Route handlers GET /products GET /{product} PUT /{product} DELETE /{product} Amazon API Gateway Handler: Will my service evolve?
- 25. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Business logic only May be shared by N handlers Isolated tests Handler: Architectural layers Handler No domain code Concise responsibilities Domain
- 26. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Handler: Responsibilities Configuration Env. vars Validation Serialize output Call domain
- 27. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. access Amazon DynamoDB Amazon API Gateway Handler Domain invoke handle output serialize Handler: High-level event flow
- 28. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Zero domain code • Short and concise • Clear responsibilities Handler: Event source parsing (after)
- 29. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Zero domain code • Short and concise • Clear responsibilities Handler: Integration (after)
- 30. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Zero domain code • Short and concise • Clear responsibilities Handler: Response (after)
- 31. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Zero domain code • Short and concise • Clear responsibilities Handler: Refactored
- 32. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Functional or OOP • Interface: • Create product • Get product • List products • Delete product Domain: Product
- 33. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Functional or OOP • Interface: • Create product • Get product • List products • Delete product • Access DynamoDB • Return output Domain: Integration
- 34. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Integration practice relates to: • Code that accesses DynamoDB • And in general: • Code that accesses API • Integration with AWS resources • Integration with external APIs Amazon DynamoDB Route handlers GET /products GET /{product} PUT /{product} DELETE /{product} Amazon API Gateway Integration: Overview
- 35. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Domain code contains integration code • When can it backfire? Integration: Should domain integrate directly?
- 36. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • DB requirements change and evolve • New queries are too slow • What if we replace DynamoDB? • Harder refactors • No isolated testing • Can we do better? Route handlers GET /products GET /{product} PUT /{product} DELETE /{product} Amazon API Gateway Integration: Testing and refactoring challenges Amazon Aurora Amazon DynamoDB
- 37. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Adapter pattern Contains API/DB code Business logic only May be shared by N handlers Isolated tests Integration: A new architecture layer Handler No domain code Concise responsibilities Domain Integration
- 38. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. invoke handle serialize output Amazon DynamoDB Amazon API Gateway Handler Domain Integration: High-level overview Integration call access output
- 39. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Integration: Agreeing on an interface • Abstract interface • Interface includes: • Create product • List products • Get product • Delete product
- 40. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Integration: Implementation example • Implements adapter pattern • The only class that has DynamoDB code • Isolated tests • Aurora handler inherits interface
- 41. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Integration: Refactoring domain • Getter for concrete implementation • Domain is not aware of the underlying DB • Easy to switch to a different database • Output conversion
- 42. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. DevX Never leave IDE Real cloud resources Local debug Amazon DynamoDB Route handlers GET /products GET /{product} PUT /{product} DELETE /{product} Amazon API Gateway Testing: Overview
- 43. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Unit No deployment Run Locally Testing: Pyramid
- 44. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. call Test runner (pytest) Isolated unit Assertion Test data verify prepare Testing: Unit test overview
- 45. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Test schema validations (Pydantic) • Test isolated integration classes • No deployment required • Fast & repeatable Testing: Model validation
- 46. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Integration Unit Requires deployment No deployment Run locally & on AWS Run locally Testing: Integration tests
- 47. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. call Test runner (pytest) Lambda handler Assertion Test data verify prepare resources Testing: Integration test overview
- 48. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Build repeatable and consistent tests • Event details: • Print real events in CloudWatch Logs • Powertools for AWS Lambda test JSON events • AWS service documentation Testing: Event generation
- 49. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Test-driven development (TDD) • Breakpoints in IDE • Work against cloud resources Testing: Local code, cloud resources
- 50. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Test-driven development (TDD) • Breakpoints in IDE • Work against cloud resources • Assert side effects and response Testing: Asserting responses
- 51. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Test-driven development (TDD) • Breakpoints in IDE • Work against cloud resources • Assert side effects and response Testing: Asserting side effects
- 52. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Test-driven development (TDD) • Breakpoints in IDE • Work against cloud resources • Assert side effects and response Testing: Integration test complete
- 53. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Simulate failures • Mock DynamoDB exceptions • Assert handler response • Why not write as a unit test? Testing: Injecting SDK failures
- 54. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. E2E Integration Unit Require deployment Require deployment No deployment Run on AWS Run locally & on AWS Run locally Testing: End-to-end (E2E) tests
- 55. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. call Test runner (pytest) Product API Assertion Test data verify prepare Endpoint Lambda function invoke Testing: E2E test overview
- 56. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Simulate real user usage • Runs 100% on AWS Testing: Black box testing
- 57. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Simulate real user usage • Runs 100% on AWS • Assert response Testing: Asserting product creation
- 58. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. • Simulate bad input • API security tests: • Invalid authorization • Invalid authentication Testing: Unhappy paths
- 59. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Pragmatism Use case Product API Stream processor Wrap-up Readiness
- 60. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Amazon DynamoDB Stream processor: Intro Amazon EventBridge AWS Lambda Stream processor Practices Project scaffolding, infra as code Lambda handler practices Business domain Integration logic, AWS services, and more Unit, integration, end-to-end testing
- 61. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Practices Same structure as CRUD API Project: Folder structure
- 62. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Practices Extend your handler with defaults Handler: Extend the signature
- 63. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Practices Extend your handler with defaults Build domain input from event Handler: Prepare and validate domain input
- 64. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Practices Extend your handler with defaults Build domain input from event Inject integrations Handler: Create and inject event emitter
- 65. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Practices Dependency injection to the rescue Domain: Use dependency and issue receipt
- 66. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Amazon EventBridge Integration: High-level overview Integration layer Domain
- 67. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Amazon EventBridge Integration: High-level overview Event Handler Domain
- 68. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Amazon EventBridge Integration: High-level overview Event Handler Domain Model to Event Standardize Event Inject metadata
- 69. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Amazon EventBridge Integration: High-level overview Event Handler Domain Model to Event Event Provider Standardize Event Inject metadata send
- 70. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Amazon EventBridge Integration: High-level overview Event Handler Domain Model to Event Event Provider Standardize Event Inject metadata send PutEvents
- 71. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Amazon EventBridge Integration: High-level overview Event Handler Domain Model to Event Event Provider Standardize Event Inject metadata Build request Error Handling Issue Receipt send PutEvents
- 72. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Integration: Model to standard Event
- 73. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Integration: Model to standard Event Model Event
- 74. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Integration: Model to provider request Model Event EventBridge
- 75. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Integration Domain Handler AWS Lambda Amazon EventBridge Testing: Isolated tests, thanks to layering Amazon DynamoDB
- 76. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Testing: Quick recap on our handler
- 77. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Integration Domain Amazon EventBridge Amazon DynamoDB Testing: Unit testing Lambda handler AWS Lambda Handler Event source Domain input EventHandler
- 78. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Testing: Preventing side effects Practices Fail tests for unexpected connections
- 79. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Testing: FakeEventHandler Practices Fail tests for unexpected connections Use in-memory Fakes
- 80. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Testing: Unit testing benefits Practices Fail tests for unexpected connections Use in-memory Fakes Test in isolation, and as a whole unit
- 81. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Amazon DynamoDB Integration Amazon EventBridge Domain Testing: Unit testing domain AWS Lambda Handler Event source Domain input EventHandler Business rule Use integ.
- 82. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Testing: Unit testing domain logic Practices Fail tests for unexpected connections Use in-memory Fakes Test in isolation, and as a whole unit
- 83. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Amazon DynamoDB Amazon EventBridge Integration Domain Testing: Unit testing integration AWS Lambda Handler Event source Domain input EventHandler Business rule Use integ.
- 84. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Testing: Contract testing Practices Fail tests for unexpected connections Use in-memory Fakes Test model -> event -> provider request
- 85. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Practices Fail tests for unexpected connections Use in-memory Fakes Test model event provider request Testing: Contract testing (event)
- 86. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Testing: Contract testing (provider) Practices Fail tests for unexpected connections Use in-memory Fakes Test model event provider request
- 87. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Testing: Asserting contract Practices Fail tests for unexpected connections Use in-memory Fakes Test model event provider request AWS SDK stubber for deeper validation
- 88. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Amazon DynamoDB Amazon EventBridge Integration Domain Testing: Asserting asynchronous behavior AWS Lambda Handler Event source Domain input EventHandler Business rule Use integ. Init provider Build event Send event
- 89. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Amazon EventBridge Testing: Listening to events AWS Lambda Amazon DynamoDB
- 90. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Amazon EventBridge Testing: Creating new side effects AWS Lambda AWS Step Functions Amazon DynamoDB Amazon DynamoDB Intercept Event Testing Construct
- 91. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Amazon EventBridge Testing: Querying intercepted events AWS Lambda AWS Step Functions Amazon DynamoDB Amazon DynamoDB Intercept Event Testing Construct pk sk data metadata receipt_id test_eventbridge_provider_send b30a7..#SAMPLE_NOTIFICATION #2023-10-08T20:17:02… {…} {…} b30a7…
- 92. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Testing: Using test name as event source Practices Fail tests for unexpected connections Use in-memory Fakes Test model event provider request Validate input with AWS SDK stubber Make it easier to trace test events
- 93. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Testing: Asserting intercepted events Practices Fail tests for unexpected connections Use in-memory Fakes Test model event provider request Validate input with AWS SDK stubber Make it easier to trace test events Consider eventual consistency
- 94. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Pragmatism Use case Product API Stream processor Wrap-up Readiness
- 95. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Powertools for AWS Lambda: Toolkit Batch processing REST/GraphQL API Input/output validation Config management Secrets handling Idempotency Observability BYO middleware Self-documented schemas Feature flags Data extraction Caching best practices, for everyone Streaming *feature set may vary across languages Python | TypeScript | Java | .NET
- 96. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Tuna: Visualizing import time (cold start)
- 97. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Py-spy: Visualizing most freq. code path
- 98. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Pyinstrument: Visualizing select code areas
- 99. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Pragmatism Use case Product API Stream processor Wrap-up Readiness
- 100. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Recap: Open source repository opinionated reference and fully documented
- 101. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Summary: Takeaways Resources and template
- 102. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Thank you! © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Please complete the session survey in the mobile app Thank you! © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Please complete the session survey in the mobile app Heitor Lessa @heitor_lessa lessa@amazon.com Ran Isenberg @IsenbergRan Ran.Isenberg@ranthebuilder.cloud www.ranthebuilder.cloud