Download as PDF, PPTX
Uploaded byCodemotion
Lars Wolff - Performance Testing for DevOps in the Cloud - Codemotion Amsterdam 2019
The document discusses performance testing for DevOps in the cloud, emphasizing the importance of continuous performance testing to enhance system responsiveness and stability. It outlines various performance testing methods, challenges in implementation, and steps to establish a performance testing culture involving stakeholders. Finally, it highlights the need for defining goals, conducting different types of tests, and creating a community of practice for effective performance optimization.
Lars Wolff - Performance Testing for DevOps in the Cloud - Codemotion Amsterdam 2019
- 1. Performance Testing for DevOpsin the Cloud Lars Wolff Amsterdam | April 2-3, 2019
- 2. Introduction @larsvegas support@stormforger.com My Journey sofar • Software engineering • Consulting and Agile coaching • Performance Testing • Ship it! • AWS User Group Cologne & AWS Community Days Germany
- 3. Performance Testing SaaSfor DevOps Empower organizations to embrace continuous performance testing
- 4.
- 5. Performance • Frontend Performance •Backend Performance
- 6. Performance The ability ofa system to execute one task within a defined dimension = efficiency of a system ↪ 1 server manages 250 rps with p99 500ms.
- 7.
- 8.
- 9. Scalability capacity 0 1000 2000 3000 4000 resources 5 10 1520 25 30 Effectivity by which the capacity can be increased through adding resources
- 10. What makes merecognize that I have a performance problem? http://www.slideshare.net/jboner/scalability-availability-stability-patterns/15-How_do_I_know_if
- 11. If your systemis slow for one single user.
- 12. What makes merecognize that I have a scaling problem? http://www.slideshare.net/jboner/scalability-availability-stability-patterns/15-How_do_I_know_if
- 13. If your systemis fast for single users but slow under load.
- 14. The Cloud™ alreadyscales for us 👍
- 15.
- 16.
- 17.
- 19. Create an understandingof your system, its cloud run time environment and behaviour
- 20. Communication Collaboration AutomationMonitoring DevOps
- 21.
- 22. “In software engineering, performancetesting is in general, a testing practice performed to determine how a system performs in terms of responsiveness and stability under a particular workload.” https://en.wikipedia.org/wiki/Software_performance_testing – Wikipedia
- 23.
- 24. Performance Tests • Collectionof non-functional test methods • Not to be defined with absolute selectivity • Different goals and perspectives Load Testing Stress Testing Spike Testing Soak Testing Endurance Testing Resilience Testing Configuration Testing Scalability Testing
- 25. Performance Tests? • Notonly for testing releases • Evaluation of… • technology, • proof of concepts, • troubleshooting and debugging
- 26. less traffic ≠ no performanceproblems 😉
- 27. How do youdo it?
- 28. Performance Testing Maturity •no performance tests, at all 😅 • tests if problems occur during production 🔥🚨🔥 • tests (shortly) before large-scale (sales & marketing) campaigns ) • regular testing ) ) • tests as a regular, integrated DevOps instrument ) ) ) ) )
- 29.
- 30.
- 31. Challenges • organizational obstacles,e.g. buy-in (management & team) • testing goal & scope • visibility in your own systems • The right performance test tool for the job – (usually) no tooling problem • start! 😉 👍😁
- 32.
- 33. Steps to PerfTests™ • bring together the stakeholders + development 🤓 operations 🐜 QA 👷 product 💰 management 😱 marketing
- 34. Steps to PerfTests™ Kick off for 15 Min – All Stakeholders Heads Up STATE We’ll NOW create a performance testing culture to make sure that we can hold our promise to the business/customer to be fast and available under different amount of usage/users/transactions. We so match performance and availability requirements to keep the business save. ASK Do you all agree? STATE OK, we start with testing a very easy use case to raise a baseline of numbers about performance and availability. This friday, it’ll take a 2 hrs timebox. + development 🤓 operations 🐜 QA 👷 product 💰 management 😱 marketing
- 36. Steps to PerfTests™ Test Case • Create a first, simple Test Case • Q: Test Data? • Q: How much traffic?
- 37. Steps to PerfTests™ Test Case • Ask Analytics for the hour of the highest number of users in a week. • Analytics: We have 10.000 visits in this hour. The Users visits flower-list, one flower detail. 30% check out and payment.
- 38. Steps to PerfTests™ Test Case • Test Environment & Load Generator Setup • Since Infrastructure is code: Just start your test environment!
- 39.
- 40. Steps to PerfTests™ Run Test • 20 Minutes • Invite ALL stakeholders! • It’s about collaboration
- 41. Steps to PerfTests™ Inspect Report • Response Time? • If errors occurred: Do you see these in your monitoring, too?
- 42. Steps to PerfTests™ Meet and communicate to stakeholders: • ONE Statement like We aim for baseline numbers. We tested flower-list and flower-detail. We created 4x traffic of peak hour. We saw 20% of errors. But always responded in under 1000ms (p99). • Ask Are you happy? • Ask Whats’ next? Checkout? + development 🤓 operations 🐜 QA 👷 product 💰 management 😱 marketing
- 43. Steps to PerfTests™ Test Case • Add Checkout
- 44.
- 45.
- 46. Steps to PerfTests™ Non-functional requirements • No errors! • Each request must be faster than 250ms p99th
- 47. Steps to PerfTests™ Automate • Run test runs from any pipeline • Run automated checks of non-functional requirements
- 48.
- 49. It’s a feedbackloop Stakeholder Requirements Definition (test scenario) Execution (test case) Result Interpretation & Reporting Business Needs
- 50. Start creating a Communityof practice Special Interest Group since performance testing is an orthogonal topic
- 51. Thanks! 😁 LarsWolff @larsvegas support@stormforger.com https://stormforger.com Slides and more: http://bit.ly/codemotion2019
- 52. Performance Testing SaaSfor DevOps Empower organizations to embrace continuous performance testing http://bit.ly/codemotion2019
- 54.
- 55.
- 56. Defining goals • Defininggoals beforehand (why are we doing tests?) • Important! Always define goals (and requirements) from the specialist side ☝ • A goal can be very simple at first, e.g. “surviving Black Friday well” or “we want to become 20% faster”.
- 57. Where to startwith testing? • defining System Under Test (SUT) • finding the right scope • perimeter/end-to-end tests • components/service tests
- 58. System Under Test •testing “from top” (end-to-end) or “sideward” against single services • for the compartment side, only integrative tests are interesting • testing single services for the purpose of trouble shooting (technical perspective)
- 59. Define requirements • makeassumptions for non-functional requirements (NFR) • assistance via “baseline” performance tests and performance budgets • simple requirements might be: “deliver sites under 2sec”, “10k checkouts per hour”, … • also here: start simple, learn, improve, repeat! • do sanity checks!
- 60. + development 🤓 operations 🐜QA scenarios • bouncing visitor • returning customer • walk-in customer • … 👷 product 💰 management 😱 marketing workloads • Black Friday peak hour • average peak hour • 3 hour fire sale • … Who? What?
- 61. Testing together • Especiallybefore the first tests: involve all the stakeholders! • GO! • Present setup, assumptions and findings • Check or rework scenario sanity • Repeat!
- 62. Performance Budget Browser 🔥Firewall🔥 Load Balancer Webserver AppServer Service A Service B … FACT-FinderFACT-FinderFACT-Finder MySQLMySQLMySQL ☁☁ MySQLMySQLCache ……
- 63. Testing Types andStrategies
- 64. Stress Testing • Testswith increased up to extreme load • Understanding system behavior under (over)load • Identify limits and capacities
- 65. Stress Testing procedure •Define quality criteria (OK / KO) • Gradually increase load
- 66. Stress Testing procedure •Monitor system behavior • Identify bottlenecks • Detect capacity per resource
- 67. Scalability Testing • Howeffective are more resources transformed into capacity? • Foundation for capacity planning and cost estimation
- 68.
- 69. Spike Testing • Howdoes the system react during (extreme) peak loads? • Are we reacting fast enough? • Occasions: Marketing 😈, Reddit, Hacker News, …, TV shows
- 70. Soak Testing • Howdoes the system behave under load during a long period of time? • (very) long load test • Are there any longtime effects? Memory leaks?
- 71. Configuration Testing • Howdoes the behavior change if the configuration modifies? • Series of test executions • Investigate the impact of the environment on your own system
- 72. …in the cloud •Types of instances • Auto scaling settings • Throughput provisioning • Ideal usage of services?
- 73. …and in general… •Hypervisor • Operating system • Web & application server • Software configuration • Software dependencies
- 74.
- 75. Availability & ResilienceTesting • (Zero-Downtime) deployments under load • Changes in infrastructure • Error scenarios • Failover testing • => Chaos engineering