Download as PDF, PPTX
Uploaded byBernd Ruecker
Kafka Summit 2021 - Apache Kafka meets workflow engines
This document discusses how workflow engines can be used with Apache Kafka to enable scalable orchestration of processes and events. A workflow engine provides capabilities for modeling business processes and orchestrating the steps in a process. When integrated with Apache Kafka, it allows processes to react to events published to Kafka topics. This enables building distributed applications through a combination of choreography (peer-to-peer communication) and orchestration (centralized workflow coordination). Examples of use cases include order fulfillment processes across microservices and vehicle maintenance workflows that react to sensor data streams. The document provides a code example of integrating Camunda workflow engine with Kafka and Confluent Cloud.
Kafka Summit 2021 - Apache Kafka meets workflow engines
- 1. Apache Kafka Meets WorkflowEngines Bernd Ruecker Co-Founder and Chief Technologist @Camunda @berndruecker
- 2.
- 3. There are prettyamazing use cases @berndruecker Order Management in Telco 5,000 process instances/s Money Transfer 200 process instances/s Perseverance Mars Rover Nasa
- 4.
- 5.
- 6. Scale Onboarding you might think of Real-life onboardinguse case in banking Money transfer or trading Process Instances per second @berndruecker
- 7. Making phone calls Scale Telco Onboarding Process Instances persecond @berndruecker
- 8. A process Steps In acertain sequence Which might be long running @berndruecker Time-controlled events
- 9. A workflow engine Ormore scalable: @berndruecker
- 10. A workflow engine Ormore scalable: @berndruecker Camunda Platform Camunda Cloud Two workflow engines: - RDBMS / traditional & on-prem - Event-Sourcing / cloud-native
- 11. https://blog.bernd-ruecker.com/zeebe- io-a-horizontally-scalable-distributed- workflow-engine-45788a90d549 “Zeebe” is theworkflow engine offered within Camunda Cloud
- 12. Workflow engines atscale https://camunda.com/blog/2019/08/zeebe-horizontal-scalability/ @berndruecker
- 13.
- 14. Technical Example Java Spring Boot Application CamundaCloud Confluent Cloud Kafka Topic Zeebe Worker Via subscription Publish record Kafka Listener Via subscription Correlate to process instance https://github.com/berndruecker/kafka-camunda-spring-simple @berndruecker
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21. The danger isthat it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
- 22. The danger isthat it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
- 23. The danger isthat it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
- 24. Peer-to-peer event chains Checkout Payment Inventory Shipment Order placed Payment received Goods shipped Goods fetched Fetchthe goods before the payment @berndruecker
- 25. Peer-to-peer event chains Checkout Payment Inventory Shipment Fetchthe goods before the payment Goods fetched Order placed Payment received Goods shipped @berndruecker
- 26. What we wanted Photoby Lijian Zhang, under Creative Commons SA 2.0 License and Wikimedia Commons / CC BY-SA 4.0 @berndruecker Choreography
- 27. Why is itso tempting? Service A Event Bus A B Service B C Service C D E F Service D G Service … Service … Service … Service … Service … Service … Service … @berndruecker
- 28. Why is itso tempting? Service A Event Bus A B Service B C Service C D E F Service D G Service … Service … Service … Service … Service … Service … Service … Adding is easy! You can „buy“ a shorter initial time-to-value by choreography. It yields in technical debt. @berndruecker
- 29.
- 30. Order Fulfillment Decide about responsibility Checkout Payment Inventory Shipment Payment received Order placed Retrieve payment @berndruecker Thisis choreography This is orchestration
- 31. Order Fulfillment Decide about responsibility Checkout Payment Inventory Shipment Order placed Retrieve payment Itcan still be Apache Kafka! @berndruecker
- 32. ? Event Command Message RecordEvent Fact, happened in the past, immutable Intend, Want s.th. to happen, The intention itself is a fact @berndruecker
- 33. ? Event Command Message RecordEvent @berndruecker
- 34. Commands in disguise TheCustomer Needs To Be Sent A Message To Confirm Address Change Event Send Message Wording of Sender Wording of recipient
- 35. Checkout Order Payment Event-driven: Decisionto couple is on the receiving side Command-driven Decision to couple is on the sending side Direction of dependency Retrieve Payment Order placed Payment received Direction of dependency @berndruecker
- 36.
- 37.
- 38. Now it iseasy to change the process flow @berndruecker
- 39.
- 40. Customer created Sam Newman:Building Microservices @berndruecker
- 41. Mix orchestration andchoreography Orchestration Orchestration Orchestration Choreography @berndruecker
- 42. Another use case:Transactions Service Kafka RDMS 1. Receive 4. Send additional events 2. Business Logic 3. Send response @berndruecker
- 43. Another use caseexample: Vehicle Maintenance
- 44. Action Insights Measurements generates Leads to oilpressure is 80 psi oil pressure is critically high Schedule maintenance Call driver to stop and inspect … @berndruecker
- 45. Action Insights Measurements generates Leads to oilpressure is 80 psi oil pressure is critically high Schedule maintenance Kafka Topic Measures Kafka Topic Insights Camunda Workflow Maintenance Stateful Stream Processor Decision Workflows Event Streams @berndruecker Custom Connector
- 46.
- 47.
- 48. Summary • A workflowengine is complementary to Apache Kafka • The technical integration is easy • There are tools that scale • Typical use cases • Microservices orchestration (balanced with choreography) • Business transactions • Actions resulting from streams @berndruecker
- 49.
- 50. Want to learnmore? https://ProcessAutomationBook.com/ Free electronic version available Featuring code examples:
- 51.
- 52. Berlin, Germany http://berndruecker.io/ mail@berndruecker.io @berndruecker Bernd Ruecker Co-founderand Chief Technologist of Camunda