Downloaded 75 times
Uploaded byGuido Schmutz
Building Event Driven (Micro)services with Apache Kafka
The document discusses the advantages of building event-driven microservices using Apache Kafka, detailing various architectural approaches, including traditional SOA and modern microservice designs. It highlights Kafka's capabilities in handling streaming data, integrating legacy applications, and performing historical data analytics, asserting that service autonomy is crucial for microservices architecture. The presentation emphasizes Kafka's role as an event store and its additional features beyond simple message passing.
Building Event Driven (Micro)services with Apache Kafka
- 1. BASEL BERN BRUGGDÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH Building event-driven (Micro)Services with Apache Kafka Guido Schmutz Liverpool – 3.12.2018 @gschmutz guidoschmutz.wordpress.com
- 2. Agenda 1. Where dowe come from? 2. Using Microservices 3. Can we do even better? 4. How does Apache Kafka help? 5. What about streaming data sources? 6. What about integrating legacy applications? 7. What about (historical) data analytics? 8. Summary
- 3. Guido Schmutz Working atTrivadis for more than 22 years Oracle Groundbreaker Ambassador & Oracle ACE Director Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data Head of Trivadis Architecture Board Technology Manager @ Trivadis More than 30 years of software development experience Contact: guido.schmutz@trivadis.com Blog: http://guidoschmutz.wordpress.com Slideshare: http://www.slideshare.net/gschmutz Twitter: gschmutz 135th edition
- 4. Where do wecome from?
- 5. Shop Rich UI ShopBackend Application Traditional Approach Search Facade Customer DAO Order DAO Order Facade Shop UI Product DAO UI Logic DataBusiness GUI Customer Fat Client App Customer BOCustomer UI DataGUI Data Storage Shared Database sync request/response
- 6. Shop UI App Business ActivityService SOA Approach Contract-first Web Services Technical layers offer their own interfaces Reuse on each level Lower layer often wraps legacy code Search BAS Customer DAO Order DAO Order BAS Shop UI Product DAO UI Logic GUI Business Entity ServiceShop Web App Shop UI UI Logic GUI Data Storage Customer Database Customer BES Payment BES Product BES Order BES Custer BAS Order and Product DB SOAP SOAP SOAP SOAP SOAP SOAP SOAP
- 7. Business Activity Service Virtualized SOAApproach The raise of the Enterprise Service Bus (ESB) Search BAS Customer DAO Order DAO Order BAS Business Entity Service Data Storage Customer Database Customer BES Payment BES Product BES Order BES Custer BAS Order and Product DB Service Virtualization Layer Service Bus SOAP SOAP SOAP SOAP SOAP SOAP SOAP Shop UI App Shop UI UI Logic GUI Shop Web App Shop UI UI Logic GUI X
- 8. Shop UI App Business ActivityService Orchestrated & Virtualized SOA Approach – Sync / Async The raise of orchestration engines (BPEL & BPMN) Search BAS Customer DAO Order DAO Order BAS Shop UI UI Logic GUI Business Entity Service Shop Web App Shop UI UI Logic GUI Data Storage Customer Database Customer BES Payment BES Product BES Order BES Custer BAS Order and Product DB Service Virtualization Layer Service Bus X Orchestration
- 9.
- 10. Microservice Approach Tightly Scopedbehind clear interfaces Responsible for managing their own data (not necessarily the infrastructure) Should be highly decoupled Independently deployable, self- contained and autonomous Is it just SOA done right ? { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST
- 11. Synchronous World ofRequest-Response leads to tight, point-to-point couplings problem in lower end of chain have a ripple effect on other service • crash of service • overloaded service / slow response time • change of interface Service 2Service 1 { } API Logic { } API Logic StateState Service 3 { } API Logic State Service 4 { } API Logic State Service 5 { } API Logic State Service 7 { } API Logic State Service 6 { } API Logic State RESTRESTRESTREST REST REST REST
- 12. Microservice Approach with APIGateway Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST API Gateway X
- 13. Microservice Approach withSide Car (i.e. K8s & Istio) Side-car can provide: • retry • load-balancing • circuit breaker, throttling • security • but still sync request/response Service 2Service 1 { } API Logic { } API Logic StateState Service 3 { } API Logic State Service 4 { } API Logic State Side Car Side Car Side Car REST RESTRESTREST
- 14. Microservice Approach withSide Car (i.e. K8s & Istio) Side-car and it’s advanced routing capabilities can be used to switch to new service version Service 2Service 1 { } API Logic { } API Logic StateState Service 3 – v1 { } API Logic State Side Car Side Car Service 3 – v2 { } API Logic State RESTREST REST REST
- 15. Side Car providesabstraction …. but Service 1 { } API Logic State Service 2 { } API Logic State Service 4 Logic State Service 3 { } API Logic State { } API New Service Logic State { } API we still have to change the invoking service …. if another service requires the same information REST REST REST REST REST
- 16. Can we doeven(t) better?
- 17. Three mechanisms throughwhich services can interact Request-Driven (Imperative) Event Driven (Functional) Service Logic State Consume Event “Event Ordered” OrderEvent Command ”Order IPad” boolean order(IPad) Publish Event ”Event Validated” OrderValidatedEvent Query ”Retrieve my Orders List<Orders> getAllOrders(for) Event Broker Service Logic State
- 18. Event-Driven (Async) Microservice Approach CustomerMicroservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST API Gateway Pub / Sub sync request/response async request/response async, event pub/sub Customer Mat View
- 19. How does ApacheKafka help?
- 20. Highly available, Pub/Subinfrastructure Highly Scalable Apache Kafka – A Streaming Platform Distributed Log at the Core Logs do not (necessarily) forget • Never • Time (TTL) or Size-based • Log-Compacted based
- 21. Example Customer Microservice { } CustomerAPI CustomerCustomer Logic Order Microservice { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Schema Registry Schema Customer (compacted) https://github.com/gschmutz/various-demos/tree/master/event-driven-microservices
- 22. Example @Configuration public class KafkaConfig{ private String bootstrapServers; private String schemaRegistryURL; @Bean public Map<String, Object> producerConfigs() { Map<String, Object> props = new HashMap<>(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class); props.put(KafkaAvroSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryURL); return props; } @Bean public ProducerFactory<String, Customer> producerFactory() { .. } @Bean public KafkaTemplate<String, Customer> kafkaTemplate() { return new KafkaTemplate<>(producerFactory()); } @Component public class CustomerEventProducer { @Autowired private KafkaTemplate<String, Person> kafkaTemplate; @Value("${kafka.topic.customer}") String kafkaTopic; public void produce(Customer customer) { kafkaTemplate.send(kafkaTopic, customer.getId().toString(), customer); } }
- 23. Adding a newservice by bootstrapping from Event Hub Customer Search Microservice { } Customer API CustomerCustomer Logic REST Schema Registry Schema Consume from Offset 0 Customer Microservice { } Customer API CustomerCustomer Logic Order Microservice { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Customer (compacted)
- 24. What about streamingsources?
- 25. How to workwith streaming data sources Customer Microservice { } Customer API Customer Customer Logic Order Microservice { } Order API Order Order Logic Product Microservice { } Product API Product Product Logic Stock Microservice { } Stock API Stock Stock Logic Shop Web App Shop UI UI Logic GUI REST REST REST REST Event Hub Location Social Click stream Sensor Data Mobile Apps Weather Data Event Stream
- 26. Hadoop Clusterd Hadoop Cluster StreamProcessing Cluster Streaming Processing & Microservices Architecture BI Tools SQ L Search / Explore Online & Mobile Apps Search Service Event Stream Results Stream Analytics Reference / Models Dashboard Location Social Click stream Sensor Data Mobile Apps Weather Data Microservice Cluster Microservice State { } API Event Stream Event Stream Event Hub Service
- 27. Apache Kafka –scalable message processing and more! Source Connector Kafka Broker Sink Connector Stream Processing Schema Registry Kafka Kafka
- 28. Kafka Connect -Overview Source Connector Sink Connector
- 29. Kafka Streams -Overview Designed as a simple and lightweight library in Apache Kafka no other dependencies than Kafka Supports fault-tolerant local state Supports Windowing (Fixed, Sliding and Session) and Stream-Stream / Stream- Table Joins Millisecond processing latency, no micro-batching At-least-once and exactly-once processing guarantees KTable<Integer, Customer> customers = builder.stream(”customer"); KStream<Integer, Order> orders = builder.stream(”order"); KStream<Integer, String> enriched = orders.leftJoin(customers, …); joined.to(”orderEnriched"); trucking_ driver Kafka Broker Java Application Kafka Streams
- 30. KSQL - Overview STREAMand TABLE as first-class citizens • STREAM = data in motion • TABLE = collected state of a stream Stream Processing with zero coding using SQL-like language Built on top of Kafka Streams Interactive (CLI) and headless (command file) ksql> CREATE STREAM customer_s WITH (kafka_topic=‘customer', value_format=‘AVRO'); Message ---------------- Stream created ksql> SELECT * FROM customer_s WHERE address->country = ‘Switzerland’; ... trucking_ driver Kafka Broker KSQL Engine Kafka Streams KSQL CLI Commands
- 31. What about integratinglegacy applications?
- 32. Hadoop Clusterd Hadoop Cluster StreamProcessing Cluster Integrate existing systems through CDC BI Tools SQ L Search / Explore Online & Mobile Apps Search Service Event Stream Results Stream Processor Reference / Models Dashboard Location Social Click stream Sensor Data Mobile Apps Weather Data Microservice Cluster Microservice State { } API Event Stream Event Stream Event Hub Service Billing & Ordering CRM / Profile Marketing Campaigns Change Data Capture
- 33. Data Store Integrate existingsystems through CDC Customer Event Hub Integration Microservice StateLogic CDC CDC Connector Customer Fat Client App Customer BOCustomer UI Stream Processing Results Stream Processor Reference / Models Dashboard Capture changes directly on database Change Data Capture (CDC) => think like a global database trigger Transform existing systems to event producer
- 34. Legacy Microservice Change DataCapture (CDC) with Kafka Broker and Kafka Connect RDBMS cdc-source trucking_ driver Kafka Broker elasticsearch- sink NoSQL Customer Topic
- 35. Legacy Microservice Change Data Capture(CDC) with Kafka Broker and Kafka Connect RDBMS cdc-source elasticsearch- sink NoSQL Enhance Kafka Broker Customer Topic CustomerEnhanced Topic State
- 36. What about (historical)data analytics?
- 37. Streaming & (Big)Data Analytics Architecture Event Stream Hadoop Clusterd Hadoop Cluster Big Data Cluster D ata Flow Parallel Processing Storage Storage RawRefined Results Microservice Cluster Microservice State { } API Stream Processing Cluster Stream Processor State { } API Event Stream Event Stream SQL Search Service BI Tools Enterprise Data Warehouse Search / Explore Online & Mobile Apps SQL Export SearchEvent Hub Service Location Social Click stream Sensor Data Mobile Apps Weather Data Billing & Ordering CRM / Profile Marketing Campaigns Change Data Capture File Import / SQL Import
- 38.
- 39. Hadoop Clusterd Hadoop Cluster BigData Summary Billing & Ordering CRM / Profile Marketing Campaigns SQL Search Service BI Tools Enterprise Data Warehouse Search / Explore Online & Mobile Apps File Import / SQL Import Event Hub D ata Flow D ata Flow Change Data Capture Parallel Processing Storage Storage RawRefined Results SQL Export Microservice State { } API Stream Processor State { } API Event Stream Event Stream Search Service Location Social Click stream Sensor Data Mobile Apps Weather Data Stream Processing Microservices
- 40. Summary • service autonomyis key in a Microservices Architecture! • not all communication need to be synchronous => separate into • commands • events • queries • Kafka is well suited as an event store • brings many more interesting features beyond just “message passing” • Log compaction allows storing latest value per key for all time • Does not provide all features for event sourcing (but maybe not needed)
- 41. References Microservices Blog Series,Ben Stopford, Confluent: • https://www.confluent.io/blog/tag/microservices Apache Kafka for Microservices: A Confluent Online Talk Series: • https://www.confluent.io/landing-page/microservices-online-talk-series/ Turning the database inside-out with Apache Samza, Martin Kleppmann, Con • https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection?, Neha Narkhede, Confluent: • https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/ Immutability Changes Everything, Pat Helland, Salesforce: • http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf Commander: Better Distributed Applications through CQRS and Event Sourcing, Bobby Calderwood: • https://www.youtube.com/watch?v=B1-gS0oEtYc
- 42. Technology on itsown won't help you. You need to know how to use it properly.