Downloaded 14 times
Message Driven and Event Sourcing
- 1. Change is thenew normal
- 2. Message Driven &Event Sourced Andrea Gioia - Partner Quantyca Paolo Castagna - Account Executive, Confluent Message Driven & Event Sourced / quantyca.it | confluent.io
- 3. Scalability & Cloud/ quantyca.it | google Andrea Gioia Partner, Quantyca Paolo Castagna Account Executive, Confluent
- 4. Intro Message Driven &Event Sourced / quantyca.it | confluent.io
- 5. Message driven Strong decoupling Lanecessità di delineare per ogni componente il giusto confine che possa garantire il più basso accoppiamento con gli altri, l’isolamento e la trasparenza sul dislocamento richiede di basare le interazioni all’interno delle moderne data platform sullo scambio asincrono di messaggi. Broker Message Driven & Event Sourced / quantyca.it | confluent.io
- 6. Inside the broker Smartendpoints and dumb pipes Attenzione a non spostare la complessità all’interno del broker. Le logiche di business dovrebbero essere il più possibili contenute nelle componenti che accedono al broker non nel broker stesso. = Big ball of mud Spaghetti in a box Message Driven & Event Sourced / quantyca.it | confluent.io
- 7. Due tipologie dibroker Service Bus e Data Bus Message Driven & Event Sourced / quantyca.it | confluent.io
- 8. Due tipologie dibroker Service Bus (orchestration over choreography) Nelle tradizionali architettura SOA il coordinamento dei servizi esposti da ogni sistema avviene tramite un Enterprise Service Bus (ESB) centralizzato. L’architettura basata su ESB permette di: 1. comporre servizi esposti da sistemi legacy 2. gestire transazioni distribuite 3. creare un anticorruption layer per isolare i sistemi monolitici 4. Facilitare il test e monitoraggio dell’infrastruttura Message Driven & Event Sourced / quantyca.it | confluent.io
- 9. Due tipologie dibroker Data Bus (choreography over orchestration) Nelle moderne architettura a microservizi si rinuncia alla centralizzazione per aumentare la scalabilità del sistema. Ogni servizio è modellato intorna ad uno specifico dominio di business ed è completamente autonoma dagli altri dal punto di vista di : 1. Implementazione 2. Storage 3. Deploy L’allineamento a livello di dati tra i vari servizi avviene tramite un data bus basato su una piattaforma di event streaming distribuita (Kafka) Message Driven & Event Sourced / quantyca.it | confluent.io
- 10. Due tipologie dibroker Architettura Ibrida e API Manager Le due architetture possono coesistere. Il layer di servizi avrà sempre più una doppia polarità: ● da una parte servizi esposti da sistemi legacy orchestrati tramite ESB (SOA) ● dall’altra servizi con coordinamento decentralizzato e altamente scalabili (Microservices) Per i consumatori di servizi questa architettura ibrida non sarà direttamente visibile in quanto l’accesso ai servizi passerà tramite API Gateway che medieranno l’accesso ai servizi reali Message Driven & Event Sourced / quantyca.it | confluent.io
- 11. How Apache Kafkaand Confluent enable event streaming architectures Message Driven & Event Sourced / quantyca.it | confluent.io
- 12. Events An immutable recordthat something as some point has happened Message Driven & Event Sourced / quantyca.it | confluent.io A Sale An Invoice A Trade A Customer Experience
- 13. 5.2 Million Citizens NorwegianWork and Welfare Administration Life is a Stream of Events
- 14. Pre-Streaming Message Driven &Event Sourced / quantyca.it | confluent.io
- 15. Request-Response Applications Message Driven& Event Sourced / quantyca.it | confluent.io App Service Service Service Service Service Service Service Service App ■ DETERMINISTIC ■ RIGID ■ TIGHT
- 16. Event-Driven Applications Message Driven& Event Sourced / quantyca.it | confluent.io App App Developer APIs Service Service Service Service Service Streaming Platform ■ RESPONSIVE ■ FLEXIBLE ■ EXTENSIBLE
- 17. Moving from Pre-Streamingto Event Driven Architectures Request-Response vs Event-Driven: You Need Both! Message Driven & Event Sourced / quantyca.it | confluent.io Request-Response Event-Driven
- 18. Event Centric Thinking AllYour Data is a Stream of Events Message Driven & Event Sourced / quantyca.it | confluent.io
- 19. The Event StreamingPlatform Message Driven & Event Sourced / quantyca.it | confluent.io
- 20. Why Didn’t itWork Before? Past solutions turned out to be insufficient Message Driven & Event Sourced / quantyca.it | confluent.io Message-Oriented Middleware No persistence Single point of failure Not fault tolerant Cannot order messages Cannot process messages in flight Order of magnitude lower throughput No “replay” functionality EAI & ESBs Not event-oriented Fragile and bespoke Weak transformation capabilities ETL Slow and batch oriented Point-to-point, not publish & subscribe Not a true infrastructure platform
- 21. The World HasChanged Apache Kafka is at the core of these industry trends and, ultimately, the digitalisation of every business Message Driven & Event Sourced / quantyca.it | confluent.io Microservices Mobile Machine Learning Internet of Things
- 22. Message Driven &Event Sourced / quantyca.it | confluent.io The Event Streaming Platform We saw that picture before, what’s different now? ■ APACHE KAFKA CAN STORE EVENTS Kafka can store events for months, years, or indefinitely if useful. The limit is the amount of physical storage available on Kafka brokers. Events are stored in order on a per partition basis and they can be replayed and reconsumed (in order) whenever necessary. ■ EVENTS ARE STORED IN ORDER (PER PARTITION) Order of events is preserved and the ability to operate on a ‘stream’ level, also via Kafka Streams APIs or KSQL, provides a powerful way to simplify data integration, data cleansing, data transformations tasks as well as use Kafka to exchange and publish entire datasets. With Kafka, events are first class citizens, but also the full history and state is available to applications (it’s like having a database and a messaging system at the same time). ■ KAFKA CAN EASILY OPERATE A COMPANY SCALE Scalability, fault-tolerance, elasticity, and multi-tenancy are built-in capability.
- 23. Data Streaming andMaturity Stages Message Driven & Event Sourced / quantyca.it | confluent.io Value Maturity (Investment & time) 2 Enterprise Streaming Pilot / Early Production LOB(s) Pilot(s); Small teams experimenting; pub/sub / integration. → 1-3 use cases quickly moved into Production - but fragmented. Pub + Sub Store Process 5 All data in the organization managed through a single Streaming Platform. → Digital natives / digital pure players - probably using Machine Learning & AI (Relational databases - redundant). Central Nervous System 1 Developer Interest Developer downloads Kafka & experiments (15 mins on laptop). Pre-Streaming Legacy systems. Batch processes; → Complex → Slow / Silo’d → Expensive 4 Global Streaming Streaming Platform managing majority of mission critical data processes, globally, with multi-datacenter replication across on-prem and hybrid clouds. In parallel with other Big Data infrastructure. 3 SLA Ready, Integrated Streaming Multiple mission critical use cases in production, with; scale, DR & SLAs. → Streaming clearly delivering business value, with C-suite visibility. Projects Platform
- 24. Designing Event-Driven Systems MessageDriven & Event Sourced / quantyca.it | confluent.io In this book Ben explains how service-based architectures and stream processing tools such as Apache Kafka can help you build business-critical systems. ● Understand why replayable logs such as Kafka provide a backbone for both service communication and shared datasets ● Explore how event collaboration and event sourcing patterns increase safety and recoverability with functional, event-driven approaches ● Apply patterns including Event Sourcing and CQRS, and how to build multi-team systems with microservices and SOA using patterns such as “inside out databases” and “event streams as a source of truth” ● Build service ecosystems that blend event-driven and request-driven interfaces using a replayable log and Kafka's Streams API ● Scale beyond individual teams into larger, department- and company-sized architectures, using event streams as a source of truth The book is available for free in PDF from the Confluent website! https://www.confluent.io/designing-event-driven-systems
- 25. Message Driven &Event Sourced / quantyca.it | confluent.io Royal Bank of Canada (RBC) We saw that picture before, what’s different now? https://www.confluent.io/customers/rbc ■ 16 Million Clients ■ 35 Countries ■ 30+ Use-cases ■ 50+ apps ■ 10+ lines of businesses “Kafka transforms even the most basic of initiatives. Adoption of Kafka at RBC has been massive and organic. Within the first six weeks after our launch of Kafka, we had 37 teams asking to use Kafka for various projects and initiatives.” -- Kerry Joel, Senior Director, Product Innovation, Data and Analytics Digital Marketi ng Securit y Consumer Credit Services SaaS Corporate Real Estate Investor Services Treasury Services …. FraudData Wareho use Microservices https://www.youtube.com/watch?v=WTxmHHJcHRc
- 26. The Streaming Platform TechnicalCapabilities Message Driven & Event Sourced / quantyca.it | confluent.io Publish & Subscribe Store Process
- 27. Takeaways Message Driven &Event Sourced / quantyca.it | confluent.io
- 28. In conclusione Elementi chiave 1.Service integration: esb, microservizi, api manager 2. Service portfolio 3. Competence portfolio 4. Piano di migrazione evolutivo Prossimi passi 1. Scalabilità 2. Governance non solo a livello dati ma anche a livello servizi Message Driven & Event Sourced / quantyca.it | confluent.io
- 29. Corso Milano, 45/ 20900 Monza (MB) T. +39 039 9000 210 / F. +39 039 9000 211 / @ info@quantyca.it www.quantyca.it