Download to read offline
Uploaded byHostedbyConfluent
Cloud-Based Event Stream Processing Architectures and Patterns with Apache Kafka | Israel Ekpo, Microsoft
The document discusses cloud-based event stream processing with Apache Kafka, focusing on deployment options, integration patterns, and best practices for success. It highlights the importance of understanding the environment, choosing between fully-managed and self-managed solutions, and the role of different architectural patterns. The conclusion emphasizes the need for the right team and clear objectives to effectively implement these technologies.
Cloud-Based Event Stream Processing Architectures and Patterns with Apache Kafka | Israel Ekpo, Microsoft
- 1. Cloud-Based Event Stream Processing Architectures and Patternswith Apache Kafka KAFKA SUMMIT EUROPE 2021 Israel Ekpo
- 2.
- 3. Outline Tips for success Patternsfor the Cloud Resources Follow up and Next Steps
- 4.
- 5.
- 6.
- 7. Where, Why, How,What, Who
- 8.
- 9. Where, Why, How,What, Who
- 10. Where, Why, How,What, Who
- 11. Cloud-Based Event Stream Processingwith Apache Kafka
- 12. • Brokers • SchemaRegistry • Kafka Streams, KSQLDB • Kafka Connect Ecosystem • Kafka Client Ecosystem
- 13. • Operations, DataEngineers, Data Scientists, Solutions Architects • System Architecture, DevOps Pipeline, Agile Methodologies • Software Product or Experience
- 14.
- 15. •Architectural Complexity •Operating Complexity •Lackof Trust and Desire for Control
- 16. Patterns for theCloud
- 17.
- 18.
- 19.
- 20. Fully-Managed on Azure ConfluentCloud Integration on Azure Azure Event Hubs
- 21. Self-Managed On Azure ConfluentPlatform on Virtual Machines Confluent Platform on Kubernetes (via Confluent Operator) Apache Kafka on Azure HDInsight
- 22.
- 23.
- 24.
- 25. Hybrid Scenarios forFlexibility Dynamic Capacity Needs (Kafka Streams, KSQLDB, etc) Some Components not available in Fully-Managed Offering Custom Scenarios
- 26.
- 27.
- 28.
- 29.
- 30.
- 31. Integration Options via ConnectFramework Source Connectors Sink Connectors https://www.confluent.io/hub/#azure
- 32. Summary Event Stream Meshesvia KSQLDB Orchestration vs Choreography Dynamic Capacity Stretch Clusters Tiered Storage
- 33. Summary Mindset Change People Deployment Options Awarenessof Environment and Integration Options
- 34. Conclusion Get the rightteam in place Understand your environment and what is possible Good process in place Clear goals and objectives Reach out
- 35.
- 36.
- 37.
Editor's Notes
- #12 We need to understand the Where, Why, How, What, Who
- #32 Azure Service Bus (Source) Azure Event Hubs (Source) Azure Blob Storage (Source and Sink) Azure Cosmos DB Connector (Source and Sink) Azure IoT Hub Connector (Source and Sink) Azure Data Lake Storage Gen1 (Source and Sink) Azure Data Lake Storage Gen2 (Sink) Azure Cognitive Search (Sink) Azure Functions (Sink) Azure Synapse Analytics (Sink) Azure Log Analytics (Sink) Azure Data Explorer (Sink)