Uploaded byconfluent
Data integration with Apache Kafka
The document presents a discussion on data integration with Apache Kafka, covering design considerations, the Kafka solution, and Kafka Connect. It highlights the architecture of a streaming platform, reliability, data handling semantics, and the benefits of using Kafka Connect for simplified data integration. The document also includes a case example of a MySQL to Hive pipeline setup.
Data integration with Apache Kafka
- 1. 1Confidential Streaming Data Integration withApache Kafka Presented by: David Tucker | Dir. Partner Engineering partners@confluent.io david@confluent.io
- 2. 3Confidential Today’s Discussion • Theevolving world of data integration • Design Considerations • The Kafka Solution • Kafka Connect • Logical Architecture • Core Components and Execution Model • Connector Examples • Wrap Up and Questions
- 3.
- 4.
- 5.
- 6.
- 7.
- 8. 12Confidential Explosion of OperationalData Stores and Processing Frameworks
- 9. 13Confidential Abstract View: ManyAd Hoc Pipelines Search Security Fraud Detection Application User Tracking Operational Logs Operational Metrics Hadoop App Data Warehouse Espresso Cassandra Oracle Databases Storage Interfaces Monitoring App Databases Storage Interfaces
- 10. 14Confidential Re-imagined Architecture: StreamingPlatform with Kafka ü Distributed ü Fault Tolerant ü Stores Messages Search Security Fraud Detection Application User Tracking Operational Logs Operational MetricsEspresso Cassandra Oracle Hadoop App Monitoring App Data Warehouse Kafka ü Processes Streams Kafka StreamsKafka Streams
- 11. 16Confidential Design Considerations :These Things Matter • Reliability and Delivery Semantics – Losing data is (usually) not OK. • Exactly Once vs At Least Once vs (very rarely) At Most Once • Timeliness • Push vs Pull • High Throughput, Varying Throughput • Compression, Parallelism, Back Pressure • Data Formats • Flexibility, Structure • Security • Error Handling
- 12.
- 13.
- 14. 20Confidential Introducing Kafka Connect Simplified,scalable data integration via Apache Kafka
- 15.
- 16. 22Confidential Kafka Connect :Separation of Concerns
- 17. 23Confidential Kafka Connect: LogicalModel Kafka Connect Apache Kafka Brokers Schema Registry
- 18. 24Confidential How is Connectdifferent than a producer or consumer? • Producers and consumers enable total flexibility; data is published and processed in any way • This flexibility means you do everything yourself. • Kafka Connect’s simple framework allows : • developers to create connectors that copy data to/from other systems • operators/users to use said connectors just by writing configuration files and submitting them to Connect -- no code necessary • community and 3rd-party engineers to build reliable plugins for common data sources and sinks • deployments to deliver scalability, fault tolerance and automated load balancing out-of- the-box
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25.
- 26.
- 27. 34Confidential Connector Hub: http://www.confluent.io/product/connectors •Confluent-supported connectors (included in CP) • Partner/Community-written connectors (just a sampling) JDBC
- 28. 35Confidential Kafka Connect Example:MySQL to Hive Pipeline • Blog at http://confluent.io/blog/how-to-build-a-scalable-etl-pipeline-with-kafka-connect/
- 29. 36Confidential MySQL to HivePipeline : Step by Step • Configure the JDBC Source Connector with the MySQL details • User authentication • Tables to replicate; polling interval for change-data-capture • Configure the HDFS Sink Connector with Hadoop Details • Target HDFS directory • Hive metastore details • Partitioning details (optional) • Watch it go !!! • What you can’t see • Source and Sink scalability • Table metadata changes are captured in Schema Registry
- 30.
- 31.