Uploaded bySenthil Kumar
Hadoop Ecosystem Architecture Overview
The document provides an overview of Hadoop's distributed cluster architecture, emphasizing its scalability through MapReduce patterns. It contrasts the performance of traditional programming with Hadoop's bootstrap processes, noting that Hadoop excels in handling large datasets, particularly in the petabyte range. Additionally, it mentions various associated Apache projects like Sqoop, Flume, Oozie, Hive, and others that enhance Hadoop's capabilities.
Hadoop Ecosystem Architecture Overview
- 1. Hadoop Technologies Architecture Overview @senthil245 Mail- senthil245@gmail.com
- 2. DISTRIBUTED CLUSTER ARCHITECTURE:MASTER/SLAVE
- 3.
- 4.
- 5. WHEN MAPREDUCE Since theMapReduce is running within a cluster of computing nodes, the architecture is very scalable. • In other words, if the data size is increased by the factor of x, the performance should be still constant if we are adding a predictable/fixed factor of y. The graph on the right is illustrating the relationship between the size of the data (xaxis) and processing time (y-axis). •The blue color curve is the process using traditional programming. On the other hand, the black color curve is the process using Hadoop. When the data size is small, traditional programming is better performance because the bootstrap of Hadoop is expensive (Copy the data within the cluster, inter-nodes communication, etc.). Once the data size is big enough, the penalty of the Hadoop bootstrap becomes invisible. •Hence Hadoop is best suited for Big Data crunching ideally in terms of petaBytes and is not suited for implementing common data integration patterns
- 7.
- 8.
- 9.
- 10.
- 11. APACHE OOZIE –WORKFLOW SCHEDULER (CHECK AZKABAN & LINKEDIN OPENSOURCE)
- 12. PIG AND HQL(DO NOT USE HQL)
- 13. APACHE S4 (STREAMPROCESSING)(ALSO CHECK KAFKA AND STORM)
- 14. APACHE ZOOKEEPER SERVICE(ALSO CHECK APACHE HUE)
- 15.
- 16.