Download as PDF, PPTX
Uploaded byAmir Sedighi
Opensource Frameworks and BigData Processing
The document discusses using open-source technologies to build a big data processing platform on commodity machines. It outlines the challenges of big data including the volume, velocity and variety of data being created. It then describes the Hadoop ecosystem as a solution, including its use of MapReduce and various Apache projects for tasks like storage, transfer, search, messaging, logging, stream processing and machine learning.
Opensource Frameworks and BigData Processing
- 1. Big-Data Processing utilizing Open-Source Technology Stack By Amir Sedighi http://www.linkedin.com/in/amirsedighi @amirsedighi Linux and Ubuntu 14.10 Release Conf 1
- 2. References ● http://www.slideshare.net/BernardMarr/140228-big-data-slide-share?qid=017848e 2-9e2a-4dc3-963c-52b6a90fba2a&v=default&b=&from_search=1 ● http://www.forbes.com/fdc/welcome_mjx.shtml ● ZYMR Spark Your Real-Time Big Data Analytics Linux and Ubuntu 14.10 Release Conf 2 ● http://dataconomy.com ● https://datakulfi.wordpress.com/2013/03/27/big-data-open-source-technology-landsca pe/ ● http://www.slideshare.net/andrefaria/big-data-abc?qid=1ac97e4a-4acc-460a-b3f8 -9122f7210440&v=qf1&b=&from_search=12 ● https://wiki.apache.org/hadoop/PoweredBy
- 3. Data Explosion Linuxand Ubuntu 14.10 Release Conf 3
- 4. Data Explosion Linuxand Ubuntu 14.10 Release Conf 4
- 5. ● Big-Data isthat everything we do is increasingly leaving a digital trace which we (or others) can gather, use and analyze. – Data Providers ● Business Companies ● People Linux and Ubuntu 14.10 Release Conf 5
- 6. Volume, Velocity, Variety ● “There was 5 exabytes of information created between the dawn of civilization through 2003, but that much information is now created every 2 days, and the pace is increasing.” Eric Schmidt Linux and Ubuntu 14.10 Release Conf 6
- 7. Big-Data Processing Linuxand Ubuntu 14.10 Release Conf 7
- 8. How to providea Big-Data processing platform using commodity machines? Linux and Ubuntu 14.10 Release Conf 8
- 9. Vertical or Horizontal? Linux and Ubuntu 14.10 Release Conf 9
- 10. Scale Up vsScale Out Linux and Ubuntu 14.10 Release Conf 10
- 11. Scale Up vsScale Out Linux and Ubuntu 14.10 Release Conf 11
- 12. Big-Data Processing Open-SourceTechnology Stack Linux and Ubuntu 14.10 Release Conf 12
- 13. Map-Reduce Linux andUbuntu 14.10 Release Conf 13
- 14. Hadoop Framework Linuxand Ubuntu 14.10 Release Conf 14
- 15. Apache Hadoop MainProjects Linux and Ubuntu 14.10 Release Conf 15
- 16. Linux and Ubuntu14.10 Release Conf 16
- 17. Data Stores Linuxand Ubuntu 14.10 Release Conf 17 ● Data Stores – KeyValue – Graph – Columnar – Document Store – In Memory
- 18. Data Transfer Linuxand Ubuntu 14.10 Release Conf 18 ● Apache Flume ● Apache Sqoop
- 19. Search Linux andUbuntu 14.10 Release Conf 19 ● Elasticsearch ● Apache SolR
- 20. Messaging and Queuing Linux and Ubuntu 14.10 Release Conf 20 ● Apache Kafka ● ZeroMQ
- 21. Log Management Linuxand Ubuntu 14.10 Release Conf 21 ● ELK ● Logstash ● FluentD
- 22. Stream Processing Linuxand Ubuntu 14.10 Release Conf 22 ● Apache Storm ● Apache Samza ● Apache Spark
- 23. Machine Learning ●Apache Mahout Linux and Ubuntu 14.10 Release Conf 23 ● MLLib ● GraphX
- 24. Questions? Linux andUbuntu 14.10 Release Conf 24