Download as PDF, PPTX
Uploaded byTrieu Nguyen
Lambda Architecture and open source technology stack for real time big data
The document discusses the Lambda Architecture, which is an approach for building data systems to handle large volumes of real-time streaming data. It proposes using three main design principles: handling human errors by making the system fault-tolerant, storing raw immutable data, and enabling recomputation of results from the raw data. The document then provides two case studies of applying Lambda Architecture principles to analyze mobile app usage data and process high-volume web logs in real-time. It concludes with lessons learned, such as studying Lambda concepts, collecting any available data, and turning data into useful insights.
Lambda Architecture and open source technology stack for real time big data
- 1. Lambda Architecture and OpenSource Tools for Real-time Big Data ● Concepts & Techniques “Thinking with Lambda” ● Case studies in Practice Trieu Nguyen - http://nguyentantrieu.info or @tantrieuf31 Principal Engineer at eClick Data Analytics team, FPT Online All contents and thoughts in this slide are my subjective ideas and compiled from Communities
- 2. Just a littleintroduction ● 2008 Java Developer, developed Social Trading Network for a small startup (Yopco) ● 2011 worked at FPT Online, software engineer in Banbe Project, Restful API for VnExpress Mobile App ● 2012 joined Greengar Studios in 6 months, scaling backend API mobile games (iOS, Android) ● 2013 back to FPT Online, R&D about Big Data & Analytics, developing the new core Analytics Platform (on JVM Platform)
- 3. Contents for thistalk ● ● ● ● ● ● ● ● The lessons from history Problems In Practice What is the Lambda Architecture? Why lambda architecture for real-time big data ? Open Source Technology Stack Lambda in Practice (Mobile Data and Web Data) Lessons I have learned Questions & Answers
- 4. History ? The bestway to predict the future is looking at the past and now ?
- 5. Big data isa buzzword for old problems
- 6.
- 7.
- 8.
- 10. Big Data +Old History http://www.youtube.com/watch?v=tp4y-_VoXdA
- 11. This is BigDATA This is most valuable things!
- 13. We can't solveproblems by using the same kind of thinking we used when we created them. Albert Einstein Think more with Lambda and Reactive
- 16. Where Big Data canbe used
- 18. BBC Horizon 2013The Age of Big Data http://www.youtube.com/watch?v=RE0ITQ7XQjM
- 23. Google’s mission isto organize the world’s information and make it universally accessible and useful.
- 25.
- 27. How did Googlescale their search engine ? How does Hadoop really work ?
- 29.
- 30. Trends of Nowand the Future MapReduce Programming Reactive Programming Functional Programming Streaming Computation => All just the special cases of Lambda ● ● ● ●
- 32. So what isthe λ (Lambda) Architecture ?
- 35. the Lambda Architecture: ●apply the (λ) Lambda philosophy in designing big data system ● equation “query = function(all data)” which is the basis of all data systems ● proposed by Nathan Marz (http://nathanmarz.com/), a software engineer from Twitter in his “Big Data” book. ● is based on three main design principles: ○ human fault-tolerance – the system is unsusceptible to data loss or data corruption because at scale it could be irreparable. (BUGS ?) ○ data immutability – store data in it’s rawest form immutable and for perpetuity. (INSERT/ SELECT/DELETE but no UPDATE !) ○ recomputation – with the two principles above it is always possible to (re)-compute results by running a function on the raw data.
- 36. Lambda In Practice 2case studies from my experiences
- 37. Case Study 1: MobileData Monitor API Backend + System KPI
- 38. Problem: Inside “mobile data”, What'sthe most valuable piece of information
- 39. I applied “Lambda” here Backend Systemfor mobile app
- 40. Web vs MobileApp Web Visitors Visits Pageviews Events Mobile App Users Sessions Events
- 41. Metrics: Cause andEffect ● ● ● ● ● ● ● Screen Size => App Design, UI/UX, Usability App version => Deployment, Marketing Connectivity => Code, User Experience Location => Marketing, User Behaviour OS => Marketing, Cost, Development Memory => User Experience Feature Session => How to engage app users
- 42. The data andthe size, not too big for a small startup! Where is the lambda ? I used Groovy + GPars (Groovy Parallel Systems) + MongoDB for fast parallel computation (actor model) on statistical data http://gpars.codehaus.org/ The GPars framework offers Java developers intuitive and safe ways to handle Java or Groovy tasks concurrently. Support: ● ● ● ● ● ● ● ● Dataflow concurrency Actor programming model CSP Agent - an thread-safe reference to mutable state Concurrent collection processing Composable asynchronous functions Fork/Join STM (Software Transactional Memory)
- 43. Mobile Apps =>Backend APIs => Statistics => Find the Trends & Insights?
- 44. Reactive Data Analytics for MobileApps It means real-time recommendation by: ➔ context (location, time) ➔ user profile (preferences, level, ...)
- 45. Big Data onSmall Devices: Data Science goes Mobile http://strataconf.com/strata2013/public/schedule/detail/27605
- 46. Case Study 2: WebData ● Real-time Data Analytics ● Monitoring Stream Data (Reactive) http://eclick.vn
- 47. at eClick wemust check campaigns in near-real-time (seconds) ! at eClick we have 30~40 GB Logs in Stream 10~20 GB Bandwidth just for tracking user actions (click, impression,...) in ONE day ! at eClick we have many types of log (video, web, mobile, system logs, ad-campaign, articles, … )
- 50. “lambda architecture” proposed by@nathanmarz
- 51. Internet Netty Http Server TCP Connection Kafka AkkaWorkers Hadoop Tools Storm Redis Redis KPI Report the open-source lambda architecture at eClick
- 52. The big-data technologystack ● Netty (http://netty.io/) a framework using reactive programming pattern for scaling HTTP system easier, by JBoss http://www.jboss.org ● Kafka (http://kafka.apache.org/) a publish-subscribe messaging rethought as a distributed commit log, open sourced by Linkedin ● Storm (http://storm-project.net/) the framework for distributed realtime computation system, by Twitter ● Redis (http://redis.io/) a advanced key-value in-memory NoSQL database, all fast statistical computations in here. ● Groovy for scripting layer on JVM, ad-hoc query on Redis ● Hadoop ecosystem: HDFS, Hive, HBase for batch processing ● RxJava https://github.com/Netflix/RxJava a library for composing asynchronous and event-based programs ● Hystrix https://github.com/Netflix/Hystrix : for Latency and Fault Tolerance for Distributed Systems
- 53. My new ideasfor the future Connecting the active functor pattern + reactive programming + stream computation + in-memory computing to make: ● real-time data analytics easier ● better recommendation system ● build more profitable in big data More Information: ● http://activefunctor.blogspot.com/ (a special case of Lambda that actively search best connections to form optimal topology) - from ideas when internship at DRD with my advisor. ● Can a function be persistent (stored as data), distributed in a cluster (cloud), reactive to right data (best value in network) ? ● http://www.reactivemanifesto.org/ (reactive pattern)
- 54. Lessons What I havelearned from Lambda and Big Data World
- 56. What I havelearned ● ● ● ● ● Study about lambda and read some books Ask questions=> analytics=> Profit & Value Collect any data you can, learn inside ! Implement it! Just right tools for right jobs. Turn your data into the things everyone can "look & feel"
- 57.
- 58. Study the “lambda” Istudied Haskell in 2007 with Dr.Peter Gammie http://peteg.org/ when internship at DRD (a non-profit organization). ● Imperative programs will always be vulnerable to data races because they contain mutable variables. ● There are no data races in purely functional languages because they don't have mutable variables.
- 59.
- 62. Improve your businessknowledge ! => read the Behavioral Economics Books http://www.goodreads.com/shelf/show/behavioral-economics
- 63.
- 64. Use your imaginationis more than just knowledge you have
- 65. Think more aboutButterfly Effect!
- 66. Z; om A to fr lget you you il “Logic w n will get in ginatio - Albert Einste ima .” ywhere ever Use you r with da imagination ta just log analytics, not ic Learn Data Visualization
- 67. Questions & Answers Thelink of this slide is here: ● http://nguyentantrieu.info/blog/lambda-architecture-andopen-source-tools-for-real-time-big-data/ More useful resources: ● http://nguyentantrieu.info/blog ● http://www.mc2ads.com