Downloaded 18 times
Uploaded byPivotalOpenSourceHub
#GeodeSummit: Democratizing Fast Analytics with Ampool (Powered by Apache Geode)
The document discusses the democratization of fast analytics using Ampool, powered by Apache Geode, to tackle challenges such as disparate tools and long time to insights. It emphasizes the use of a smart distributed in-memory object store and emerging technologies like storage class memory to enhance analytics speed and efficiency. The authors propose a pluggable distributed memory layer to facilitate managed flexibility and support multiple analytics stakeholders without altering user experience.
#GeodeSummit: Democratizing Fast Analytics with Ampool (Powered by Apache Geode)
- 1. Democratizing Fast Analyticswith Ampool (Powered by Apache Geode-incubating) Avinash Dongre and Robert Geiger, Ampool Inc.
- 2. " " " Analytics # # # # Apps Multi-Device Testing $ % & | ) * Analytics needs to work in CLOSED LOOPwith AppsAnalytics Needs to be Faster!
- 3. " " " Analytics # # # # Apps Multi-Device Testing $ % & | ) * What are the CHALLENGES? ⚠Many data users/ stakeholders ⚠ Disparate tools & processing needs ⚠ Long time to insights
- 4.
- 5.
- 6.
- 7. AnalyticsIngest App UseETL … FAST OBJECT ACCESS across the pipeline ## # , , - , … . Data Architect Data Developers . . Business Analysts Data Scientists .
- 8. What ENABLERS can help here? In-Memory Fabric Technology • Apache Geode! •Flexible, stable, and proven distributed in-memory technology New memory technologies and fast network fabrics • Storage Class Memory • low latency, high throughput, persistent • Initially exposed via file system interface • Regular or memory mapped
- 9. Emerging Storage Class Memory (SCM) is DISRUPTIVE Challenges the value proposition of in-memory solutions Near DRAM latency and throughput at lower cost Based on one of several types of memory technology • MRAM (magnetic) • ReRAM(resistive) • FRAM (ferroelectric), PCM (phase change) • 3D-XPointTM (Intel/Micron) Accessible via Java and C/C++ libraries • Mnemonic (Java) • Pmem.io (C++)
- 10.
- 11. In-Memory Technology CHALLENGES Line between memory and storage is blurring File systems getting really fast, so the speed gap is closing • SCM File Systems will also be low latency • File system overhead still limits latency improvements •Before: disk based vs. in-memory • After: file system vs. byte addressable object store Managing multiple layers and types of memory
- 12. Fast Closed Loop Analytics, Powered by a Smart, Distributed In-Memory Fabric… High throughput and large data handling matters • Throughput, latency, and capacity: •each pipeline stage values these differently Common interfaces, multiple region types • Meet the needs of many types of best of breed engines Managing multiple layers of memory and storage • Speed (latency, throughput) differentiator will diminish More classifications for data now • Hot, cold => hot, warm, lukewarm, cold
- 13.
- 14. What Matters for App, DB, and Compute? The flexibility, suitability, and ease of use of the interfaces Memory & storage are managed transparently to provide QoS The service guarantee abstractions are provided Conflicts are managedand prevented Freeing developersfrom re-inventing the wheel
- 15.
- 16.
- 17. Smart Distributed In-Memory Object Store … for MANAGED FLEXIBILITY... + 3DXPointTM ...... ✅ Flexible regions and interfaces for ‘Best of breed’ engines ✅ Extensible Core ✅ Pluggable stores
- 18. AnalyticsIngest App UseETL …and FAST OBJECT ACCESS across the pipeline # ## , , - , … . Data Architect Data Developers . . Business Analysts Data Scientists .
- 19.
- 20.
- 21. No change in data application code Config. changes only No change in user experience Performance benefits No added hassles Current mgmt. tools …and deliver VALUE to all Analytics stakeholders . Data Architect Data Developers . .Business Analysts Data Scientists . . Data Admins Infra/ Sys Admins .
- 22. Contributing Back Plan for contributions back to Apache Geode: • Storage plug-ability layer • Off-heap memory plug-ability •SCM plugin (Mnemonic) • Impersonation support for security • Region type plug-ability
- 23.