Download to read offline
Uploaded byScyllaDB
Designing an Energy-efficient Architecture for Geo Databases by Yichen Wei
The cost and efficiency of IP geolocation data services depends on careful system design. This talk covers how storage architecture, data compression, network optimization, memory layout, and CPU-level tuning all play a role. We also show how techniques like NUMA awareness, SIMD, zero-copy, and zero-serialization enable higher traffic with lower resource use.
Designing an Energy-efficient Architecture for Geo Databases by Yichen Wei
- 1. A ScyllaDB Community Designingan Energy-efficient Architecture for Geo Databases Yichen Wei Engineer Manager
- 2. Yichen Wei Engineer Manager ■HA Distributed Platform domains with machine learning and GenAI ■ Evangelist of HPC, HTAP, Rust and FP/CT ■ WASM/WASI, FaaS
- 3.
- 4. Challenge ■ Critical Service ●Low latency ● High availability ● Backward and forward compatibility ■ Cost ● Data size ● Query performance
- 5.
- 6. Why not existinglocal databases? Why not using sqlite, duckdb or rocksdb? ■ Different user cases ● Daily/Weekly full size dump ● No update operations ■ Performance and cost concerns ● Read/Write amplification and memory footprint ● Long tail lookups
- 7.
- 8.
- 9.
- 10. Optimization 2: Parsing ■Encoding ● Dictionary encoding - Country, Region, City, Metro, Zip… ● Delta encoding - longitude and latitude
- 11. Optimization 2: Parsing ■Memory layout and alignment
- 12. Optimization 2: Parsing ■Memory layout and alignment
- 13.
- 14. Optimization 3: Lookup ChunkSize ■ Graviton instance ● L1: 64K ● L2: 1M
- 15.
- 16.
- 17. Optimization 3: Lookup SIMD ■truth table ● 16 bytes vector - 4 IPv4 value IP1 IP2 IP3 IP4 T T T T T T T F T T F F T F F F F F F F
- 18. Optimization 3: Lookup SIMD ■Does Golang have SIMD support? ● Yes and No
- 19. Optimization 3: Lookup SIMD ■Does Golang have SIMD support? ● ●
- 20. Optimization 3: Lookup SIMD ■Cgo is not Go!
- 21. Optimization 3: Lookup SIMD ■https://github.com/alivanz/go-simd ■ `linkname` directive to call C code, bypass CGO
- 22.
- 23.
- 24. Optimization 3: Lookup- One More Thing premature optimization is the root of all evil
- 25. Optimization 3: Lookup- One More Thing
- 26. Optimization 3: Lookup- One More Thing
- 27. Optimization 3: Lookup- One More Thing
- 28. Optimization 3: Lookup- One More Thing
- 29. Optimization 3: Lookup- One More Thing Understand compiler first before optimizations!
- 30. Looking Forward ■ Compression& encoding ■ Histogram Distribution ■ Data prefetching ■ Change Language
- 31. Takeaways ■ No oneknows your data better than you ■ Don’t be afraid to build the infrastructure for your data ■ Optimization is fun ● … and hard ■ Golong doesn’t support SIMD well
- 32. Thank you! Let’sconnect. Yichen Wei yichen.wei@disney.com