Downloaded 70 times
![Document Model • Matches Application Objects • Flexible • High performance { "customer_id" : 123, "first_name" : ”John", "last_name" : "Smith", "address" : { "street": "123 Main Street", "city": "Houston", "state": "TX", "zip_code": "77027" } policies: [ { policy_number : 13, description: “short term”, deductible: 500 }, { policy_number : 14, description: “dental”, visits: […] } ] }](https://image.slidesharecdn.com/mongodb-at-scale-150527195531-lva1-app6891/75/Scaling-MongoDB-6-2048.jpg)
![mongod log files Sun Jun 29 06:35:37.646 [conn2] query test.docs query: { parent.company: "22794", parent.employeeId: "83881" } ntoreturn:1 ntoskip:0 nscanned:806381 keyUpdates:0 numYields: 5 locks(micros) r:2145254 nreturned:0 reslen:20 1156ms](https://image.slidesharecdn.com/mongodb-at-scale-150527195531-lva1-app6891/75/Scaling-MongoDB-17-2048.jpg)
Uploaded byMongoDB
Scaling MongoDB
The document discusses strategies for scaling MongoDB, emphasizing schema design, indexing, and the use of the WiredTiger storage engine to improve performance and reduce storage needs. It provides tips for optimizing operations, such as vertical and horizontal scaling, and highlights the importance of monitoring and using the MongoDB Management Service (MMS) for efficient database management. Key points include schema patterns, indexing strategies, and best practices for selecting effective shard keys to enhance database performance.
In this document
Powered by AI
Overview of achieving scale with MongoDB with a focus on optimization tips, different scaling types, and operations team scaling.
Discussion on schema design for MongoDB, highlighting flexible document models, denormalization, and design patterns.
Focus on optimization through indexing including B-Tree indexes, common mistakes, and strategies to improve query performance.
Importance of monitoring, setup of MongoDB Management Services (MMS) for automated backup and monitoring.
Benefits of the WiredTiger storage engine with increased performance (7x-10x) and storage efficiency (50%-80% less storage).
Vertical and horizontal scaling details, including factors affecting scaling, sharding overview, and shard key characteristics.
Strategies for scaling operations teams effectively, showcasing the role of MongoDB Management Service in simplifying management tasks.
Scaling MongoDB
- 2. Achieving Scale withMongoDB Thomas Boyd Director, SolutionsArchitecture, MongoDB
- 3. MongoDB Scaling 0 5,000 10,000 15,000 20,000 25,000 30,000 1 23 4 5 6 7 8 Operations/Second Number of Nodes MongoDB Cluster Throughput
- 4. Agenda • Optimization Tips –Schema Design – Indexes – Monitoring – WiredTiger • Vertical Scaling • Horizontal Scaling • Scaling your Operations Team
- 5.
- 6. Document Model • MatchesApplication Objects • Flexible • High performance { "customer_id" : 123, "first_name" : ”John", "last_name" : "Smith", "address" : { "street": "123 Main Street", "city": "Houston", "state": "TX", "zip_code": "77027" } policies: [ { policy_number : 13, description: “short term”, deductible: 500 }, { policy_number : 14, description: “dental”, visits: […] } ] }
- 7. The Importance ofSchema Design • Very different from RDBMS schema design • MongoDB Schema: – denormalize the data – create a (potentially complex) schema with prior knowledge of your actual (not just predicted) query patterns – write simple queries
- 8. Real World Example Productcatalog for retailer selling in 20 countries { _id: 375, en_US: { name: …, description: …, <etc…> }, en_GB: { name: …, description: …, <etc…> }, fr_FR: { name: …, description: …, <etc…> }, fr_CA: { name: …, description: …, <etc…> }, de_DE: …, <… and so on for other locales …> }
- 9. Not a GoodMatch for Access Pattern Actual application queries: db.catalog.find( { _id: 375 }, { en_US: true } ); db.catalog.find( { _id: 375 }, { fr_FR: true } ); db.catalog.find( { _id: 375 }, { de_DE: true } ); … and so forth for other locales
- 10. Inefficient use ofresources Data in RED are being used. Data in BLUE take up memory but are not in demand. { _id: 375, en_US: { name: …, description: …, <etc…> }, en_GB: { name: …, description: …, <etc…> }, fr_FR: { name: …, description: …, <etc…> }, fr_CA: { name: …, description: …, <etc…> }, de_DE: …, de_CH: …, <… and so on for other locales …> } { _id: 42, en_US: { name: …, description: …, <etc…> }, en_GB: { name: …, description: …, <etc…> }, fr_FR: { name: …, description: …, <etc…> }, fr_CA: { name: …, description: …, <etc…> }, de_DE: …, de_CH: …, <… and so on for other locales …> }
- 11. Consequences of SchemaRedesign • Queries induced minimal memory overhead • 20x as many products fit in RAM at once • Disk IO utilization reduced • Application latency reduced { _id: "375-en_GB", name: …, description: …, <… the rest of the document …> }
- 12. Schema Design Patterns •Pattern: pre-computing interesting quantities, ideally with each write operation • Pattern: putting unrelated items in different collections to take advantage of indexing • Anti-pattern: appending to arrays ad infinitum • Anti-pattern: importing relational schemas directly into MongoDB
- 13. Schema Design Resources •Data Modeling Deep Dive, 2pm Robertston Auditorium 1 • Blog series, "6 rules of thumb" – Part 1: http://goo.gl/TFJ3dr – Part 2: http://goo.gl/qTdGhP – Part 3: http://goo.gl/JFO1pI • Webinars, training, consulting, etc…
- 14.
- 15. B-Tree Indexes • Tree-structuredreferences to your documents • Single biggest tunable performance factor • Indexing and schema design go hand in hand
- 16. Indexing Mistakes andTheir Fixes • Failing to build necessary indexes – Run .explain(), examine slow query log, mtools, system.profile collection • Building unnecessary indexes – Talk to your application developers about usage • Running ad-hoc queries in production – Use a staging environment, use secondaries
- 17. mongod log files SunJun 29 06:35:37.646 [conn2] query test.docs query: { parent.company: "22794", parent.employeeId: "83881" } ntoreturn:1 ntoskip:0 nscanned:806381 keyUpdates:0 numYields: 5 locks(micros) r:2145254 nreturned:0 reslen:20 1156ms
- 18. mtools • http://github.com/rueckstiess/mtools • logfile analysis for poorly performing queries – Show me queries that took more than 1000 ms from 6 am to 6 pm: – mlogfilter mongodb.log --from 06:00 --to 18:00 --slow 1000 > mongodb-filtered.log
- 19. Indexing Strategies • Createindexes that support your queries! • Create highly selective indexes • Eliminate duplicate indexes with compound indexes – db.collection.ensureIndex({A:1, B:1, C:1}) – allows queries using leftmost prefix • Order index columns to support scans & sorts • Create indexes that support covered queries • Prevent collection scans in pre-production environments db.getSiblingDB("admin").runCommand( { setParameter: 1, notablescan: 1 } )
- 20.
- 21. JUST DO IT NOW INPRE-PROD/STRESS
- 22. MongoDB Management Services(MMS) Backup Monitoring Automation
- 23.
- 24.
- 25. Cloud Version ofMMS 1. Go to http://mms.mongodb.com 2. Create an account 3. Install one agent in your datacenter 4. Add hosts from the web interface 5. Enjoy!
- 26.
- 27. 7x-10x Performance, 50%-80%Less Storage How: WiredTiger Storage Engine • Same data model, same query language, same ops • Write performance gains driven by document-level concurrency control • Storage savings driven by native compression • 100% backwards compatible • Non-disruptive upgrade MongoDB 3.0MongoDB 2.6 Performance
- 28.
- 29. Factors: – RAM – Disk –CPU – Network We are Here to Pump you Up Primary Secondary Secondary Replica Set Primary Secondary Secondary Replica Set
- 30. Working Set ExceedsPhysical Memory
- 31. Real world Example •Status changes for entities in the business • State changes happen in batches – sometimes 10% of entities get updated – sometimes 100% get updated
- 32. Initial Architecture Sharded Cluster,4 shards backed by spinning disk Application / mongos mongod
- 33. Horizontal Scaling Rapidly growingbusiness means more shards Application / mongos …16 more shards… mongod
- 34. Vertical Scaling Scaling randomIOPS with SSDs Application / mongos mongod SSD
- 35. Before you addhardware.... • Make sure you are solving the right scaling problem • Remedy schema and index problems first – schema and index problems can look like hardware problems • Tune the Operating System – ulimits, swap, NUMA, NOOP scheduler with hypervisors • Tune the IO subsystem – ext4 or XFS vs SAN, RAID10, readahead, noatime • See MongoDB "production notes" page • Heed logfile startup warnings
- 36.
- 37. Sharding Overview Primary Secondary Secondary Shard 1 Primary Secondary Secondary Shard2 Primary Secondary Secondary Shard 3 Primary Secondary Secondary Shard N … Query Router Query Router Query Router …… Driver Application
- 38. Range Sharding mongod Read/Write Scalability KeyRange 0..100
- 39. Range Sharding Read/Write Scalability mongodmongod Key Range 0..50 Key Range 51..100
- 40. Sharding mongod mongod mongodmongod Key Range 0..25 Key Range 26..50 Key Range 51..75 Key Range 76.. 100 Read/Write Scalability
- 41. Shard Key characteristics •A good shard key has: – sufficient cardinality – distributed writes – targeted reads ("query isolation") • Shard key should be in every query if possible – scatter gather otherwise • Choosing a good shard key is important! – affects performance and scalability – changing it later is expensive
- 42. Beware of AscendingShard Keys • Monotonically increasing shard key values cause "hot spots" on inserts • Examples: timestamps, _id Shard 1 mongos Shard 2 Shard 3 Shard N [ ISODate(…), $maxKey )
- 43.
- 44. MongoDB Management Service(MMS) Scale EasilyMeet SLAs Best Practices, Automated Cut Management Overhead
- 45. Without MMS Example Deployment– 12 Servers Install, Configure 150+ steps …Error handling, throttling, alerts Scale out, move servers, resize oplog, etc. 10-180+ steps Upgrades, downgrades 100+ steps
- 46.
- 47. Common Tasks, Performedin Minutes • Deploy – any size, most topologies • Upgrade/Downgrade – with no downtime • Scale – add/remove shards or replicas, with no downtime • Resize Oplog – with no downtime • Specify users, roles, custom roles • Provision AWS instances and optimize for MongoDB
- 48. MonoDB at Scale 250MTicks/Sec 300K+ Ops/Sec 500K+ Ops/SecFed Agency Performance 1,400 Servers 1,000+ Servers 250+ Servers Entertainment Co. Cluster Petabytes 10s of billions of objects 13B documents Data Asian Internet Co.
Editor's Notes
- #45 MMS can do a lot for [ops teams]. Best Practices, Automated. MMS takes best practices for running MongoDB and automates them. So you run ops the way MongoDB engineers would do it. This not only makes it more fool-proof, but it also helps you… Cut Management Overhead. No custom scripting or special setup needed. You can spend less time running and managing manual tasks because MMS takes care of a lot of the work for you, letting you focus on other tasks. Meet SLAs. Automating critical management tasks makes it easier to meet uptime SLAs. This includes managing failover as well as doing rolling upgrades with no downtime. Scale Easily. Provision new nodes and systems with a single click.
- #46 It is, of course, possible to do these things without MMS. But it takes work. Typically manual work, or custom scripting. In either case, these things take time, require you to check for mistakes and are more prone to having things go wrong.
- #49 More info: http://www.mongodb.com/mongodb-scale