Sharding MongoDB with MongoDB
•
2 gostaram•1,313 visualizações
Sharding with MongoDB allows scaling a database horizontally across multiple servers. It involves splitting data into chunks and distributing those chunks across shards. The mongos router directs read and write operations to the appropriate shards. Documents are sharded based on a shard key to ensure related data resides on the same shard. Queries are routed efficiently based on the shard key. Splits and migrations balance data as needs change over time.
Recomendados
Mais conteúdo relacionado
Destaque
Destaque (20)
Semelhante a Sharding MongoDB with MongoDB
Semelhante a Sharding MongoDB with MongoDB (20)
Último
Último (20)
Sharding MongoDB with MongoDB
- 1. Sharding with MongoDB Tyler Brock tyler@10gen.com @TylerBrock
- 3. Philosophy
- 4. Philosophy MongoDB is a database for developers.
- 5. Philosophy Build
- 6. Philosophy Build Scale
- 7. Philosophy How to Draw an Owl
- 8. Philosophy How to Draw an Owl
- 9. Philosophy Draw Two Circles > db.runCommand({enablesharding: "<dbname>" }) > db.runCommand({ shardcollection: "<namespace>", key: <shardkeypatternobject> })
- 10. Concepts
- 11. Simple Web Application app Read/Write datastore
- 12. What happens when your working set exceeds memory?
- 13. What happens if your write load is enormous?
- 14. Vertical Scaling app datastore
- 15. Vertical Scaling app datastore
- 16. Vertical Scaling app app app datastore 68 GB Ram Raid10 EBS
- 17. Vertical Scaling app app app datastore 128 GB Ram Raid10 SSD
- 19. Horizontal Scaling app datastore 60gb
- 20. Horizontal Scaling app datastore datastore datastore 20gb 20gb 20gb
- 21. Horizontal Scaling app Routing Logic metadata datastore datastore datastore 20gb 20gb 20gb
- 22. Horizontal Scaling app Routing Logic metadata datastore datastore datastore 60gb 20gb 20gb
- 23. Horizontal Scaling app Routing Logic Balancer metadata datastore datastore datastore 60gb 20gb 20gb
- 24. Horizontal Scaling app Routing Logic Balancer metadata datastore datastore datastore 30gb 30gb 30gb
- 25. Architecture
- 26. Shard Really is just a mongod (or replica set) mongod Where your data lives
- 27. Config Server Mongod started with --configsvr option config Must have 3 (or 1 in development) Data is commited using 2 phase commit
- 28. mongos Acts just like shard router / proxy One or as many as you want mongos Light weight -- can run on App servers Caches meta-data from config servers
- 29. Routing Logic Balancing metadata datastore datastore datastore
- 30. mongos metadata datastore datastore datastore
- 31. app mongos metadata datastore datastore datastore
- 32. app mongos config datastore datastore datastore
- 33. app mongos config config config datastore datastore datastore
- 34. app mongos config config config mongod mongod mongod
- 35. app mongos config config config RS RS RS mongod mongod mongod mongod mongod mongod mongod mongod mongod
- 36. app mongos config config config RS RS RS mongod mongod mongod mongod mongod mongod mongod mongod mongod
- 37. Mechanics
- 38. How does MongoDB balance my data?
- 39. Keys test.users { name: “Joe”, email: “Joe@fake.com”, }, { name: “Bob”, email: “bob@fake.com”, }, { name: “Tyler”, email: “tyler@fake.com”, }
- 40. Keys test.users { name: “Joe”, email: “Joe@fake.com”, > db.runCommand({ }, shardcollection: “test.users”, { key: { email: 1 } name: “Bob”, }) email: “bob@fake.com”, }, { name: “Tyler”, email: “tyler@fake.com”, }
- 41. Keys test.users { name: “Joe”, email: “Joe@fake.com”, }, shardcollection: “test.users”, { key: { email: 1 } name: “Bob”, email: “bob@fake.com”, }, { name: “Tyler”, email: “tyler@fake.com”, }
- 42. Keys test.users { name: “Joe”, email: “Joe@fake.com”, }, { key: { email: 1 } name: “Bob”, email: “bob@fake.com”, }, { name: “Tyler”, email: “tyler@fake.com”, }
- 43. Chunks -∞ +∞
- 44. Chunks -∞ +∞ joe@fake.com tyler@fake.com moe@fake.com
- 45. Chunks Split! -∞ +∞ joe@fake.com tyler@fake.com moe@fake.com
- 46. Chunks This is a Split! This is a chunk chunk -∞ +∞ joe@fake.com tyler@fake.com moe@fake.com
- 47. Chunks -∞ +∞ joe@fake.com tyler@fake.com moe@fake.com
- 48. Chunks -∞ +∞ joe@fake.com tyler@fake.com moe@fake.com
- 49. Chunks Split! -∞ +∞ joe@fake.com tyler@fake.com moe@fake.com
- 50. Splitting config mongos config config Shard 1 Shard 2 Shard 3 Shard 4
- 51. Splitting config mongos config config Shard 1 Shard 2 Shard 3 Shard 4
- 52. Splitting config mongos config config Shard 1 Shard 2 Shard 3 Shard 4
- 53. Splitting config mongos config config Shard 1 Shard 2 Shard 3 Shard 4
- 54. Splitting Split this big chunk into 2 config chunks mongos config config Shard 1 Shard 2 Shard 3 Shard 4
- 55. Splitting config mongos config config Shard 1 Shard 2 Shard 3 Shard 4
- 56. Splitting config mongos config These config chunks have split Shard 1 Shard 2 Shard 3 Shard 4
- 57. Balancing config mongos config config Shard 1 Shard 2 Shard 3 Shard 4
- 58. Balancing Shard1, move a chunk to config Shard2 mongos config config Shard 1 Shard 2 Shard 3 Shard 4
- 59. Balancing config mongos config config Shard 1 Shard 2 Shard 3 Shard 4
- 60. Balancing Shard1, move another chunk config to Shard3 mongos config config Shard 1 Shard 2 Shard 3 Shard 4
- 61. Balancing config mongos config config Shard 1 Shard 2 Shard 3 Shard 4
- 62. Balancing Shard1, move another chunk config to Shard4 mongos config config Shard 1 Shard 2 Shard 3 Shard 4
- 63. Balancing config mongos config config Shard 1 Shard 2 Shard 3 Shard 4
- 64. Balancing config mongos config config Shard 1 Shard 2 Shard 3 Shard 4
- 65. How does MongoDB route my queries?
- 66. Routed Request mongos shard shard shard
- 67. Routed Request 1 1. Query arrives at Mongos mongos shard shard shard
- 68. Routed Request 1 1. Query arrives at Mongos 2. Mongos routes query to a mongos single shard 2 shard shard shard
- 69. Routed Request 1 1. Query arrives at Mongos 2. Mongos routes query to a mongos single shard 3. Shard returns results of 2 query 3 shard shard shard
- 70. Routed Request 1 1. Query arrives at Mongos 4 2. Mongos routes query to a mongos single shard 3. Shard returns results of 2 query 4. Results returned to client 3 shard shard shard
- 71. Scatter Gather Request mongos shard shard shard
- 72. Scatter Gather Request 1 1. Query arrives at Mongos mongos shard shard shard
- 73. Scatter Gather Request 1 1. Query arrives at Mongos 2. Mongos broadcasts query mongos to all shards 2 2 2 shard shard shard
- 74. Scatter Gather Request 1 1. Query arrives at Mongos 2. Mongos broadcasts query mongos to all shards 3. Each shard returns results for query 2 2 2 3 3 3 shard shard shard
- 75. Scatter Gather Request 1 1. Query arrives at Mongos 4 2. Mongos broadcasts query mongos to all shards 3. Each shard returns results for query 2 2 2 4. Results combined and 3 3 returned to client 3 shard shard shard
- 76. Distributed Merge Sort Req. mongos shard shard shard
- 77. Distributed Merge Sort Req. 1 1. Query arrives at Mongos mongos shard shard shard
- 78. Distributed Merge Sort Req. 1 1. Query arrives at Mongos 2. Mongos broadcasts query mongos to all shards 2 2 2 shard shard shard
- 79. Distributed Merge Sort Req. 1 1. Query arrives at Mongos 2. Mongos broadcasts query mongos to all shards 3. Each shard locally sorts results 2 2 2 shard shard shard 3 3 3
- 80. Distributed Merge Sort Req. 1 1. Query arrives at Mongos 2. Mongos broadcasts query mongos to all shards 3. Each shard locally sorts results 2 2 2 4. Results returned to mongos 4 4 4 shard shard shard 3 3 3
- 81. Distributed Merge Sort Req. 1 1. Query arrives at Mongos 2. Mongos broadcasts query mongos 5 to all shards 3. Each shard locally sorts results 2 2 2 4. Results returned to mongos 4 4 4 5. Mongos merges sorted results shard shard shard 3 3 3
- 82. Distributed Merge Sort Req. 1 1. Query arrives at Mongos 6 2. Mongos broadcasts query mongos 5 to all shards 3. Each shard locally sorts results 2 2 2 4. Results returned to mongos 4 4 4 5. Mongos merges sorted results shard shard shard 3 3 3 6. Combined results returned to client
- 83. Queries By Shard Key Routed db.users.find({email: “bob@10gen.com”}) Sorted by Routed in order db.users.find().sort({email:-1}) shard key Find by non Scatter Gather db.users.find({state:”NY”}) shard key Sorted by Distributed merge db.users.find().sort({state:1}) sort non shard key
- 84. Writes Inserts Requires shard key db.users.insert({ name: “Bob”, email: “Bob@fake.com”}) Removes Routed db.users.delete({ email: “bob@fake.com”}) Scattered db.users.delete({name: “Bob”}) Updates Routed db.users.update( {email: “bob@10gen.com”}, {$set: { state: “NY”}}) Scattered db.users.update( {state: “CA”}, {$set:{ state: “NY”}} )
- 85. How do I choose my shard key?
- 86. Rule of Thumb Choose a field that is common to your queries.
- 87. Write Scaling Writes should be distributed.
- 88. Writes should be distributed { node: "ny153.example.com", application: "apache", time: "2011-01-02T21:21:56Z", level: "ERROR", msg: "something is broken" } Bad { time : 1 }
- 89. Writes should be distributed { node: "ny153.example.com", application: "apache", time: "2011-01-02T21:21:56Z", level: "ERROR", msg: "something is broken" } Bad { time : 1 } Better {node:1, application:1, time:1}
- 90. Query Isolation & Data Locality Queries should be routed to one shard.
- 91. Queries should be routed to one shard { node: "ny153.example.com", application: "apache", time: "2011-01-02T21:21:56Z", level: "ERROR", msg: "something is broken” } Bad {msg: 1, node: 1}
- 92. Queries should be routed to one shard { node: "ny153.example.com", application: "apache", time: "2011-01-02T21:21:56Z", level: "ERROR", msg: "something is broken” } Bad {msg: 1, node: 1} Better {node: 1, time: 1}
- 93. Cardinality Chunks should be able to split.
- 94. Chunks should be able to split { node: "ny153.example.com", application: "apache", time: "2011-01-02T21:21:56Z", level: "ERROR", msg: "something is broken" } Bad {node: 1}
- 95. Chunks should be able to split { node: "ny153.example.com", application: "apache", time: "2011-01-02T21:21:56Z", level: "ERROR", msg: "something is broken" } Bad {node: 1} Better {node:1, time:1}
- 96. Configuration
- 97. Bring up mongods or Replica Sets mongod --shardsvr mongod --replSet --shardsvr RS RS RS mongod mongod mongod mongod mongod mongod mongod mongod mongod
- 98. Bring up Config Servers mongod --configsvr config config config RS RS RS mongod mongod mongod mongod mongod mongod mongod mongod mongod
- 99. Bring up Mongos mongos --configdb <list of configdb uris> mongos config config config RS RS RS mongod mongod mongod mongod mongod mongod mongod mongod mongod
- 100. Connect to Mongos + Add Shards > use admin > db.runCommand({"addShard": <shard uri>}) Enable Sharding > db.runCommand( { enablesharding : "<dbname>" } ); Shard a Collection > db.runCommand( { shardcollection : "<namespace>", key : <key> });