In version 2.4, MongoDB introduces hash-based sharding, allowing the user to shard based on a randomized shard key to spread documents evenly across a cluster. Hash-based sharding is an alternative to range-based sharding, making it easier to manage your growing cluster. In this talk, we'll discuss provide an overview of this new feature and discuss the pros and cons of using a hash-based sharding vs. range-based approach.

1. Senior Solutions Architect, 10gen
James Kerr
2.4 Sharding Features

2. Agenda
• Mechanics of sharding
– Key space
– Chunks
– Balancing
• Types of requests
• Hashed shard keys
– Why use hashed shard keys
– How to enable hashed shard keys
– Limitations

3. Sharded Cluster

4. Sharding your data

5. What is a Shard Key
• Shard key is used to partition your collection
• Shard key must exist in every document
• Shard key is immutable
• Shard key values are immutable
• Shard key must be indexed
• Shard key is used to route requests to shards

6. The key space
{x: 10} {x: -5} {x: -9} {x: 7} {x: 6} {x: 0}

7. Inserting data
{x: 0}{x: 6}{x: 7}{x: -5}{x: 10} {x: -9}

8. Inserting data
{x: 0} {x: 6} {x: 7}{x: -5} {x: 10}{x: -9}

9. Chunk range and size
{x: 0} {x: 6} {x: 7}{x: -5} {x: 10}{x: -9}

10. Inserting further data
{x: 0} {x: 6} {x: 7}{x: -5} {x: 10}{x: -9}
{x: 9}{x: -7} {x: 3}

11. Chunk splitting
{x: 0} {x: 6} {x: 7}{x: -5} {x: 10}{x: -9}
0 0
• Achunk is split once it exceeds the maximum size
• There is no split point if all documents have the same shard key
• Chunk split is a logical operation (no data is moved)
• If split creates too large of a discrepancy of chunk count across cluster
a balancing round starts

12. Data distribution
• MinKey to 0 lives on Shard1
• 0 to MaxKey lives on Shard2
• Mongos routes queries appropriately

13. Mongos routes data
minKey  0 0  maxKey
db.test.insert({ x: -1000 })

14. Mongos routes data
minKey  0 0  maxKey
db.test.insert({ x: -1000 })

15. Unbalanced shards
minKey  0 0  maxKey

16. Balancing
• Migration threshold
• Number of chunks less than 20, migration threshold of 2
• 21-80, migration threshold 4
• >80, migration threshold 8

17. Moving the chunk
• One chunk of data is copied from Shard 1 to Shard 2

18. Committing Migration
• Once everyone agrees the data has moved, that chunk gets
deleted from Shard 1.

19. Cleanup
• Other mongos' have to find out about new configuration

20. Migrations' effect
• Expensive
• Can take a long time
• Competes for limited resources

21. Picking a shard key
• Cardinality
• Optimize routing
• Minimize (unnecessary) traffic
• Allow best scaling

22. What about Object Id?
ObjectId("51597ca8e28587b86528edfd”)
• Used for _id
• 12 byte value
• Generated by the driver if not specified
• Theoretically globally unique

23. What about Object Id?
ObjectId("51597ca8e28587b86528edfd”)
12 Bytes
Timestamp
MAC
PID
Counter

24. // enabling sharding on test database
mongos> sh.enableSharding("test")
{ "ok" : 1 }
// sharding the test collection
mongos> sh.shardCollection("test.test",{_id:1})
{ "collectionsharded" : "test.test", "ok" : 1 }
// create a loop inserting data
mongos> for (x=0; x<10000; x++) {
... db.test.insert({value:x})
... }
Sharding on ObjectId

25. shards:
{ "_id" : "shard0000", "host" : "localhost:30000" }
{ "_id" : "shard0001", "host" : "localhost:30001" }
databases:
{ "_id" : "test", "partitioned" : true, "primary" : "shard0001" }
test.test
shard key: { "_id" : 1 }
chunks:
shard0001 3
{ "_id" : { "$minKey" : 1 } } -->> { "_id" : ObjectId(”...") }
on : shard0001 { "t" : 1000, "i" : 1 }
{ "_id" : ObjectId(”...”) } -->> { "_id" : { "$maxKey" : 1 } }
on : shard0001 { "t" : 1000, "i" : 2 }
ObjectId chunk distribution

26. ObjectId gives a hot shard
minKey  0 0  maxKey

27. Sharding on incremental
values like timestamp is
not optimum for even
distribution

28. Hashed Shard Keys

29. Hashed Shard Keys
{x:2} md5 c81e728d9d4c2f636f067f89cc14862c
{x:3} md5 eccbc87e4b5ce2fe28308fd9f2a7baf3
{x:1} md5 c4ca4238a0b923820dcc509a6f75849b

30. Hashed shard key eliminates hot
shards
minKey  0 0  maxKey

31. Under the hood
• Create a hashed index used for sharding
• Uses the first 64-bits of md5 hash of field
• Uses existing hash index, or creates a new one
on a collection
• Hash both data and BSON type
• Represented as a NumberLong in the JS shell

32. // hash on 1 as an integer
> db.runCommand({_hashBSONElement:1})
{
"key" : 1,
"seed" : 0,
"out" : NumberLong("5902408780260971510"),
"ok" : 1
}
// hash on “1” as a string
> db.runCommand({_hashBSONElement:"1"})
{
"key" : "1",
"seed" : 0,
"out" : NumberLong("-2448670538483119681"),
"ok" : 1
}
Hash on simple or embedded BSON
values

33. Enabling hashed indexes
• Create index
– db.collection.ensureIndex( {field : ”hashed”} )
• Options
– Seed, specify a different seed to use
– hashVersion, at the moment only version 0 (md5).

34. Using hash shard keys
• Enable sharding on collection
– sh.shardCollection(“test.collection”, {field: “hashed”})
• Options
– numInitialChunks, specifies the number of initial chunks
per shard. Default is two chunks per shard (use
“sh._adminCommand” to specify options)

35. // enabling sharding on test database
mongos> sh.enableSharding("test")
{ "ok" : 1 }
// shard by hashed _id field
mongos> sh.shardCollection("test.hash",{_id:"hashed"})
{ "collectionsharded" : "test.hash", "ok" : 1 }
Sharding on hashed ObjectId

36. databases:
{ "_id" : "test", "partitioned" : true, "primary" : "shard0001" }
test.hash
shard key: { "_id" : "hashed" }
chunks:
shard0000 2
shard0001 2
{ "_id" : { "$minKey" : 1 } } -->> { "_id" : NumberLong("-4611686018427387902") } on :
shard0000 { "t" : 2000, "i" : 2 }
{ "_id" : NumberLong("-4611686018427387902") } -->> { "_id" : NumberLong(0) } on : shard0000
{ "t" : 2000, "i" : 3 }
{ "_id" : NumberLong(0) } -->> { "_id" : NumberLong("4611686018427387902") } on : shard0001 {
"t" : 2000, "i" : 4 }
{ "_id" : NumberLong("4611686018427387902") } -->> { "_id" : { "$maxKey" : 1 } } on : shard0001
{ "t" : 2000, "i" : 5 }
Pre-splitting the data

37. // create a loop inserting data
mongos> for (x=0; x<10000; x++) {
... db.hash.insert({value:x})
... }
Inserting into hashed shard key
collection

38. test.hash
shard key: { "_id" : "hashed" }
chunks:
shard0000 4
shard0001 4
{ "_id" : { "$minKey" : 1 } } -->> { "_id" : NumberLong("-
7374407069602479355") } on : shard0000 { "t" : 2000, "i" : 8 }
{ "_id" : NumberLong("-7374407069602479355") } -->> { "_id" : NumberLong("-
4611686018427387902") } on : shard0000 { "t" : 2000, "i" : 9 }
{ "_id" : NumberLong("-4611686018427387902") } -->> { "_id" : NumberLong("-
2456929743513174890") } on : shard0000 { "t" : 2000, "i" : 6 }
{ "_id" : NumberLong("-2456929743513174890") } -->> { "_id" : NumberLong(0)
} on : shard0000 { "t" : 2000, "i" : 7 }
{ "_id" : NumberLong(0) } -->> { "_id" : NumberLong("1483539935376971743")
} on : shard0001 { "t" : 2000, "i" : 12 }
Even distribution of chunks

39. Routing Requests

40. Cluster Request Routing
• Targeted Queries
• Scatter Gather Queries
• Scatter Gather Queries with Sort

41. Cluster Request Routing: Targeted
Query

42. Routable request received

43. Request routed to appropriate shard

44. Shard returns results

45. Mongos returns results to client

46. Cluster Request Routing: Non-Targeted
Query

47. Non-Targeted Request Received

48. Request sent to all shards

49. Shards return results to mongos

50. Mongos returns results to client

51. Cluster Request Routing: Non-Targeted
Query with Sort

52. Non-Targeted request with sort
received

53. Request sent to all shards

54. Query and sort performed locally

55. Shards return results to mongos

56. Mongos merges sorted results

57. Mongos returns results to client

58. Hash keys are great for equality
queries
• Equality queries directed to a specific shard
• Will use the index
• Most efficient query possible

59. mongos> db.hash.find({x:1}).explain()
{
"cursor" : "BtreeCursor x_hashed",
"n" : 1,
"nscanned" : 1,
"nscannedObjects" : 1,
"millisShardTotal" : 0,
"numQueries" : 1,
"numShards" : 1,
"indexBounds" : {
"x" : [
[
NumberLong("5902408780260971510"),
NumberLong("5902408780260971510")
]
]
},
"millis" : 0
}
Explain plan of an equality query

60. But not so good for a range query
• Range queries scatter gather
• Won’t use index

61. mongos> db.hash.find({x:{$gt:1, $lt:99}}).explain()
{
"cursor" : "BasicCursor",
"n" : 97,
"nChunkSkips" : 0,
"nYields" : 0,
"nscanned" : 1000,
"nscannedAllPlans" : 1000,
"nscannedObjects" : 1000,
"nscannedObjectsAllPlans" : 1000,
"millisShardTotal" : 0,
"millisShardAvg" : 0,
"numQueries" : 2,
"numShards" : 2,
"millis" : 3
}
Explain plan of a range query

62. Other limitations
• Cannot use a compound key
• Key cannot have an array value
• Tag-aware sharding
– Only makes sense to assign the full hashed shard key
collection to particular shards
– By design, there’s no real way to know or control what
data is in what range
• Key with poor cardinality is going to give a hash
with poor cardinality
– Floating point numbers are squashed. E.g. 100.4 will be
hashed as 100

63. Summary
• Range-based Sharding
– Most efficient for applications that operate on ranges
– Requires careful shard key selection
• Hash-based Sharding
– Uniform writes,
– No routed range queries
• TagAware Sharding
– That’s another talk!

64. Resources
• Tutorial:Sharding (laptop friendly)
• Tutorial:Converting a Replica Set to a Replicated Sharded Cluster(laptop
friendly)
• Manual:Select a Shard Key
• Manual:Hash-based Sharding
• Manual:TagAware Sharding
• Manual:Strategiesfor Bulk Inserts in ShardedClusters
• Manual:Manage Sharded Cluster Balancer

65. Questions?

66. James Kerr
Thank You
Senior Solutions Architect, 10gen