This will cover what to consider for high write throughput performance from hardware configuration through to the use of replica sets, multi-data centre deployments, monitoring and sharding to ensure your database is fast and stays online.

1. Webinar: Ensuring High Availability
for Real-time Analytics

2. ~3333 write ops/s 0.07 - 0.05 ms response

3. David Mytton
Woop Japan!

5. MongoDB at Server Density

6. MongoDB at Server Density
•27 nodes

7. MongoDB at Server Density
•27 nodes
• June 2009 - +3yrs

8. MongoDB at Server Density
•27 nodes
• June 2009 - +3yrs
•MySQL -> MongoDB

9. MongoDB at Server Density
•27 nodes
• June 2009 - +3yrs
•MySQL -> MongoDB
•15TB data per month

10. Why?

11. Why?
• Replication

12. Why?
• Replication
• Official drivers

13. Why?
• Replication
• Official drivers
• Easy deployment

14. Why?
• Replication
• Official drivers
• Easy deployment
• Fast out of the box

15. Fast out of the box?
Photo: dannychoo.com

16. ~3333 write ops/s 0.07 - 0.05 ms response

17. Fast out of the box?
• Softlayer cloud (1 core, 8GB)
Photo: dannychoo.com

18. Fast out of the box?
• Softlayer cloud (1 core, 8GB)
• Local instance storage
Photo: dannychoo.com

19. Fast out of the box?
• Softlayer cloud (1 core, 8GB)
• Local instance storage
• Ubuntu 10.04 LTS
Photo: dannychoo.com

21. Fast out of the box?
• Journaling
Photo: dannychoo.com

22. Fast out of the box?
• Journaling
• Replication
Photo: dannychoo.com

23. Fast out of the box?
• Journaling
• Replication
• Unsafe writes
Photo: dannychoo.com

24. Fast out of the box?
Picture is unrelated! Mmm, ice cream.

25. Fast out of the box?
• Fast network
Picture is unrelated! Mmm, ice cream.

26. Fast out of the box?
• Fast network
• Working set in RAM
Picture is unrelated! Mmm, ice cream.

27. mongos> db.metrics_20120508_15_1m.stats()
{
"sharded" : true,
"flags" : 1,
"ns" : "metrics.metrics_20120508_15_1m",
"count" : 2752934,
"numExtents" : 46,
"size" : 746837640,
"storageSize" : 823717888,
"totalIndexSize" : 517581680,
"indexSizes" : {
"_id_" : 130358144,
"a_1_i_1" : 155711920,
"a_1_i_1_m_1_t_1" : 231511616
},
"avgObjSize" : 271.2878841265355,
"nindexes" : 3,
"nchunks" : 61,

28. "size" : 746837640,
"totalIndexSize" : 517581680
Indexes = 493MB Data = 712MB

29. "size" : 746837640,
"totalIndexSize" : 517581680
Indexes = 493MB Data = 712MB

30. "size" : 746837640,
"totalIndexSize" : 517581680
Indexes = 493MB Data = 712MB
Total = 1205MB

31. Where should it go?
Should it be in
What?
memory?
Indexes Always
Data If you can
http://www.flickr.com/photos/comedynose/4388430444/

32. How you’ll know
1) Slow queries
Thu Oct 14 17:01:11 [conn7410] update sd.apiLog
query: { c: "android/setDeviceToken", a: 1466, u:
"blah", ua: "Server Density Android" } 51926ms
www.flickr.com/photos/tonivc/2283676770/

33. How you’ll know
2) Timeouts
cursor timed out (20000 ms)

34. How you’ll know
3) Disk i/o spikes
www.flickr.com/photos/daddo83/3406962115/

35. Fast out of the box?
• Fast network
• Working set in RAM
Picture is unrelated! Mmm, ice cream.

36. Fast out of the box?
• Fast network
• Working set in RAM
• Fast disks (optional)
Picture is unrelated! Mmm, ice cream.

39. Fast out of the box?
• Fast network
• Working set in RAM
• Fast disks (optional)
Picture is unrelated! Mmm, ice cream.

40. Fast out of the box?
• Fast network
• Working set in RAM
• Fast disks (optional)
• Sharding (optional)
Picture is unrelated! Mmm, ice cream.

41. Scaling writes

42. Scaling writes
• Global DB lock

43. Scaling writes
• Global DB lock
• Concurrency

44. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html

45. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html

46. http://bit.ly/mongolock

47. Scaling writes
• Global DB lock
• Concurrency
• Sharding

48. Scaling writes

49. Failover
•Replica sets

50. Failover
•Replica sets
•Master/slave

51. Failover
•Replica sets
•Master/slave
•Min 3 nodes

52. Failover
•Replica sets
•Master/slave
•Min 3 nodes
•Automatic failover

53. rs.status()
{
! "_id" : 1,
! "name" : "rs3b:27018",
! "health" : 1,
! "state" : 2,
! "stateStr" : "SECONDARY",
! "uptime" : 1886098,
! "optime" : {
! ! "t" : 1291252178000,
! ! "i" : 13
! },
! "optimeDate" : ISODate("2010-12-02T01:09:38Z"),
"lastHeartbeat" : ISODate("2010-12-02T01:09:38Z")
},
www.ex-astris-scientia.org/inconsistencies/ent_vs_tng.htm (yes it’s a replicator from Star Trek)

54. rs.status()
1) myState
Value Meaning
0 Starting up (phase 1)
1 Primary
2 Secondary
3 Recovering
4 Fatal error
5 Starting up (phase 2)
6 Unknown state
7 Arbiter
8 Down

55. rs.status()
2) Optime
"optimeDate" : ISODate("2010-12-02T01:09:38Z")
www.flickr.com/photos/robbie73/4244846566/

56. rs.status()
3) Heartbeat
"lastHeartbeat" : ISODate("2010-12-02T01:09:38Z")
www.flickr.com/photos/drawblindfaith/3400981091/

57. Scaling reads

58. Scaling reads
•Replica slaves

59. Scaling reads
•Replica slaves
•Consistency

60. Scaling reads
•Replica slaves
•Consistency
•w flag / tags

61. Unsafe writes

62. Unsafe writes
• Fast by default

63. Unsafe writes
• Fast by default
•J

64. Unsafe writes
• Fast by default
•J
• fsync

65. Unsafe writes
• Fast by default
•J
• fsync
•w

66. Tags
{
_id : "someSet",
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_id : 2, host : "C", tags : {"dc": "sf"}},
{_id : 3, host : "D", tags : {"dc": "sf"}},
{_id : 4, host : "E", tags : {"dc": "cloud"}}
]
settings : {
getLastErrorModes : {
veryImportant : {"dc" : 3},
sortOfImportant : {"dc" : 2}
}
}
}
> db.foo.insert({x:1})
> db.runCommand({getLastError : 1, w : "veryImportant"})

67. Tags
{
_id : "someSet",
(A or B) + (C or D) + E
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_id : 2, host : "C", tags : {"dc": "sf"}},
{_id : 3, host : "D", tags : {"dc": "sf"}},
{_id : 4, host : "E", tags : {"dc": "cloud"}}
]
settings : {
getLastErrorModes : {
veryImportant : {"dc" : 3},
sortOfImportant : {"dc" : 2}
}
}
}
> db.foo.insert({x:1})
> db.runCommand({getLastError : 1, w : "veryImportant"})

68. Tags
{
_id : "someSet",
(A + C) or (D + E) ...
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_id : 2, host : "C", tags : {"dc": "sf"}},
{_id : 3, host : "D", tags : {"dc": "sf"}},
{_id : 4, host : "E", tags : {"dc": "cloud"}}
]
settings : {
getLastErrorModes : {
veryImportant : {"dc" : 3},
sortOfImportant : {"dc" : 2}
}
}
}
> db.foo.insert({x:1})
> db.runCommand({getLastError : 1, w : "sortOfImportant"})

69. Bottlenecks
•EC2

70. Bottlenecks
•EC2
•Local storage

71. Bottlenecks
•EC2
•Local storage
•EBS: RAID10 4-8 volumes

72. Bottlenecks
•EC2
•Local storage
•EBS: RAID10 4-8 volumes
•i/o: rand but not sequential

73. http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs

74. http://bit.ly/
ec2mongodb

75. Bottlenecks
•CPU
•Index building

76. mongostat

77. mongostat
Locks/Queues

78. mongostat
Diagnostics

79. Current operations
db.currentOp();
{
! ! ! "opid" : "shard1:299939199",
! ! ! "active" : true,
! ! ! "lockType" : "write",
! ! ! "waitingForLock" : false,
! ! ! "secs_running" : 15419,
! ! ! "op" : "remove",
! ! ! "ns" : "sd.metrics",
! ! ! "query" : {
! ! ! ! "accId" : 1391,
! ! ! ! "tA" : {
! ! ! ! ! "$lte" : ISODate("2010-11-24T19:53:00Z")
! ! ! ! }
! ! ! },
! ! ! "client" : "10.121.12.228:44426",
! ! ! "desc" : "conn"
! ! },
www.flickr.com/photos/jeffhester/2784666811/

80. Monitoring tools
Run yourself
Ganglia

81. Monitoring tools
Server Density

82. Fast out of the box?
• Fast network
• Working set in RAM
Picture is unrelated! Mmm, ice cream.

83. Fast out of the box?
• Fast network
• Working set in RAM
• bit.ly/benchrun
Picture is unrelated! Mmm, ice cream.

84. David Mytton
@davidmytton
david@serverdensity.com
www.serverdensity.com
Woop Japan!