This document discusses MongoDB infrastructure at Server Density. It notes that Server Density uses 27 MongoDB nodes to store 20TB of data per month from their MySQL database. Some key reasons for choosing MongoDB include replication, official drivers, easy deployment, and fast performance out of the box. The document then discusses various MongoDB performance and infrastructure considerations like network throughput, replication lag, failover processes, disk types, backups, and monitoring.

1. NoSQL Infrastructure

2. ~3333 write ops/s 0.07 - 0.05 ms response

3. David Mytton
Woop Japan!

5. MongoDB at Server Density

6. •27 nodes
MongoDB at Server Density

7. • June 2009 - 4yrs
•27 nodes
MongoDB at Server Density

8. •MySQL -> MongoDB
•27 nodes
MongoDB at Server Density
• June 2009 - 4yrs

9. •MySQL -> MongoDB
•20TB data per month
•27 nodes
MongoDB at Server Density
• June 2009 - 4yrs

10. Why?

11. Why?
• Replication

12. Why?
• Replication
• Official drivers

13. Why?
• Replication
• Official drivers
• Easy deployment

14. Why?
• Replication
• Official drivers
• Easy deployment
• Fast out of the box

15. It’s a little different.

16. Picture is unrelated! Mmm, ice cream.
• Fast network
Performance

17. • Fast network
Performance
EC2 10 Gigabit Ethernet
- Cluster Compute
- High Memory Cluster
- Cluster GPU
- High I/O
- High Storage

18. • Fast network
Performance
Workload: Read/Write?
Result set size
What is being stored?

19. • Fast network
Performance
Use Network Throughput
Normal 0-100Mb/s
Replication (Initial Sync) Burst +100Mb/s
Replication (Oplog) 0-100Mb/s
Backup Initial Sync + Oplog

20. • Fast network
Performance
Inter-DC LAN

21. • Fast network
Performance
Inter-DC LAN
Cross USA Washington, DC - San Jose, CA

22. • Fast network
Performance
Location Ping RTT Latency
Within USA 40-80ms
Trans-Atlantic 100ms
Trans-Pacific 150ms
Europe - Japan 300ms

23. Failover
•Replica sets

24. Failover
•Replica sets
•Master/slave

25. Failover
•Replica sets
•Min 3 nodes
•Master/slave

26. Failover
•Replica sets
•Min 3 nodes
•Master/slave
•Automatic failover

27. • Replication lag
Performance
Location Ping RTT Latency
Within USA 40-80ms
Trans-Atlantic 100ms
Trans-Pacific 150ms
Europe - Japan 300ms

28. Replication Lag
1. Reads: eventual consistency

29. Replication Lag
1. Reads: eventual consistency
2. Failover: slave behind

30. Eventual Consistency
Stale data

31. Eventual Consistency
Stale data
Inconsistent data

32. Eventual Consistency
Stale data
Inconsistent data
Changing data

33. Eventual Consistency
Use Case Needs consistency?
Graphs No
User profile Yes
Statistics Depends
Alert config Yes

34. Replication Lag
1. Reads: eventual consistency
2. Failover: slave behind

35. Slave behind
Failover: out of date master

36. MongoDB WriteConcern
Changed Nov 27 2012

37. • Safe by default
>>> from pymongo import MongoClient
>>> connection = MongoClient(w=int/str)
Value Meaning
0 Unsafe
1 Primary
2 Primary + x1 secondary
3 Primary + x2 secondaries
MongoDB WriteConcern

39. Tags
{
_id : "someSet",
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_id : 2, host : "C", tags : {"dc": "sf"}},
{_id : 3, host : "D", tags : {"dc": "sf"}},
{_id : 4, host : "E", tags : {"dc": "cloud"}}
]
settings : {
getLastErrorModes : {
veryImportant : {"dc" : 3},
sortOfImportant : {"dc" : 2}
}
}
}
> db.foo.insert({x:1})
> db.runCommand({getLastError : 1, w : "veryImportant"})

40. {
_id : "someSet",
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_id : 2, host : "C", tags : {"dc": "sf"}},
{_id : 3, host : "D", tags : {"dc": "sf"}},
{_id : 4, host : "E", tags : {"dc": "cloud"}}
]
settings : {
getLastErrorModes : {
veryImportant : {"dc" : 3},
sortOfImportant : {"dc" : 2}
}
}
}
> db.foo.insert({x:1})
> db.runCommand({getLastError : 1, w : "veryImportant"})
(A or B) + (C or D) + E
Tags

41. {
_id : "someSet",
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_id : 2, host : "C", tags : {"dc": "sf"}},
{_id : 3, host : "D", tags : {"dc": "sf"}},
{_id : 4, host : "E", tags : {"dc": "cloud"}}
]
settings : {
getLastErrorModes : {
veryImportant : {"dc" : 3},
sortOfImportant : {"dc" : 2}
}
}
}
> db.foo.insert({x:1})
> db.runCommand({getLastError : 1, w : "sortOfImportant"})
(A + C) or (D + E) ...
Tags

42. Picture is unrelated! Mmm, ice cream.
• Fast network
•More RAM
Performance

43. http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs

44. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html

45. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html

46. More RAM = expensive
Performance
x2 4GB RAM 12 month Prices

47. RAM
SSDs
Spinning disk
Cost Speed

48. Softlayer disk pricing
Performance

49. EC2 disk/RAM pricing
Performance
$2232/m
$2520/m
$43/m
$295/m

50. SSD vs Spinning
Performance

51. SSD vs Spinning
Performance

52. SSD vs Spinning
Performance

53. Tips: rand()
•Field names

54. Tips: rand()
•Field names
•Covered indexes

55. Tips: rand()
•Field names
•Covered indexes
•Collections / databases

56. Backups
What is the use case?

57. Backups
What is the use case?
Fixing user errors?
Point in time restore?
Disaster recovery?

58. Backups
•Disaster recovery
Offsite

59. Backups
•Disaster recovery
Age
Offsite

60. Backups
•Disaster recovery
Age
Offsite
Restore time

61. Backups
Frequency
Consistency
Verification

62. www.flickr.com/photos/daddo83/3406962115/
Monitoring
•System
Disk i/o
Disk use

63. www.flickr.com/photos/daddo83/3406962115/
Monitoring
Disk i/o
Disk use
•System
Swap

64. www.flickr.com/photos/daddo83/3406962115/
Monitoring
Optime
State
•Replication

65. mongostat

66. Monitoring tools
Run yourself
Ganglia

67. Monitoring tools
www.serverdensity.com/nosql

68. www.serverdensity.com/nosql

69. David Mytton
david@serverdensity.com
@davidmytton
Woop Japan!
www.serverdensity.com/nosql