1. MongoDB
Debugging Performance Problems
@ConradIrwin

3. 20M crashes/day
2TB data / 50GB index
11 nodes / 3* availability zones

4. I don't care if MongoDB is slow
I care if my app is slow

5. How to make
app fast agai
it's slow bec
of the way I'm

6. What does slow mean?

8. Performance over time

9. Performance over time

11. It's slow :(

14. Solution 1
Denormalize

16. It's fast :)

17. It's slow :(

18. db.errors.
find({project_id: x}).
sort({ _id: -1}).
limit(30)

19. db.errors.
find({project_id: x}).
sort({ _id: -1}).
limit(30).
explain()

20. {
"cursor":"BtreeCursor_id_reverse",
"isMultiKey":false,
"n":0,
"nscannedObjects":227756,
"nscanned":227756,
"nscannedObjectsAllPlans":227756,
"nscannedAllPlans":227756,
"scanAndOrder":false,
"indexOnly":false,
"nYields":1779,
"nChunkSkips":0,
"millis":461,
"indexBounds":{
"_id":[
[
{
"$maxElement":1
},
{
"$minElement":1
}
]
]
},
"server":"Jaroussky.local:27017",
"filterSet":false
}

21. {
"cursor" :
"BtreeCursor _id_ reverse",
"nscanned" : 227756,
"indexBounds" : {"_id" : [
[{"$maxElement" : 1},
{"$minElement" : 1}]
]
}
}

22. {
"cursor" :
"BtreeCursor _id_ reverse",
"nscanned" : 227756,
"indexBounds" : {"_id" : [
[{"$maxElement" : 1},
{"$minElement" : 1}]
]
}
}

23. Solution 2
Index

24. db.errors.ensureIndex(
{project_id: 1, id: -1}
)

25. {"cursor" : "BtreeCursor
project_id_1__id_1",
"nscanned" : 6,
"indexBounds" : {
"project_id" : [[
ObjectId(x),ObjectId(x)
]],
"_id" : [[
{"$minElement" : 1},
{"$maxElement" : 1}
]]
}}

26. It's fast :)

27. It's slow :(

28. Only 1 query...
Indexed properly...

29. mongostat

30. insert queryupdatedeletegetmorecommandflushesmapped
5 97 57 *0 95 127|0 0 320g
5 98 61 *0 113 146|0 0 320g
8 94 61 *0 95 137|0 0 320g
vsize resfaults lockeddbidxmiss% qr|qw ar|aw
641g 8.65g 7bugsnag:47.7% 0 0|0 0|0
641g 8.66g 2bugsnag:21.0% 0 0|0 0|0
641g 8.64g 3bugsnag:23.4% 0 0|0 0|0
netInnetOut conn setrepl time
47k 73k 145bugsnag1 PRI 06:34:05
63k 99k 146bugsnag1 PRI 06:34:06
98k 124k 146bugsnag1 PRI 06:34:07

31. command locked db time
127|0 bugsnag:47.7% 06:34:05
146|0 bugsnag:21.0% 06:34:06
137|0 bugsnag:23.4% 06:34:07

33. Solution 3
Shard

35. sh.shardCollection("errors"
{
project_id: 1,
_id: -1
})

36. command locked db time
83|0 bugsnag:3.1% 06:46:12
69|0 bugsnag:4.1% 06:46:13
73|0 bugsnag:2.4% 06:46:14

37. It's fast :)

38. It's slow :(

39. Only 1 query...
Indexed properly...
Lock % ok...

40. iostat

41. Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/savgrq-sz
xvdi 0.00 2.00 15.00 743.00 436.00 7905.50 22.01
xvdi 0.00 2.00 0.00 663.00 0.00 6706.00 20.23
xvdi 0.00 1.00 0.00 726.00 0.00 5593.50 15.41
avgqu-sz awaitr_awaitw_await svctm %util
11.02 14.54 5.33 14.73 0.60 45.60
8.04 12.12 0.00 12.12 0.51 33.60
13.01 17.92 0.00 17.92 0.50 36.40

42. rkB/s wkB/s await %util
7660 138 14.73 83.60
2248 2043 12.12 65.20
7905 436 17.92 45.60

43. db.stats()['indexSize']
= 3852071824
= 3932614656
free -b(m3.medium)

44. Solution 4
Scale

45. db.stats()['indexSize']
= 3852071824
= 15775363072
free -b(r3.large)

46. It's fast :)

47. It's slow :(

48. Main query seems fine...
"Quick" queries sometimes slow...

49. mongotop

50. ns total read write 2
014-06-22T19:11:35
bugsnag.events 35ms 0ms 35ms
bugsnag.errors 30ms 6ms 24ms
bugsnag.system.namespaces 26ms 26ms 0ms
bugsnag.projects 12ms 5ms 7ms
bugsnag.users 15ms 2ms 13ms
bugsnag.error_aggregates 4ms 0ms 4ms
bugsnag.deploys 4ms 3ms 1ms
bugsnag.event_tallies 3ms 0ms 3ms

51. ns tot r w
bugsnag.events 35 0 35
bugsnag.errors 30 6 24
bugsnag.projects 12 5 7
bugsnag.users 15 7 8

52. Solution 5
Tag shards

53. It's fast :)

54. Solution 1
Denormalize

55. Solution 2
Index

56. Solution 3
Shard

57. Solution 4
Scale

58. Solution 5
Tag shards

59. Solution 6
...

60. NewRelic / Skylight
explain()
mongostat
iostat
...

61. It will
be slow

62. You can
speed it up

63. Thanks!@ConradIrwin