MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation

1. Enteros, Inc.
MongoDB
Performance Tuning
Load Testing
NoSQL Now! 2013 Conference
San Jose
ron@enteros.com
Ron Warshawsky
August 23, 2013

2. 2
MongoDB Overview
Agenda
Overview performance
tuning
Overview load testing

3. 3
MongoDB Areas Of Performance Tuning
OS tuning
Storage tuning
Database tuning
Load Testing

4. 4
MongoDB Areas Of Performance Tuning
• OS Tuning
• Follow ulimitsper mongo manual
• http://docs.mongodb.org/manual/reference/ulimit/
• Follow production notes per mongo manual
• http://docs.mongodb.org/manual/administration/production-notes/
• No atime on fs
• No hugepages
• Use NTT to sync time between the nodes
• Try “tuned”
• Best when used in staging/load testing environment under
realisticload
• CPU, Network and Storage tuning

5. 5
MongoDB Areas Of Performance Tuning
• Storage Tuning
• RAID 10
• Ext4 or XFS
• Delaysharding with better I/O
• SSD + FlashCache – shutterflytested
• https://github.com/facebook/flashcache/
• Provisioned IOPS on AWS

6. 6
MongoDB Areas Of Performance Tuning
• Storage Tuning
• Ext4 + external journal on SSD – good I/O results
• Make sure to test
• journal_async_commit is used
• http://www.raid6.com.au/posts/fs_ext4_external_journal/

7. 7
MongoDB Areas Of Performance Tuning
Softlayerstudy/ benchmarking
“MongoDB Performance Analysis: Bare Metal v. Virtual”
http://blog.softlayer.com/2012/mongodb-performance-analysis-bare-metal-v-
virtual/
Highlights:
• When a working data set is smaller than available memory, query performance increases.
• The number of clients performing queries has an impact on query performance because more
data is being actively cached at a rapid rate.
• The addition of a separate Journal Mount volume significantly improves performance.
Because the Small (SM) engineered server does not include a secondary mount for Journals,
whenever MongoDB began to journal, the disk I/O associated with journaling was disruptive to
the query and update operations performed on the Data Mount.
• The best deployments in terms of operations per second,stability and control were the
configurations with a RAID10 SSD Data Mount and a RAID1 SSD Journal Mount. These
configurations are available in both Medium and Large offerings,and I’d highly recommend
them.

8. 8
MongoDB Areas Of Performance Tuning
OS Performancemonitoring and analysis tools:
• A) Linux utilities: iostat, vmstat, mpstat, sar, free -tm
• B) Open source monitoring / performance analysissystems: Nagios,
Cacti
• http://tag1consulting.com/blog/mongodb-cacti-graphs
• https://github.com/mzupan/nagios-plugin-mongodb
C) Oracle OS Watcher– Free tool, performance collection and chart
generator, easy to operate, very reliable, covers CPU, processes,
top processes, I/O, network, etc
OSWatcher Black Box Analyzer User Guide (Doc ID 461053.1)

9. 9
MongoDB Areas Of Performance Tuning
Database Tuning
• Database configuration
• Preallocatespace (noprealloc = false)
• journalCommitInterval – range of 2 to 300, default 100. Can be
adjusted to for better performance
• Option to disable services:
1. HTTP Interface using nohttpinterface = true.
2. ScriptingEngine using noscripting = true (server-side java
script).
3. RESTservice using rest = false.

10. 10
MongoDB Areas Of Performance Tuning
Database Tuning
• Database monitoring tools
• Mongostat– based on “eBay” “freecon” utility.
• Use “–discover” option for membersof replicaSet / shared
cluster
• Displays critical performance metrics:
• Flushes/Faults/locked db/idxmiss/qr/qw
• Collect 24x7 and save for analysis

11. 11
MongoDB Areas Of Performance Tuning
Database Tuning
• Database monitoring tools
• Mongotop
• Per collection I/O time tracking
• Per database lock tracking (--lock option)

12. 12
MongoDB Areas Of Performance Tuning
Database networktraffic analysis
• Database networkmonitoringtools
• Mongosniff– sniffing MongoDb requests
• Can capture invalid BSON requests from a network (--objcheck option)
• Wireshark– support for MongoDB protocol (not BSON), great
options for network performanceand latencyanalysis

13. 13
MongoDB Areas Of Performance Tuning
Database Tuning
• Collections/ Fragmentation
Collections in a MongoDB database can become fragmented. This can be a particularly serious problem if
data usage patterns are relatively unstructured. In the long run, this will result in databases taking up more
space on disk and in RAM to hold the same amount of data, it will make many database operations noticeably
slower, and it can reduce overall query capacity significantly.
compactcollection:
db.runCommand ( { compact: '<collection>', paddingBytes: 100 } )
Automate compaction:
http://blog.parse.com/2013/03/26/always-be-compacting/
./mongo_compact.rb -d userdata1,userdata2,userdata3

14. 14
MongoDB Areas Of Performance Tuning
Database Tuning
Profiler
db.setProfilingLevel(level, slowms)
db.setProfilingLevel(1)– collects slow requests
Output:
system.profile.op – operation
system.profile.ns – namespace
system.profile.ntoreturn – documentsreturned
system.profile.nscanned – documentsscanned
…
http://docs.mongodb.org/manual/reference/database-profiler/
Profilerdeep dive
https://speakerdeck.com/kgorman/mongodb-profiler-deep-dive

15. 15
MongoDB Areas Of Performance Tuning
Database Tuning
• Indexes
• Explain
db.collection.find({…}).explain()
Watch for scans
• Indexes + Profiler= DEX - suggest what indexesare needed
https://github.com/jwilder/mongodb-tools
http://architects.dzone.com/articles/mongodb-performance-tuning-
dex

16. 16
MongoDB Areas Of Performance Tuning
Database Monitoring and performance analysisUI
a) MongoDB monitoring service (MMS)
b) Open Source monitoringsoftware
http://docs.mongodb.org/manual/administration/monitoring/
c) Enteros High Load Capture for MongoDB

17. 17
MongoDB Areas Of Performance Tuning
• Replication/ Replica set tuning
Watch for replicationlag
Problem:
a) Weaksecondary
b) Write bursts
c) Index build
d) Secondarylocked for backup
e) Secondaryoffline
Measure lag:
db.printSlaveReplicationInfo()
db.printReplicationInfo()
Get time difference
MMS provides replication lag chart and alerts

18. 18
MongoDB Areas Of Performance Tuning
• Sharding Tuning
• Shard size
• Migration of chunks
• Failed migrationof chunks

19. 19
MongoDB Load Testing
Load testing
1) Whyload testing?
• Validate upgrades
• Validate fixes
• Validate hardware and platform changes (to / from cloud, etc)
• Validate multiplesof production load

20. 20
MongoDB Load Testing
2) Options for Load testing
A) BenchmarkLoad Testing
• Mongo-Perf - Create standard TPCC-like load on MongoDB
• https://github.com/mongodb/mongo-perf
B) Disk I/O load testing
• Mongoperf
• Iozone http://www.iozone.org/

21. 21
MongoDB Load Testing
C) Real traffic Load Testing
Parse PCAP files and generate production-like load
http://docs.mongodb.org/meta-driver/latest/legacy/mongodb-wire-
protocol/
Create custom tool
Enteros Load2Test (pre-GA)

22. Understandbusiness requirements
and it’s impact on production system
design
Designfor performance, scalability
and availability
Implement monitoring and performance
analysis across layers
Implement reliable production-like
load testing
MongoDB Summary

23. Enteros, Inc
http://www.enteros.com/
Enteros is an innovative software company specializing in
Performance Management and Load Testing Software for
Production Databases - RDBMS and NOSQL/Big Data
Enteros solutions enable IT professionals to identify
and remediate performance problems in business-critical
databases with unprecedented speed, accuracy and scope.
Ron Warshawsky; ron@enteros.com
408-207-8408
MongoDB Summary