Ticketmaster is the world leader in selling tickets. After more than a decade of developing applications extensively on Oracle and MySQL, Ticketmaster made the move to MongoDB. The reasons for the move are generally in line with those of other companies – increased flexibility and performance, and decreased costs and time-to-market. In this session we’ll discuss how the conversion to MongoDB went at Ticketmaster and we’ll take a deeper dive into some of the successes and set-backs that we faced. We’ll give an overview of the MongoDB topology at Ticketmaster, discuss exactly what data we’re writing to MongoDB and comment on the MongoDB support model that we’re using. We’ll also touch on the transition from relational DBA to NoSQL DBA at Ticketmaster.

1. A Front-Row Seat to Ticketmaster’s
use of MongoDB!
Ed Presz – VP, Database Services
Linda Xu – Director Database Development/Architecture
Tuesday, June 24 1:30-2:30PM

2. Live Nation Entertainment
The world's leading live entertainment and ecommerce company, comprised of
four market leaders: Ticketmaster.com, Live Nation Concerts, Front Line
Management Group and Live Nation Network
Ticketmaster.com
One of the world's top five ecommerce sites
Database with 350 million customers/fans worldwide
27 million monthly unique visitors to www.ticketmaster.com and
www.livenation.com
Overview – Live Nation/Ticketmaster
Ticketmaster and
Live Nation
merged in 2010
www.livenation.com
> 15,000 tickets/minute sold during large on-sales
Live Nation Concerts produces over 22,000 shows annually for more than
2,300 artists
Ticketmaster was official ticket company for 2012 London Olympics
2

3. MongoDB at Ticketmaster
3
WriteConcern
Replica Sets JSON/BSON
getLastError collections documents
election
oplog

4. The Move to MongoDB
Ticketmaster Database Services team
26 DBAs located globally
Strong Oracle/MySQL/MS SQL Server background
Consists of Operational DBAs, Architects and Database
Engineers
Database performance/scale is a huge part of what we
do
Transition to MongoDB
Flexible, development friendly
Ease of deployment/speed-to-market
Cost savings
Re-training relational DB experts (new challenge)
Performance trade-offs

5. MongoDB at Ticketmaster
February, 2012
Ticketmaster starts evaluating MongoDB
June, 2012
Begin development on MongoDB
May, 2013
Enterprise Subscription support for MongoDB
July, 2013July, 2013
Go live with MongoDB 2.2.4 for Ticketmaster
Resale
March, 2014
Upgrade to MongoDB 2.4 (currently using 2.4.9)
April, 2014
Reviewing MongoDB 2.6
Concurrency improvements in MongoDB 2.8,
including document-level locking.

6. MongoDB at Ticketmaster
6"Humongous“ – huge , monstrous

7. Ticketmaster MongoDB use-cases
MongoDB is used as a compliment to relational databases
(MySQL and Oracle)
Not used for highly transactional applications and/or sensitive
customer/financial data
Currently for TM+ (resale)
Order details/history (secondary market)Order details/history (secondary market)
Shopping cart and messages
Configuration management
Release tools
Concerts-related services in the future
Venue view (View from section)
Event Content Service
B2B (session recovery)
Client Reports

8. Operational challenge 1 – MongoDB Security
NoSQL security concerns
Earlier versions of MongoDB (prior to 2.0),
auth mode was not available (run in trusted
environment).
Big advancements in security with later
versions of MongoDBversions of MongoDB
Ticketmaster has enabled authentication
By default authentication is disabled
Edit your MongoDB configuration file to enable auth (auth = true) or
specify a shared key file for replica sets
MongoDB supports simple role-based authentication at the database
level
Only two permissions are available: read-only and read/write.
Read/write is the default.
Regular user setup is local to each database

9. Requirements: Implement MongoDB replica set that is highly redundant, fault tolerant,
and highly available in 2 datacenters. Need to provide a solution that allows datacenter #1
to be brought off-line for routine maintenance.
Operational challenge 2 – Geographically
Distributed Replica Sets
MongoDB Nodes:
Red Hat Enterprise Linux Server
release 5.8 (64-bit)
XenSever VM
Intel Xeon dual quad-core
processors w/32 GB RAM

10. Operational challenge 2 – Geographically
Distributed Replica Sets
Step #1:
Add Arbiter
Step #2:
Remove Arbiter
Step #4: Shutdown
Shift in primary
New Primary
Step #3:
Shutdown

11. Moved Arbiter to third datacenter
Operational challenge 2 – Geographically
Distributed Replica Sets

12. Operational challenge 3 – Performance on NFS
Performance on NFS
2 of 3 MongoDB outages with data rollback in
production were cased by NFS mount issues
Journal write performance is common bottleneck on
NFS
Journal file has direct disk mapping every 100ms
(default) or more frequently
MongoDB is sensitive to IO contentionMongoDB is sensitive to IO contention
Compete for resource on back-end filer with other
services on Live Nation system
Use NFS to take snapshot/backups. Application is
running 24*7, we can’t quiet writes to take consistent
backup

13. Operational challenge 3 - Performance on NFS
MongoDB writing to NETAPP filer.

14. Operational challenge 3 - Performance on NFS
Solution
Moved journal and data files to block device/local Storage
Build extra secondary replica on each data center and run
data and journal files on filer for (for snapshot/backup
purposes)
Notes
We tried moving journal only to block devices with limitedWe tried moving journal only to block devices with limited
performance gains.
Block device has limited quota size. Extending storage is not
easy. We added monitoring script to check space usage <
250GB
Non-standard setup using symlinks to point to data files and
journal files
Extra nodes have priority set to 1 to have less chance of
being promoted to primary

15. Operational challenge 3 - Performance on NFS
MongoDB writing to block device/local storage

16. Operational challenge 3 - Performance on NFS
Benefits
• Internal testing showed 20x speedup over NFS v3 when we moved to
local drive/block devices
• Besides performance gains, also less page faults, less queuing and %
locks
• Development also confirmed improved performance

17. Other Operational challenges
Understanding the write behavior ( WriteConcern vs
performance)
Write concern describes the guarantee on a write operation (i.e.
the strength of the write)
The weaker the WriteConcern, the faster the write operations
Backups (mongodump vs NETAPP filer snapshots vs
MMS)
Delay replicate slavesDelay replicate slaves
Adding indexes in MongoDB
Create index for replica set on one member at a time
Take nodes out of replica set to add indexes (bring up Mongo
instance with different port#)
Resizing the oplog
Shrinking the oplog was tricky (documentation geared towards
making oplog bigger)
We built database with default (5% of available space) with
default size for oplog. 130GB oplog at go-live.

18. MongoDB Support
Enterprise Support
42 JIRA tickets open for MongoDB support
with company = “Ticketmaster”
Overall, good turnaround
Sample JIRA tickets….
Setting up Arbiter documentation - still initializingSetting up Arbiter documentation - still initializing
Resizing oplog does not seem to shrink the size of the local data files
Very high CPU/iowait and long read/write locks on primary Master yesterday
evening
MongoDB query review

19. MongoDB Monitoring (MMS 1.4)

20. Let’s have some fun!
• O2 Arena in London, U.K.
• Phones 4u Arena in Manchester, U.K.
• Barclays Center in Brooklyn, NY
• Bon Jovi
• Michael Jackson The Immortal World Tour
• P!nk
Top Tours of 2013

21. A Front-Row Seat to Ticketmaster’s
use of MongoDB!
Ed Presz – VP, Database Services, ed.presz@ticketmaster.com
Linda Xu – Director Database Development and Architecture,
linda.xu@ticketmaster.com
Question and Answer!