Mais conteúdo relacionado Semelhante a MySQL@king (20) Mais de Ted Wennmark (17) MySQL@king1. © King Digital Entertainment plc 2014MySQL @ KingScaling Candy CrushOlle Nilsson, Senior Database Engineer @ KingTed Wennmark, Senior Principal Sales Consultant @ Oracle 2. © King Digital Entertainment plc 2014
Table of contents
2
Introductions
Scaling MySQL
Scaling Candy Crush Saga
Q & A 3. © King Digital Entertainment plc 2014
Introductions
3
Olle
•Senior Database Engineer @ King
•Previously in MySQL Consulting @ Oracle EMEA
•Developer and Consultant @ Upsys (MySQL Pratner)
•Often colleagues with Ted
Ted
•Senior Principal Sales Consultant @ Oracle EMEA
•Developer and Consultant @ Upsys (MySQL Pratner)
•Long time MySQL user & friend of Olle 4. We make great games
4
-More than190 funtitles played in over 200 countries around the world
-345 million averagemonthlyuniqueusers(Q2 2014)
-Studios in Stockholm, London, Barcelona, Bucharest, Malmo, Berlin and Singapore. Offices in San Francisco, Malta, Tokyo, Seoul and Shanghai
About King
© King Digital Entertainment plc 2014 5. © King Digital Entertainment plc 2014 5
Some stats and factsMillions of players around the world
•345mm MUUs as of Q2 2014
•138mm DAUs as of Q2 2014
•More players playing at least one of our games each month than thepopulation of the US! Games popular across platforms, and can be played anywhere, anytime on any deviceGlobal leader in cross- platform casual games
•Three games in all three major top ten grossing lists for Q2 2014
•Our Saga games allow players to switch platform without losing their progress
•Four global franchises: Candy Crush Pet Rescue Farm Heroes Bubble Witch
•Founded in 2003, studios in Stockholm, London, Barcelona, Malmo, Bucharest, Berlin and Singapore
•900+ employees 7. “Saga” concept
7
Play a leveland get a score and earn1-3 stars
Progress tothe nextlevel
Seethe progress ofyourfriends!
Progress is storedand accessiblecross platform/device.
Historyand background 9. © King Digital Entertainment plc 2014
Scaling MySQL
9
Scale up / Vertical scaling
•Buy more or faster disk
•Buy more memory
•Buy more or faster CPU’s
Scale out / Horizontal scaling
•Shard by function
•Shard by columns
•Shard by rows 11. © King Digital Entertainment plc 2014
What is sharding?
11
Divide your dataset into several pieces, think of horizontal partitioning where each partition resides on dedicated instance/HW.
Statements are spread amongst all partitions.
Each shard can be located on separate servers. 12. © King Digital Entertainment plc 2014
Why shard?
12
Performance
•Smaller working set of data on each shard. Working set fits (better) in memory
•Fetch data in parallel from many shards.
Size
•Database too big and will not fit on disk/memory on one server
GEO latency
•You want to shard on GEO and have ’local’ shards close to your users 13. © King Digital Entertainment plc 2014
Getting started
13
Choosing shard-key and type of shard distribution
•RANGE, HASH, etc
•USER, TIME, etc
Operations
•Shard move
•Shard split
•Fail-over / Switch-over
Monitoring
•Identify quickly growing or ”Hot” shards
Interfaces and metadata for sharding
•DIY
•MySQL Fabric or similar framework 14. © King Digital Entertainment plc 2014
Sharding challenges
14
JOINs
•Tables are spread over multiple servers
AUTO_INCREMENTs / Sequences
•In MySQL these are per table
Backups
•Coordinating over multiple servers
Complex operations
•Schema changes, shard splitting, sharding SW
Complex HA
•Stand-by servers also need to be sharded
Management (deploy / orchestration)
•Automation a must 15. © King Digital Entertainment plc 2014
Sharding architecture
15
1.Lookup shard for User 100
2.User 100 is on Shard 1
3.Fetch data related to User 100 from shard Shard 1
4.User data for User 100 sent from Shard 1
sh4
sh3
sh2
sh1
META
Sharding API
Apps
1
2
3
4 16. © King Digital Entertainment plc 2014
Sharding Frameworks
16
MySQL Fabric
•Based on plain old MySQL Replication
•Requires GTID
•Fabric aware connectors
•Automatic failover 17. © King Digital Entertainment plc 2014
Sharding Frameworks
17
MySQL Cluster
•Automatic sharding
•Shared nothing architecture
•No SPOF
•IMDB
•ndbmtd, ndb_mgm, and API nodes 18. © King Digital Entertainment plc 2014
Sharding Frameworks
18
Gizzard (Twitter)
DbShards
Hibernate Shards
ScaleBase
Many in-house custom made solutions 20. 20
Scaling Candy Crush Saga
Candy Crush Numbers:
849M game plays per day
84M Candy Crush DAU
Served by MySQL
As of Q2 2014
© King Digital Entertainment plc 2014 21. 21
ShardingCandy Crush Saga
Sharding architecture in place before Candy Crush explosion
Only 4 people in Operations
10-20 new DB servers deployed every month at peak (+ ~50 App servers + Memcached + BI)
No dedicated DB staff
© King Digital Entertainment plc 2014 23. © King Digital Entertainment plc 2014
Architecture overview
23
250 000requests / s
50 000writes / s
(60 MB/s)
1 800 000reads / s
250 000writes / s
As of Q2 2014 24. © King Digital Entertainment plc 2014
Regular operation
24
MySQL Sharding 25. © King Digital Entertainment plc 2014
MySQL Numbers
25
User-Shards:
•> 50TB
•> 500K qps
•10/1 Read/Write
•Several billion row tables
•100+ instances
•MySQL 5.1 and 5.6 (5.0 and 5.5 used for inhouse applications)
•Memcached: Roughly 2M qps
•App servers caches in front of Memcached
As of Q2 2014 26. © King Digital Entertainment plc 2014
MySQL Numbers
26
•Hardware
################################################################################
#### hostXXX (debian/7.4 -Linux 3.2.0-4-amd64) ####
#### XYXXYZXYZXYZXYZ member of: SOME IMPORTANT GROUP ####
#### ProLiant DL380p Gen8 ####
#### Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz 48x2.4 Ghz ####
#### 379 GB ####
################################################################################
SSDs in RAID10 for data
HDDs in RAID 10 for logs 27. MySQL @ King
27
Main database technology at King
Backs all our games
Plenty of other applications are backed by MySQL
•Customer support
•Custom built BI/Analysis applications
•Buisness Systems (Collaboration, issue tracking, etc)
•Other internal applications, e.g., developer support utilities
© King Digital Entertainment plc 2014 28. © King Digital Entertainment plc 2014
Database tier
28
The butter and cream of the storage world:
MySQL
+
Memcached
Makes everything better! 29. © King Digital Entertainment plc 2014
MySQL Requirements
29
Mission Critical
Critical functions break without it
Mobile clients can playwithout connectivity. Client game state overrides on reconnect
Sub millisecond response time expected. > 10ms generates alarm and throttling of database usage
No bank 30. © King Digital Entertainment plc 2014
Database Architecture and Setup
30
Game state data for all users are stored
No pruning or purging
State stored as JSON blob
Sharded MySQL databases
UID sharding key
One central MySQL shard catalogue data 31. © King Digital Entertainment plc 2014
Application load patterns
31
Almost exclusively Key-Value
Friend data read on client start up. Populates Saga.
No joins in shards
Some JOIN and more SQL in shard catalogue
Sharded, highly available, KV-Store, JSON (Recognize this architecture?) 32. © King Digital Entertainment plc 2014
Sharding Implementation
32
Central store used for
•User/Shard lookup
•Purchases
•Registration
•Application management console for
Configuring Shard/Instance mappings
Application user configuration
Marketing etc
Performance monitioring
User Shards used for game state
Data volume main issue
Rich toolkit for application management
Building tools for MySQL management 33. © King Digital Entertainment plc 2014
HA and Tooling
33
KISS
Master/Slave pairs. No automatic failovers
Shard catalogue has multiple slaves
Home built tools for switching, installing, provisioning, moving shards, etc
Cacti, VividCortex, MEM, Ganglia, Graphite
Nagios
Xtrabackup/mysqldump
Percona toolkit 34. © King Digital Entertainment plc 2014
Monitoring of our databases!
34
35. 35
Challenges @ King
HDD performance
Data Volume
HDD => SSD 100G => SSD 200G => SSD 400G
Delivery times for new HW
Racking
Buy batches of pre-rackedHW
Network capacity issues (Memcached/backups)
Hetergenous environment
© King Digital Entertainment plc 2014 36. 36
Challenges @ King
Relations/Friendships/Joins: joins avoided
Volume: one of our main issues
Performace/Scalability: Working great
Complex Operations: problematic, lots of new components needed + consolidations, standardization
Backups: Taken lots of work. Needs more
Monitoring: App good. DB not yet stable
DB Connections: Typical app server issues. 10K+ connections to DB
© King Digital Entertainment plc 2014 37. 37
Solutions
Sharding. Just worked
Throwing HWat problems
MySQL, easy to use, well known, stable, plenty of 3rd party tools, multiple vendors, etc.
Get more staff
© King Digital Entertainment plc 2014 38. Conclusion
38
Pros: Scaled out with very few problems. All games on one DB tier.
Cons: Very rawMySQL environment. No automationand tooling. Resource allocationskew.
© King Digital Entertainment plc 2014 39. © King Digital Entertainment plc 2014
Current & Future Work
39
Managing the environment: Fire fighting, maintenance, new features, etc
Data Volume management: InnoDB native compression, application compression, avoiding fragmentation (current workload fragments a lot)
Automation, tooling, and Spring Cleaning of organically grown MySQL environment
Document & Standardize
Resource Utlization: CPU in abundance, storage scarce
Migrate to 5.6 (so we can migrate to 5.7?)
Improvementsto data-model, HW, new-games/new features
Backupsolution improvements. DR improvements.
Consolidationof minor in-house apps
Handling number of connections 40. © King Digital Entertainment plc 2014
Q & A
Yes! We are hiring!!
about.king.com/jobs
40