The document discusses the evolution of architecture at the gaming company Wooga. It began by using Ruby on Rails with a MySQL database, which worked for low user numbers but struggled as users grew. A second team then used Redis as the main database instead of MySQL for its speed. The teams combined approaches, migrating static data to MySQL and keeping dynamic data in Redis. This allowed scaling to millions of daily users. The architecture evolved further to make servers and databases stateless to improve reliability.
3. DEV -‐ OPS
Some say: Admins learn to be ‘Agile’
• Scrum and Kanban
4. DEV -‐ OPS
Some say: Admins learn to be ‘Agile’
• Scrum and Kanban
Others say: Admins learn to program their setup
• Chef, Puppet
5. DEV -‐ OPS
Some say: Admins learn to be ‘Agile’
• Scrum and Kanban
Others say: Admins learn to program their setup
• Chef, Puppet
I say: This is all nice, but not enough
• Developers need to learn a lot about opera=on
30. 2 Developers to do it all
Typical team setup
4 product managers
4 ar=sts
4 frontend engineers
2 backend engineers
-‐ design, implementa7on, opera7on
45. Welcome to 6 weeks of pain!
Heavy opMmizaMons were necessary
46. Welcome to 6 weeks of pain!
Heavy opMmizaMons were necessary
Numerous small fixes regarding DB config
47. Welcome to 6 weeks of pain!
Heavy opMmizaMons were necessary
Numerous small fixes regarding DB config
More shards
48. Welcome to 6 weeks of pain!
Heavy opMmizaMons were necessary
Numerous small fixes regarding DB config
More shards
Even more shards
49. Welcome to 6 weeks of pain!
Heavy opMmizaMons were necessary
Numerous small fixes regarding DB config
More shards
Even more shards
SpliNng the model to get more shards
50. Early sharding hell: 8 master and 8 slaves
lb
app app app app app app app app app
app app app app app app app app app
My My My My My My My My
SQL SQL SQL SQL SQL SQL SQL SQL
slave slave slave slave slave slave slave slave
51. At 500K daily users we were at a dead end
&$!!!$!!!"
%$#!!$!!!"
%$!!!$!!!"
#!!$!!!"
!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
62. Migrate data from MySQL to Redis
No downMmes allowed
Migrate data on demand
63. Migrate data from MySQL to Redis
No downMmes allowed
Migrate data on demand
On every access check if data needs migra=on
64. Migrate data from MySQL to Redis
No downMmes allowed
Migrate data on demand
On every access check if data needs migra=on
Migrate if needed
65. Migrate data from MySQL to Redis
No downMmes allowed
Migrate data on demand
On every access check if data needs migra=on
Migrate if needed
Con=nue using Redis
67. Big and staMc data in MySQL, rest goes to Redis
256 GB data 60 GB data
10% writes 50% writes
hOp://www.flickr.com/photos/erix/245657047/
68. One team saved the other one
&$!!!$!!!"
%$#!!$!!!"
%$!!!$!!!"
#!!$!!!"
!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
69. One team saved the other one
&$!!!$!!!"
%$#!!$!!!"
%$!!!$!!!"
#!!$!!!"
!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
70. We now have more than 2 million users / day
&$!!!$!!!"
%$#!!$!!!"
%$!!!$!!!"
#!!$!!!"
!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
71. We now have more than 2 million users / day
&$!!!$!!!"
%$#!!$!!!"
%$!!!$!!!"
AWS outage
#!!$!!!"
in Ireland
!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
72. 10 single-‐points-‐of-‐failure -‐ no fun at all!
lb lb
app app app app app app app app app app app app app
app app app app app app app app app app app app app
app app app app app app app app app app app app app
My My My My My
redis redis redis redis redis
SQL SQL SQL SQL SQL
slave slave slave slave slave slave slave slave slave slave
91. Stateful servers are not as hard as you think
Server
session
session
session
session
92. Stateful servers are not as hard as you think
Server
session
session
session
session
S3
93. Stateful servers are not as hard as you think
Server
session
session
session
session
S3
94. Stateful servers are not as hard as you think
Server
session
session
session
session
S3
95. Stateful servers are not as hard as you think
Server
session
session
session
session
S3
96. Stateful servers are not as hard as you think
Server
session
session
session
session
S3
97. Stateful servers are not as hard as you think
Server
session
session
session
session
S3
98. Stateful servers are not as hard as you think
Server Server Server
session
session session
session session
session
session
session session
session session
session
S3
99. With stateful server the DB is less used
Ruby Stateless Erlang Stateful
30,000
22,500
15,000
7,500
0
database operations / sec
100. With stateful server the DB is less used
Ruby Stateless Erlang Stateful
30,000
22,500
15,000
700
7,500
0
database operations / sec
101. Deploying with a stateful server
In order to bring up a new version
102. Deploying with a stateful server
In order to bring up a new version
Just deploy it
Hot code replacement is great!
103. There are even more advantages
Faster than Ruby (5,000 rps / node)
-‐ CPU bound
104. There are even more advantages
Faster than Ruby (5,000 rps / node)
-‐ CPU bound
Very few SPOFs
-‐ ... and those are easy to recover
105. There are even more advantages
Faster than Ruby (5,000 rps / node)
-‐ CPU bound
Very few SPOFs
-‐ ... and those are easy to recover
TransacMonal logic
-‐ Invariants instead of explicit error handling
113. Architecture EvoluMon at Wooga
The Start: Ruby
The Next Step: Erlang
Best of Two Worlds
Company Values
114. Aug 2011: 4th team wanted both
Erlang is great
Concurrency, robustness
Great for opera=on
115. Aug 2011: 4th team wanted both
Erlang is great
Concurrency, robustness
Great for opera=on
Ruby is great
Concise, expressive, testable
Great for development
116. Aug 2011: 4th team wanted both
Erlang is great
Concurrency, robustness
Great for opera=on
Ruby is great
Concise, expressive, testable
Great for development
117. Aug 2011: 4th team wanted both
Erlang is great
Concurrency, robustness
Great for opera=on
Ruby is great
Concise, expressive, testable
Great for development
118. EvoluMon IV: The best out of two worlds
Aug 2011
Oct 2010
Jan 2010
Oct 2009
119. The basic setup looks exactly like before
Server Server Server
session
session session
session session
session
session
session session
session session
session
S3
121. Bringing 2 worlds together
Server
session
sender
session
...
session
122. Bringing 2 worlds together
Server Worker
session
sender Worker
session
Worker
...
Worker
session
Worker
123. Bringing 2 worlds together
Server Worker
session
sender Worker
session
Worker
...
Worker
receiver
session
Worker
124. Bringing 2 worlds together
Server Worker
session
sender Worker
session
Worker
...
Worker
receiver
session
Worker
125. Bringing 2 worlds together
Server Worker
session
sender Worker
session
Worker
...
Worker
receiver
session
Worker
126. Game state
Game state is split in mulMple parts
user, map, fruit_trees etc.
127. Game state
Game state is split in mulMple parts
user, map, fruit_trees etc.
Erlang does not care about content
Serialized Ruby objects
128. Game state
Game state is split in mulMple parts
user, map, fruit_trees etc.
Erlang does not care about content
Serialized Ruby objects
Erlang does know mapping of state parts to URLs
Mapping provided by Ruby on startup
131. Looking back at the game acMon
Mapping of state parts to game acMons
132. Looking back at the game acMon
Mapping of state parts to game acMons
Worker knows mapping
133. Looking back at the game acMon
Mapping of state parts to game acMons
Worker knows mapping
Worker pushes mapping to Erlang on startup
134. Looking back at the game acMon
Mapping of state parts to game acMons
Worker knows mapping
Worker pushes mapping to Erlang on startup
Erlang can query mapping if needed
143. A good value system
We’ve learned to value
Small teams over Big teams
CollaboraMon over Compe==on
Generalists over Specialists
Effort reducMon over Cost reduc=on
InnovaMon over Risk mi=ga=on