Mais conteúdo relacionado Semelhante a Painful success - lessons learned while scaling up (20) Painful success - lessons learned while scaling up9. 7M players / month
10K API calls / second
200K DB operations / second
10. 7M players / month
10K API calls / second
200K DB operations / second
100K DB writes / second
11. 7M players / month
10K API calls / second
200K DB operations / second
100K DB writes / second
2 devops since 2009
24. Browser
lb
app app app
SQL SQL
sql sql
40. app
1. create new DBs
master master
slave slave
41. app
1. create new DBs
master master
2. setup replication
slave slave
42. app
1. create new DBs
master master
2. setup replication
slave slave
43. app
1. create new DBs
master master
2. setup replication
3. start using masters
slave slave
44. app
1. create new DBs
master master
2. setup replication
3. start using masters
4. cut replication
slave slave
45. app
1. create new DBs
master master
2. setup replication
3. start using masters
4. cut replication
slave slave
5. remove old DBs
46. app
1. create new DBs
master
2. setup replication
3. start using masters
4. cut replication
slave
5. remove old DBs
47. app
1. create new DBs
master master
2. setup replication
3. start using masters
slave slave
48. app
1. create new DBs
master master
2. setup replication
3. start using masters
4. remove old DBs
slave slave
49. app
1. create new DBs
master
? 2. setup replication
3. start using masters
4. remove old DBs
slave
50. app
1. create new DBs
master
2. setup replication
3. start using masters
4. remove old DBs
slave
5. cut replication
51. app
1. create new DBs
master master
2. setup replication
3. start using masters
slave slave
52. app
1. create new DBs
master master
2. setup replication
3. start using masters
4. drop database;
slave slave
53. app
1. create new DBs
master master
2. setup replication
3. start using masters
4. drop database;
slave slave
54. app
1. create new DBs
master master
2. setup replication
3. start using masters
4. drop database;
slave slave
55. app
1. create new DBs
master master
2. setup replication
3. start using masters
4. drop database;
slave slave
56. app
1. create new DBs
master master
2. setup replication
3. start using masters
4. drop database;
slave slave
57. app
1. create new DBs
2. setup replication
3. start using masters
4. drop database;
58. app
1. create new DBs
2. setup replication
3. start using masters
4. drop database;
5. Nooooooooo!
59. app
1. create new DBs
2. setup replication
3. start using masters
4. drop database;
5. Nooooooooo!
62. app
1. create new DBs
master master
2. setup replication
3. start using masters
4. cut replication
slave slave
5. truncate data
63. app
1. create new DBs
master master
AB AB 2. setup replication
3. start using masters
4. cut replication
slave slave
AB AB 5. truncate data
64. app
A B
1. create new DBs
master master
AB AB 2. setup replication
3. start using masters
slave slave
AB AB
65. app
A B
1. create new DBs
master master
AB AB 2. setup replication
3. start using masters
4. cut replication
slave slave
AB AB
66. app
A B
1. create new DBs
master master
AB AB 2. setup replication
3. start using masters
4. cut replication
slave slave
AB AB 5. truncate data
67. app
A B
1. create new DBs
master master
AB AB 2. setup replication
3. start using masters
4. cut replication
slave slave
AB AB 5. truncate data
68. app
master master master master
AC BD
slave slave slave slave
AC BD
69. app
master master master master
AC BD AC BD
slave slave slave slave
AC BD AC BD
70. app
master master master master
AC BD AC BD
slave slave slave slave
AC BD AC BD
71. app
master master master master
AC BD AC BD
slave slave slave slave
AC BD AC BD
77. 44 GB
in 8 days
24 GB
82. 38 GB
in 3 days
24 GB
83. v2.2
in
ixed
F
38 GB
in 3 days
24 GB
85. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL
sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd
86. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL
sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd
87. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app SQL
app app app app app
EBS EBS EBS EBS
SQL SQL SQL SQL SQL SQL
sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd
88. lb app app
lb app app
Browser SQL EBS
lb app app
lb app app
89. lb app app
lb app app
Browser SQL EBS
lb app app
lb app app
90. lb app app
lb app app
Browser SQL EBS
lb app app
lb app app
91. lb app app
lb app app
Browser SQL EBS
lb app app
lb app app
92. lb app app
lb app app
Browser SQL EBS
lb app app
lb app app
93. lb app app
lb app app
Browser SQL EBS
lb app app
lb app app
94. lb app app
lb app app
Browser SQL EBS
lb app app
lb app app
95. lb app app
lb app app
Browser SQL EBS
lb app app
lb app app
99. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL
sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd
100. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL
sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd
120. Server Database
One Game Session
121. Server Database
One Game Session
122. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL
sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd
123. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
Rd
rd
124. lb lb lb lb
app app app
Rd
rd
129. 1. Separate user and world DBs
2. Migrate to JRuby
3. Introduce life cycle
4. Remove (user) DBs
130. 1. Separate user and world DBs
2. Migrate to JRuby
3. Introduce life cycle
4. Remove (user) DBs
5. Enjoy!
132. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL
sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd rd
133. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL
sql sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd rd
134. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL
sql sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd rd
135. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL
sql sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd rd rd rd
143. Throughput
200
a rk
150
c hm
ben
n o
100
50
0
Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8
MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2
144. Throughput
200
a rk
150
c hm
ben
n o
100
50
0
Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8
MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2
145. Throughput
200
a rk
150
c hm
ben
n o
100
multi
threaded
50
0
Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8
MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2
146. Throughput
200
a rk
150
c hm
ben tuning
n o
100
multi
threaded
50
0
Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8
MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2
147. Throughput
200
cpu
a rk saturation
150
c hm
ben tuning
n o
100
multi
threaded
50
0
Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8
MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2
148. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL
sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd
149. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL
sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd
162. servlet archiver
User
app
Registry
163. servlet archiver
User
app
Registry
S3
164. servlet archiver
User
app
Registry
SQL Rd
S3
165. servlet archiver
User
app
Registry
SQL Rd
S3
166. servlet archiver
User
app
Registry
World
SQL Rd
Data
S3
167. Load Balancer
servlet archiver
User
app
Registry
World
SQL Rd
Data
S3
168. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL
sql sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd rd rd rd
169. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL
sql sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd rd rd rd rd
171. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL
sql sql sql sql sql sql
Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd rd rd rd rd rd rd
172. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL SQL
sql sql
Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd
173. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL
sql sql
Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd
175. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL SQL
sql sql
Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd
176. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
SQL SQL
sql
Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd
177. lb lb lb lb
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
app app app app app app app app app app app app app app app
Rd Rd Rd Rd Rd Rd
rd rd rd rd rd rd
189. User
Cus-
Avatar
tomer
190. User
Cus-
Avatar Garden
tomer
191. User
Cus-
Avatar Garden
tomer
Deco-
Tiles
rations
192. User
Cus-
Avatar Garden
tomer
Deco-
Tiles
rations
193. User User
Cus- Cus-
Avatar Garden Avatar Garden
tomer tomer
Deco- Deco-
Tiles Tiles
rations rations
194. User User
Cus- Cus-
Avatar Garden Avatar Garden
tomer tomer
Deco- Deco-
Tiles Tiles
rations rations