Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (20)
Semelhante a 2012-11-30-scalable game servers
Semelhante a 2012-11-30-scalable game servers (20)
Mais de Wooga
Mais de Wooga (20)
Último
Último (20)
2012-11-30-scalable game servers
- 1. Scalable game servers Knut Nesheim @knutin tirsdag 4. desember 12
- 2. tirsdag 4. desember 12
- 3. What we do • Casual social games • Casual: simpler, easier, nicer, cuter • Social: interact with your friends • Average title: 1 000 000 players every day tirsdag 4. desember 12
- 4. Purpose of backend • Play anywhere, HTTP • Cheat prevention • Ensure state integrity • Highscores • Multiplayer • Payments tirsdag 4. desember 12
- 5. The first backend tirsdag 4. desember 12
- 6. The first backend: What • Stateless: state only in DB, mutate in app • Linux, on EC2 • Apache • MySQL • Ruby on Rails tirsdag 4. desember 12
- 7. The first backend: Challenges • MySQL becomes a bottleneck • Sharding necessary • Working data set must fit in memory • Slow restores tirsdag 4. desember 12
- 8. The first backend: Challenges app db 10+ db reqs per HTTP req tirsdag 4. desember 12
- 9. The first backend: Challenges • Very sensitive to network latency, synchronous IO • 16 ruby processes per node * 125 nodes = 2000 concurrent reqs • 10K HTTP rps / 2000 workers = 5 reqs/worker/second • Each request must be served in 1000ms / 5 = 200 ms tirsdag 4. desember 12
- 10. The first backend: Solutions Challenge Solution MySQL bottleneck Shard, tune, use Redis Sensitive to latency Redis, more app servers ~200 nodes Chef, Scalarium tirsdag 4. desember 12
- 11. The first backend: Conclusions • Pain to operate massive DB cluster • Must read & write on master • Many single points of failure • Inefficient • All data in memory tirsdag 4. desember 12
- 12. The second backend tirsdag 4. desember 12
- 13. The second backend: What • Evolution • Dedicated hardware and network • Ruby on Rails • Redis only • Fewer and faster DB reqs tirsdag 4. desember 12
- 14. The second backend: Conclusions • Good progress • Still sensitive to latency • Single points of failure • Can we do better? tirsdag 4. desember 12
- 15. The third backend tirsdag 4. desember 12
- 16. The third backend: What • Revolution • Stateful: app stores state while user is online, flush to disk when user goes offline • Only used data in memory • Not sensitive to latency • Use Database-as-a-service tirsdag 4. desember 12
- 17. The third backend: Challenges • Erlang • State • Resource locking • Load balancing • Deployment • Lack of libraries tirsdag 4. desember 12
- 18. The third backend: Challenges State • Data structures • Functional language • Immutability • Transactions tirsdag 4. desember 12
- 19. The third backend: Challenges Resource locking • Only one process per user • Central serialization • Redis, CAS, SETNX • Single point of failure tirsdag 4. desember 12
- 20. The third backend: Challenges Load balancing • Users play for 3-5 minutes • “proxy” forwards requests • “worker” notifies “proxy” about health tirsdag 4. desember 12
- 21. The third backend: Challenges Deployment • Use Erlang code reload • “git pull && make && ./upgrade.sh” • State migration on the fly • Rolling upgrades tirsdag 4. desember 12
- 22. The third backend: Challenges Lack of libraries • No CPAN, PiP, RubyGems • Roll our own • Eredis, github.com/wooga/eredis tirsdag 4. desember 12
- 23. The third backend: Conclusions First backend Second backend Third backend Frequent downtime Little downtime No downtime 150+ servers 10+ servers 3 servers 100k DB reqs 10k DB reqs 1k DB reqs :( :| :) tirsdag 4. desember 12
- 24. The fourth backend tirsdag 4. desember 12
- 25. The fourth backend: What • Evolution • Multiplayer! • Real-time push updates • Improve operations • Use hosted databases • Remove SPOF tirsdag 4. desember 12
- 26. The fourth backend: Challenges Multiplayer • User and world as Erlang processes • Serialization • Reason about concurrency • Scale tirsdag 4. desember 12
- 27. The fourth backend: Challenges Real-time push updates • Push updates to client • Need browser support • “Transfer-Encoding: chunked” • Elli, github.com/knutin/elli tirsdag 4. desember 12
- 28. The fourth backend: Challenges Improve operations • Better load balancing • Nodes broadcasts “alive” message • Local failure detector • Divergent cluster views tirsdag 4. desember 12
- 29. The fourth backend: Challenges Improve operations, cont. • Collect more metrics • ETS • Histograms • github.com/knutin/statman tirsdag 4. desember 12
- 30. The fourth backend: Challenges Hosted databases • Amazon AWS • DynamoDB • S3 • Low operational overhead • Higher availability tirsdag 4. desember 12
- 31. The fourth backend: Challenges Remove SPOF • “locker”: distributed consistent leases • Atomic CAS • Write quorum, 2PC • Eventually consistent replication • 330 SLOC tirsdag 4. desember 12
- 32. The fourth backend: Conclusions • Still in development • Benchmarks shows good results • Must prove it live tirsdag 4. desember 12
- 33. The future backends tirsdag 4. desember 12
- 34. The future backends • Erlang: processes, good IO, clustering • Harder, better games • Faster, stronger backends • Stateful, also with JVM • Another revolution? tirsdag 4. desember 12
- 35. Conclusions • Understand the past • The future is stateful • Identify, think, do tirsdag 4. desember 12