This document summarizes the scaling of a fantasy strategy game called Utopia Kingdoms from 4 users to over 90,000 users. It discusses the technology stack including Python, CherryPy, Amazon SimpleDB, and frontend technologies. It also describes some performance issues encountered with the database and solutions attempted, such as migrating from SimpleDB to MongoDB and implementing caching, queueing actions, and optimizing database queries. The importance of ongoing measurement and monitoring to identify bottlenecks is emphasized.
4. Utopia Kingdoms
● Fantasy strategy game
● Build your own Kingdom
● Create armies and attack other Kingdoms
● Join other Kingdoms in an Alliance
● Manage resources
● Available in Facebook and Kongregate
http://www.facebook.com/UtopiaKigdomsGame
http://www.kongregate.com/games/JoltOnline/utopia-kingdoms
9. Some points of interest
(will discuss them later)
●
Your resources (population, gold, food, etc)
grows with time
●
You actions (build something, attack a player)
typically takes some time
●
Players are ranked against the rest
●
You can add friends and enemies
11. Measurement tools
● OS tools
● Task manager (top)
●
IO Monitor (iostat)
● Monitoring tools (Munin, Nagios)
● Logs
●
Needs to find a good compromise detailed/relevance
● Profiling
12. You've got to love profiling
●
Generate profiles with cProfile module
Profile whole application with
python -m cProfile -o file.prof my_app.py
(not very useful in a web app)
●
If you're using a framework, profile only your
functions to reduce noise
14. Analyzing profile
●
gprof2dot
●
Using dot, convert to graph
gprof2dot -f pstats file.prof | dot -Tpng -o file.png
●
Good for workflows
●
RunSnakeRun
●
Good for cumulative times
18. All static should be out of python
●
Use a good web server to serve all static
content (Static HTML, CSS, JavaScript code)
●
Some options
●
Apache
●
Nginx
●
Cherokee
●
Amazon S3
20. Example
●
Asking for friends/enemies to DB
●
Costly request in SimpleDB (using SQL statement)
●
On each request
●
Cache the friends on memcache for 1 hour
●
Invalidate the cache if adding/removing
friends or enemies
21. Caching caveats
●
Cache only after knowing there is a problem
●
Do not trust in cache for storage
●
Take a look on size of cached data
●
Choosing a good cache time can be difcult /
Invalidate cache can be complex
●
Some data is too dynamic to be cached
22. Caching is not just memcached
●
More options available:
●
Get on memory on start
●
File cache
●
Cache client side
23. Parse templates just once
●
The template rendering modules have options
to parse the templates just once
●
Be sure to activate it in production
●
In development, you'll most likely want to
parse them each time
●
Same apply to regex, specially complex ones
25. Rankings
●
Sort players on the DB is slow when you grow
the number of players
●
Solution:
●
Independent ranking server (operates just in
memory)
●
Works using binary trees
●
Small Django project, communicates using xmlrpc
●
Inconvenient:
●
Data is not persistent, if the rankings server goes
down, needs time to reconstruct the rankings
26. Database pulling - Resources
●
There was a process just taking care of the
growth of resources.
●
It goes element by element, and increasing the
values
●
It pulls the DB constantly, even when the user has
their values to maximum
●
Increment the resources of a user just the next
time is accessed (by himself or by others)
●
No usage of DB when the user is not in use
●
The request already reads from DB the user
27. Database pulling - Actions
●
Lots of actions are delayed. Recruit a unit,
buildings, raids...
●
A process check each user if an action has to
be done NOW.
●
Tons of reads just to check “not now”
●
Great delay in some actions, as they are not
executed in time
28. Database pulling - Actions
●
Implement a queue to execute the actions at
the proper time:
●
Beanstalk (allows deferred extraction)
●
A process listen to this queue and performs the
action, independently from request servers.
●
The process can be launched in a diferent
machine.
●
Multiple process can extract actions faster.
30. Amazon SimpleDB
●
Key – Value storage
●
Capable of SQL queries
●
Store a dictionary (schemaless, multiple
columns)
●
All the values are strings
●
Access through boto module
●
Pay per use
31. Problems with SimpleDB
●
Lack of control
●
Can't use local copy
– In development, you must access Amazon servers (slow
and costly)
●
Can't backup except manually
●
Can't analyze or change DB (e.g. can't define
indexes)
●
Can't monitor DB
32. Problems with SimpleDB
●
Bad tool support
●
Slow and high variability (especially on SQL
queries)
●
Sometime, the queries just timeout and had to be
repeated.
34. MongoDB
●
NoSQL
●
Schemaless
●
Fast
●
Allow complex queries
●
Retain control (backups, measure queries, etc)
●
Previous experience using it from ChampMan
35. Requisites of the migration
●
Low-level approach
●
Objects are basically dictionaries
●
Be able to save dirty fields (avoid saving
unchanged values)
●
Log queries to measure performance
36. MongoSpell
●
Thin wrap over pymongo
●
Objects are just dictionary-like elements
●
Minimal schema
●
Fast!
●
Able to log queries
●
It will probably be released soon as Open
Source
42. Scalable vs Efcient
Scalable Efficient
●
Can support more ●
Can support more
users adding more users with the same
elements elements
Work on both to achieve your goals