Amitt Mahajan discusses the techniques used in the development of the hit social game FarmVille. This presentation covers the processes used by the FarmVille development team to quickly release the game and then scale it to support millions of daily players.
(Originally presented at GDC 2010)
6. My Goals Reduce the amount of time it takes you to develop your games Give you a blueprint for how to reliably scale your games to millions without your servers falling over
7. What slows down developers? Other developers Design / Art Production (Copy) Lack of knowledge
8. Increasing Developer Efficiency Flash (Client) PHP (Server) All developers know both Design DOESN’T rule all Developers co-own features with designers
9. Data Driven Design Shoot for a content pipeline that doesn’t need a developer Data driven can be as easy as a designer-editable XML file No devneeded!
10. String Tables A string table is an external file that holds strings for the app Best practice to prepare for localization Developers are not blocked by production Allows quick response to Facebook TOS changes
11. Abstracted Network Layer Feature developers get client/server communication and serialization for free Goal
12. Abstracted Network Layer PHP Flash Feature A Feature B Feature A Service Feature B Service Networked Actions Action Dispatcher Network Layer Network Layer Validation Checks Validation Checks AMF Protocol (RPC) Action Batching Result Batching
13. Network Layer Batching No need to do the work twice Reduced server load in FarmVille by 50%! Server Client Only One Request onPlow() Plow Plow onPlow() Plant Plant onPlant() Plow …
14. Network Layer Validation Solves problem of unfiltered input Ensures data is received by server in-order it was sent
15. Social Network Wrapper Single place to perform Facebook API calls Working with a highly dynamic API can be difficult Abstracting FB calls makes them easier to integrate Allows for cross-platform games
16. Continuous Deployment Build latest version of source-repo Deploy to test “auto app” on Facebook to surface production issues early
19. Scalable Server Architecture We had no choice but to scale on the cloud Every part of the server architecture scales horizontally No single points of failure Take the DB out of the equation
20. Round Robin DNS … Load Balancer #1 Load Balancer #N Auto-scaling web array (PHP) … Web Server #M Web Server #1 User Data & Updates Memcache Pool Lazy Writes DB Layer Data Reads … DB-#K Master DB-#1 Master … DB-#K Slave DB-#1 Slave
21. All you need to implement is… … Web Server #M Web Server #1 …everything else is off-the-shelf components! …
22. Reducing Load-times Users want responsive pages Show something immediately Player’s won’t wait for your app to load Stream non-critical content
23. Avoid Remote Calls Remote server calls are slow and unreliable Aim for no remote calls during load Embed data into Javascript Facebook Iframe Inline-JS here Flash ClientEmbed
24. Caching Slow Calls Build in FacebookAPI caching within the social network wrapper Write a DB wrapper to generate and cache SQL
26. Fault Tolerance Servers do go down randomly Build redundancy on all levels of server architecture Facebook is a dependency also Aim for DB-less & Facebook-less modes “Defcon”-style error management
27. Runtime Config How do we progressively keep the game running? Create all features with kill-switches Create a web dashboard to allow non-technical folks to help out
28. Is it still running? Notify your team when things break using server monitoring (Nagios/Munin) What to watch Server load/traffic graphs Memcacheevictions
29. Stats Tracking How do we know what users are seeing? Have the client send back statistics Number of errors Load-times Take a metrics driven approach to error handling
30. Before Launching Get confidence that your stuff actually works Perform load testing Social games are a marathon not a sprint Sleep before launching!