For Killzone Mercenary and Killzone ShadowFall, Guerrilla Games and SCE Online Technology Group switched from an inflexible, classic 3-tier architecture to one using well understood tools and AWS technologies. This switch allowed them to deliver more title-specific features without losing the benefits of sharing code between titles easily. This session covers problems in the previous architectures, how they fundamentally changed how they built game servers, some of the problems they faced while rebuilding from the ground up, and rabbit holes they got stuck down—both generally, and specific to their use of AWS Elastic Beanstalk, Amazon DynamoDB, and other AWS services. They show how they were able to react quicker to the changing needs of the title, make last-minute feature changes, whip up servers at a moment's notice, and switch out implementations when a new AWS feature came along that would deliver performance or cost improvements.
7. KZ : Liberation
Killzone
(2004)
•
Original real time backend plus …
•
Java 3-tier backend for community
features
•
Shared tech again
•
Customization at presentation layer, plus
configuration
•
Plugins for score processing and
tournament progression in ‘singleton’
•
Leaderboards sat on the DB
Killzone :
Liberation
Killzone
2
Killzone
3
Killzone : Mercenary
Killzone : ShadowFall
(2006)
(2009)
(2011)
(2013)
8. Killzone 2
Killzone
(2004)
•
Applying lessons learned from Killzone :
Liberation
•
Materialized views added to leaderboard
backend
•
Battle Replay using shared file system
(NFS)
•
Plugins in ‘singleton’ servers for
Tournament progression
•
Additional webapp delivering XML for
website
Killzone :
Liberation
Killzone
2
Killzone
3
Killzone : Mercenary
Killzone : ShadowFall
(2006)
(2009)
(2011)
(2013)
9. Killzone 3
Killzone
(2004)
•
Iteration on previous backend
(java and C++)
•
Redone leaderboards from
Materialized Views to MV + TSDB
•
Plugins to middle tier
•
C++ side matchmaking server with
title-specific MM rules plugin
•
And it wasn’t just us feeling the
constraints…
Killzone :
Liberation
Killzone
2
Killzone
3
Killzone : Mercenary
Killzone : ShadowFall
(2006)
(2009)
(2011)
(2013)
10. A Busy Year
September 2013
Killzone
(2004)
Killzone :
Liberation
Killzone
2
Killzone
3
Killzone : Mercenary
Killzone : ShadowFall
(2006)
(2009)
(2011)
(2013)
November 2013
15. Scheduled Jobs
•
Scheduler webapp simple spring-quartz
•
Generates jobs, posts to Amazon SQS
•
Main webapp runs SQS receivers
•
Scheduled job receivers spawn worker threads
per job
•
Wrappers around these jobs provide
–
–
•
Amazon CloudWatch duration/error tracking
Distributed job management
Admin app provides ability to manually
schedule/fire jobs, and view distributed job
state info
16. Score Posting #1
•
Score posts pushed to SQS for async
processing
•
Listeners on component for game
history store triggered on successful
write to Amazon DynamoDB
•
Individual listeners perform many
secondary tasks
•
Failure modes in secondary tasks not
easily recovered
17. Score Posting #2
•
Changed game history listeners to
post to secondary game jobs SQS
queue
•
Added separate receiver threads for
secondary game jobs SQS queue
messages
•
Receivers then run secondary jobs
•
Failure modes in secondary jobs now
result in non-confirmation of SQS
message, and re-presentation by SQS
18. Leaderboards
•
SimpleDB, transitioned to DynamoDB
when available
•
DynamoDB required secondary indices
in some form for multiple sort columns on
a single board
•
Single Table vs. Multiple Table
implementations
•
Fast ranking a problem for both
•
Fast ranking provided by reverse index
rank calculator or statistical predictor
20. Issues
•
Old-style monitoring of dynamically
scaled environments
•
Manual scaling management on
DynamoDB
•
DynamoDB, leaderboards, and hotspots
•
Surprise! Changes to console
•
Limits of AWS CloudFormation
21. Lessons Learned
•
Appreciate the change in mindset going
from SQL to NoSQL
•
Swappable implementations generate
flexibility
•
Flexibility around architecture adds
options
•
Dynamic scaling/management doesn’t
necessarily stop at server boxes
•
Still no silver bullet for leaderboards