An online game server's functional and non-functional features lead to non-standard challenges for both architecting as well as testing. This talk starts with an overview and then discusses one testing scenario in depth. We stress particularly on testing the asychronous nature of the application's method calls. A few general approaches of testing such applications terms are alluded to at the end.
2. Massively Multi-Player Online Game
Nirmalya Sengupta, Independent Consultant
Srinivas Chillara, Principal Consultant at CeeZone
3. Background
Dutch company – Online lottery giant,
decided to enter Online Poker market
Scope: Building a complete offering of
online betting Poker game environment
(first version in 12 months)
Technology choices: Java, C++, GWT,Test
Scripting, Linux, MySQL, RabbitMQ,
RedDwarf
4. Application: a bird’s eye view
Player Portal
Database
Game Server Tournament
Server
RabbitMQ
Lobby Server
7. Poker Lobby – showing all table types, tournament types etc
8. Servers must be ...
Robust
Network ready
Support for ‘Statefulness’
Easy to interface with external systems
Highly threaded
Minimum supportable concurrent player base: ~20000
Logged in players don't always play but add load
Speed and Order of message exchange between Client
and Server are of essence
9. Crucial Challenges
Randomness must be satisfactorily built in and proven
(legal requirement)
Arbitrarily fired Timers affect software's behaviour
Player may log in just to watch others
Network connection with player can drop anytime: game
must continue
Players may decide to sit out and rejoin anytime
Multiple simultaneous tables, Multiple simultaneous
tournaments
Game History must be retained (legal reqmt.)
10. Tricky to test software which..
is inherently asynchronous
is supposed to support many users, while
maintaining performance
is asking testers to be good/bad Poker players
is affected indeterminately by Timers
(arbitrariness)
requires preservation of history of every move
is expected to display minimum latency always
11. Example of asynchronous behaviour
NEW HAND!
P1
P4 P2
ASYNCHRONOUS
MESSAGE!
step3. Hand over! step1. P2 plays
Now P3 is
P3 informed of
new Hand
allowed to re-
step2. P3 sits-out / join
gets disconnected
12. Some of bugs detected
Disconnected player not able to login
Negative balance for a player
If the player's connection to the Game Server is lost
while playing his table balance is not returned to his
real balance.
The raise rules are not correctly applied for micro-stake
tables
The tournament ended abruptly and winners are
declared
Application crash when player tries to join a table
Server running out of memory
13. Steps taken to increase testability
Separation of rules of game/business from infrastructure
– Model using a Finite State Machine
– Write separate State Transition TestDriver
Impregnate codebase with EventWatchers
– Continuously emit info about the goings on
– JMX bean for specific data elements
– Specialized sniffer Tasks (abstraction of threads)
Carefully structured Log messages
– Programmatically consume log message and correlate
happenings in the applications
– Simple utilities to filter logs
14. Testing infrastructure
Human Players
BOTs Player Portal
. Database
.
.
.
.
.
. `
Game Server Tournament
.
. Server
.
RabbitMQ
JMX Console
Lobby Server
15. Steps taken to increase testability
Create controlled load on the Server
– Java-based Robots written which could play Poker
– Count, time and intelligence of the Robots,
configurable
– Homegrown binary protocol carried Roundtrip time
data
Deal with Randomness
– Handcrafted card-sets were dealt to players
(repetition)
16. Experience reminds us that
Testing cannot be an afterthought for planning or design
Collaboration between testers, developers, analysts and
architect is crucial
Extra measures taken initially, make many very difficult
problems easy to handle later on
Competent testing starts with good thinking which
considers multiple perspectives