This document provides background on a project to build an online poker environment for a Dutch company. Key details include: - The project involved building a complete online poker game environment within 12 months using technologies like Java, C++, and MySQL. - The application included components like a player portal, game server, lobby server, and database connected via RabbitMQ. - Major challenges included the asynchronous nature of the game, random elements, maintaining game history, and supporting many concurrent players. - The initial architecture had some problems and underwent changes over time to improve performance and integration of components like the lobby server.

1. Saturday, 27th July, 2013

2. Introduction and Background
 Sajit Vasudevan, Technical Architect, Cognizant
Technology Solutions
 Nirmalya Sengupta, Software Craftsman and Coach,
CeeZone Consulting

3.  Is to show that
 Complex projects can be executed by
following Scrum
 Adaptability to emerging architectural
needs is achievable
 Adoption of Scrum mindset helps in
succeeding
 Is to give some concrete examples
Overview

4. Project Background
Dutch company – Online lottery giant,
decided to enter Online Poker market
Scope: Building a complete offering of
online betting Poker game environment
(first version in 12 months)
Technology choices: Java, C++, GWT,Test
Scripting, Linux, MySQL, RabbitMQ, XUnit,
Cruisecontrol, Artifactory, Maven,
RedDwarf

5. Application: a bird’s eye view
Player Portal
Game Server
Lobby Server
Database
RabbitMQ
Tournament
Server

6. Poker Lobby – showing all table types, tournament types etc

7. Poker lobby – Tables list

8. Game table

9. Servers must be ...
 Robust
 Network ready
 Support for ‘Statefulness’
 Easy to interface with external systems
 Highly threaded
 Minimum supportable concurrent player base: ~20000
 Logged in players don't always play but add load
 Speed and Order of message exchange between Client
and Server are of essence

10. Crucial Challenges
 Inherently asynchronous
 Arbitrarily fired Timers affect software's behaviour
 Players may decide to sit out and re-join anytime
 Network connection with player can drop anytime: game
must continue
 Randomness must be satisfactorily built in and proven
(legal requirement)
 Game History must be retained (legal requirement)
 Multiple simultaneous tables, Multiple simultaneous
tournaments

11. Example of asynchronous behaviour
P1
P2P4
P3
step1. P2 plays
step2. P3 sits-out /
gets disconnected
NEW HAND!
step3. Hand over!
ASYNCHRONOUS
MESSAGE!
Now P3 is
informed of
new Hand
allowed to re-
join

12. Tricky to test software which..
is inherently asynchronous
is supposed to support many users, while
maintaining performance
is asking testers to be good/bad Poker players
is affected indeterminately by Timers (arbitrariness)
requires preservation of history of every move
is expected to display minimum latency always

13. `
Infrastructure - revisited
Player Portal
Game Server
Lobby Server
Database
RabbitMQ
Tournament
Server
.
.
.
.
.
.
.
.
...
.
BOTs
Human Players
JMX Console

14. Some initial Architectural
decisions
 Game logic implemented as a handcrafted FSM
 Game Server separate from Tournament Server
 All components inside a Game/Tournament Server
communicate through messages
 { Game | Tournament | Lobby } server exchange
information through Database
 Clients ‘pull’ information from Lobby Server

15. Poker Lobby – showing all table types, tournament types etc

16. Problems popped up: Pig headed
or headless chicken
 Understanding of tournament server was clearer over
time
 Performance bottleneck:
 Identified earlier that lobby would be heavily loaded
 Clients ‘pulling’ information from Lobby Server was a
bottleneck
 Exchanging information through Database was
difficult
 Plugging in 3rd party authentication mechanism was not
straightforward

17. Architectural adaptations led to
 Replacing Game Server with Tournament Server
 We realised that every game played can be considered to be a
part of ‘default’ tournament with no competition
 Game/Tournament and Lobby Servers decoupling
through a Queue
 Lobby Server keeps its own database
 Lobby Server gets latest update from queue
 Previous updates can be forgotten – saves memory and
processing

18. Some architectural changes made..
 Replaced Game Server with Tournament Server
 We realised that every game played can be considered to be a
part of ‘default’ tournament with no competition
 Game/Tournament and Lobby Servers decoupled
through a Queue
 Lobby Server keeps its own database
 Lobby Server gets latest update from queue
 Previous updates can be forgotten – saves memory and
processing

19. Building functionality
 Significant focus on unit testing
 Regular and frequent builds
 By not having adequate automated tests, we paid the
price in
 Incorrectly calculating how to divide money at
stake, amongst players
 Incorrectly recording the HandHistory of games
(both of the above, legal requirements and hence,
extremely important)
 Debugging effort and cost was huge

20. Our Scrum experience
reminds us that
 Collaboration between product management,
developers, testers and architect is crucial
 Changes from the product management could be
accommodated at any point of time
 Architectural alterations can be made to the initial
architecture

21. Brickbats