IGDA Malta, June 2012

1. Five Clichés
Of Online Game Development
That We Wish Weren’t True
(But Will Probably Ruin Your Life Some Day)
Ian Dundore – TRC Family Entertainment Ltd

2. Who is he?
THIS GUY STANDING UP

3. True facts.
• Started making games as a pre-teen.
• Game journalist, 1999 - 2006
• B.Sc. Computer Science, 2004
• First game programming job in 2006
– Gods & Heroes: Rome Rising
• CCP Games, 2008 – 2012
– Several EVE Online expansions
– Dust 514

4. This slide is an excuse for me to make jokes in grey text that I will not
read out loud. For those of you who actually read this far along a
slide, bravo. I salute you, literati.
THIS IS THE SLIDE WHERE I TALK
ABOUT WHAT I’M GOING TO SAY.

5. That man has a chart. Look at him. He’s probably never worked a day in
his life. He’s not even wearing a suit. By the way, don’t Google Image
Search for “Getting Down to Business” with SafeSearch off.
LET’S GET DOWN TO BUSINESS.

6. Cliché the first.
THE CLIENT IS IN THE HANDS
OF THE ENEMY

7. Ah, Client, my old foe…
“Never trust the client. Never put
anything on the client. The client is
in the hands of the enemy.
Never, ever forget this.”
– Apocryphal, often misattributed to Raph Koster.

8. The numerology is strong with these slides.
TWO MEANINGS

9. #1: Input
• Data from the client is raw input, period.
• Validate or escape everything you receive.
– Should be familiar to Web devs.
• Thought Exercise: What if the user could call
my function/class/code snippet?

10. Corollary: Don’t Melt Your Server
• Light/vital systems: Double-check everything.
– Speech, wallet
• Heavy/fuzzy systems: Calculate on
client, verify possibility on server.
– Physics, movement.
• How much error can you tolerate?

11. Case Study: Speedhacks
• Movement based on client-supplied position
updates.
• Server verifies for maximum possible speed.
• How to make a speedhack: figure out
maximum range, scale movement vector.
– This is how real speedhacks work: WoW, EQ, etc.

12. #2: Output
• Data to the client = data to the player.
• Anything you send to the client, the user will
see.
• Anything measurable via the client will be
decoded: game systems, etc.

13. Case Study: BACON
• EVE Online comes with debug output viewer.
• Identifying info logged each time a player
entered your vicinity.
• This was not displayed to the player in-game.
• Result: External program plays audible alerts
when enemy players enter the area.
– Logs introduced in 2002, BACON released 2008.

14. Addendum
• Anything in the client, the player will see.

16. Nuance
• Logical extreme: Gaikai.
• Compare vulnerabilities of trust to advantages
in user experience.
– Offloading heavy work to the client = good!
– Lots of bling = happy players!

17. Cliché the second.
PREMATURE OPTIMIZATION IS THE
ROOT OF ALL EVIL

18. The Quote
“We should forget about small
efficiencies, say about 97% of
the time:
Premature optimization is
the root of all evil.”
- Donald Knuth, super genius

19. No “Two Meanings” Slide This Time
• The proper order:
– Find the fun
– Make it good
– Make it fast
• However, don’t cut corners for the sake of it.
– Avoid the most obvious blunders.
– Test, test, test.

20. The Case for Quick
• EVE - Planetary Interaction
• 4 month development cycle
– 10 weeks of “real” development
• Fluctuating requirements
• Major new features injected halfway through

21. The Case for Quick?
• Heavy overtime
• Shipped first-revision architecture
– ~10 major bugs discovered after release
– 1 item duplication bug discovered
– Two hotfixes
– Memory leaks relied on daily server reboots

22. The Case for Quick!
• Concurrency goals exceeded
– 30,000+ concurrent users after launch
– 25% peak CPU usage or less
– Memory not an issue, EVE already rebooted daily
• Post-launch rewrite: 2 + 1 weeks
– Eliminated memory issues
• 50% memory usage reduction by using Python Slots
– Eliminated duplication & high-priority issues
– ~4 new bugs filed after 2 years in the wild

24. Case Study: The Five Bug
• Gods & Heroes – in production for > 4 years
• Fully home-grown, pure C++ engine
• Largest scale test: 30-50 users, ~2-3 hours uptime
• Target: ~5000 users, 72 hours uptime.
• No automated test tools

25. The Law of Five
• Server occasionally crashed
– Corrupted stacktrace, clear memory corruption
• Usually the value 0x5, hence the name
– Random code module
• But usually combat or special abilities
– Cause not clearly evident, debugger useless
– Deprioritized until beta

26. Beware the Fives of March
Highest beta concurrency: ~1000 users, 30
minutes uptime
Average concurrency: 300-500 users, 10-15
minutes uptime

27. The Fives Have It
• ~6 weeks spent debugging.
• Deep bug in 5-year-old inter-module
communication
• Very rare in the wild…
– As users rise, “very rare” approaches “certain”
• Bug fixed October 8, 2007
• Company closed October 9, 2007

28. Learn From These Mistakes
Too Big to Fail Rewriteable
• Networking code • Low-level code
• Scene layout • Individual box features
• Art style • User interface
• Genre • Lore & character details
Technical & creative direction  Stuff built upon that stuff

29. Cliché the third.
THERE ARE TWO TYPES OF
UNKNOWNS.

30. Obligatory quote slide.
“There are known unknowns; …
things that we know we don’t
know.
But there are also unknown
unknowns … things that we don’t
know we don’t know.”
- Donald Rumsfeld, defense guru

31. The “engineer’s rule of thumb” holds
• Any given project will have work injected
– Technical requirements, design
changes, optimization, iteration, platform
upgrades…
• Account for this when planning
• Planning and tracking tools are invaluable
– But you will hate them every step of the way

32. How It Works
• Come up with some estimation benchmark
– Homework: Look up “Complexity Points”
• Estimate ALL THE THINGS
• Keep track of what you finish in a constant time
period (2 weeks, 1 month, etc.)
– Use these to calculate Ultra Nerdy Stats
• Averages, medians, standard deviations…
• BAM. You have a rough estimate of how long
your project will take.

33. More Importantly…
• Keep track of everything you add.
– Note when you added it.
• Everything you add must be estimated too.
– Use the same metric as before.
• BAM. Now you have a guideline of how much
unknown work to expect.

34. Examples
• Newly-formed team, new feature, established
tech framework & art style
– ~50% of work completed was injected during
development
• Gelled team, iteration on existing feature
– ~25% of work completed was injected
• Your numbers will vary!

35. Cliché the fourth.
ANY SUFFICIENTLY CREATIVE TOOL
WILL BE USED TO DRAW A…

36. Problem Users
• 1% of your playerbase will generate 90% of
your support load.
• Good logging, data retention plans are key
• Log everything involving money, real or
otherwise

37. Case Study: Zero-Day Exploit
• Item duplication exploit due to subtle bug in a
game feature’s code.
• Not readily apparent without hours/days of
observation.
• Easily disrupted through normal play.
• If manipulated, would generate perpetual
stream of items for free.
– Unattended!

38. Incidence
• 136 different abusers in prior 6 months
• ~200 bugged item generators
• Circa $30,000 worth of in-game currency
• 120 abusers were short-time offenders
– Likely unnoticed, small-scale
• 3 abusers generated over 90% of exploit-
driven in-game currency

39. The Value of Logs
• Excellent logs allowed us to:
– Pinpoint start of item duplication
– Trace duplicated items through “fence” accounts
– Measure likely scale of duping operations over
time
– Ban them all!

40. Creativity vs. Safety
• Any sufficiently advanced tool…
– FPS sprays, Minecraft…
• Carefully weigh support cost vs. user fun
– Family image? Intolerant audience?
• Have good support tools in place
– Habbo blockade 

41. Cliché the fifth isn’t really a cliché at all.
NEVER WRITE YOUR PRESENTATION’S
TITLE BEFORE YOUR PRESENTATION

42. Questions?