This presentation is a quick introduction to software testing and game testing. It should be used as a starting point, and links have been provided for further reading.

1. +
Software Testing and
Game TestingAn Introduction to Software Testing in Games

2. +
Introduction
 This presentation gives an overview introduction to software
testing in general, and how game testing adds additional
complexity to the equation
 This is just an introduction, but there are links at the end of the
presentation to facilitate further learning
 Game QA adds an additional layer on top of this, but that will
not be covered here

3. +
What is Testing?
“Testing is an empirical, technical investigation conducted to
provide stakeholders with information about the quality of the
product or service under test”
Black Box Software Testing: Foundations course [1]

4. +
What is a tester?
 Someone with test competence that performs test activities
Software Developer
Software Developer in
Test
Embedded Tester
Acceptance Tester
System Tester
???
Title does not
matter. If you have
test competence
and you perform
test activities you
are (but you may
have many roles)
a tester

5. +
Quality?
“Quality is value to some person” [12]
Quality means different things to different
people. But in the end, when someone says
they want high quality – what they want is
something that is valuable to them

6. +
Different Layers of Tests [13]
 Unit Test
 Integration Test
 Component Interface Test
 System Test
 Acceptance Test
Which layers you choose to
separate different types of tests
does not really matter – but it
can be good to have some kind
of separation of different types
of tests to facilitate
understanding and separation
of work

7. +
The Boxes of Testing [13]
Black Box Gray Box White Box
No insight into the code and
underlying architecture
Some insight into the code
and underlying architecture
Full understanding of the code
and the software architecture

8. +
Test Techniques [21]
 People-based techniques (User, Alpha, Beta, Pair, Bug Bash, Expert)
 Who does the testing?
 Coverage-based techniques (Boundary testing, Equivalence class analysis)
 What gets tested?
 Problem-based techniques (Input-, Output-, Data- and Computation constraints)
 Why are you testing?
 Activity-based techniques (Scripted, Exploratory, Smoke, Regression)
 How are you testing?
 Evaluation-based techniques (Comparison with specification, Comparison with saved data)
 How do you know if a test is passed or failed?

9. +
Risk-based Testing [14]
1. Make a prioritized list of risks.
2. Perform testing that explores each risk.
3. As risks evaporate and new ones emerge, adjust your test
effort to stay focused on the current crop.

10. +
Risk Input
 There are many things that could have impact on perceived risk
level
 Recent fixes
 Code coverage in changed components
 Code complexity in changed components
 Historical data
 Known dependencies for changed components
 Known bugs in changed components
 Etc.

11. +
Exploratory Testing
"a style of software testing that emphasizes the personal
freedom and responsibility of the individual tester to continually
optimize the quality of his/her work by treating test-related
learning, test design, test execution, and test result interpretation
as mutually supportive activities that run in parallel throughout the
project.”
Cem Kaner With few exceptions designing manual test cases
upfront is not an efficient way of testing, but
exploratory testing requires a more mature test
process and organization

12. +
Agile Testing Quadrants [22]

13. +
Test Automation
Understand the
Software System
Plan your Test Design
Create manual tests
Create automated tests
Ensure Software System
Testability
Designing effective and efficient
automated tests is a science in itself –
never underestimate the time and
cost of setting up a large scale,
continuous integration system
It is time, and not cost, that you will
reduce with automation
Never try to automate tests that are
better performed manually

14. +
Testing Checklist: ISO 25010 [11]
Functional suitability Reliability Operability
Performance efficiency Compatibility Maintainability
Security
Transferability
It can be good to have a checklist of different categories of test that
you need to cover. This ISO standard is one example, but there are
many others [20].

15. +
Combined Engineering [23]

16. +
Test-driven Development [18]
 Test Driven Development is a software development process,
not a test process
 “First the developer writes an (initially failing) automated test
case that defines a desired improvement or new function, then
produces the minimum amount of code to pass that test, and
finally refactors the new code to acceptable standards.”

17. +
Crowdsourced Testing [19]
 “Crowdsourced testing is an emerging trend in software testing
which exploits the benefits, effectiveness, and efficiency of
crowdsourcing and the cloud platform. It differs from traditional
testing methods in that the testing is carried out by a number of
different testers from different places, and not by hired
consultants and professionals. The software is put to test under
diverse realistic platforms which makes it more reliable, cost-
effective, fast, and bug-free.”

18. +
Testing: There is so much to learn
Black Box Software
Testing Foundation [1]
Test Automation [2]
Context-Driven Testing
[3]
Testing at Google &
Microsoft [4][5]
ISO29119[6] Testing & Checking [7]

19. +
Software Testing vs. Game Testing
General Software
Testing
Game Testing

20. +
What makes Game Testing Unique
Fun Factor Testing[8] Balance Testing[8]
Game Level/World
Testing[8]
AI Testing[8][10]
Multiplayer/Network
Testing[8]
Audio Testing[8]
Physics Testing[8] Realism Testing[9] Modification API Testing

21. +
There are many similarities
 Software is software whether it is a game or not
 Software testing is an engineering discipline whether it is games,
applications or other types of software
 Functionality, performance, stability, compatibility, localization, etc.
are mostly the same, but with different focus depending on context
 A game tester should have the same general knowledge base as
any software tester, but with a special focus on what makes
games unique

22. +
Fun Factor Testing
 User experience and usability are valid testing areas for all
types of software
 What differs games from other types of software is that it has to
be fun – it is entertainment and it needs to have a fun factor
that other types of software may not require
 To be able to work with Fun Factor Testing requires a large
domain knowledge of games and what makes them fun

23. +
Balance Testing
 Balancing different options is something that is much more
prevalent in games compared to other software
 Balancing different character options such as race, class, and
attribute choices
 Balancing weapons, boosters, equipment, and other paraphernalia
 Balancing levels
 Balancing difficulty

24. +
Game Level/World Testing
 Somewhat unique to games is the number of instances in which assets
are used
 An application may use a feature in one place, while a game might use it in 20
different levels or parts of the world
 This means that you need to test that the asset is used correctly in all these
instances
 Especially 3D worlds require a unique approach
 Stuck/Sticky spots
 Invisible walls
 Map holes
 Missing geometry

25. +
AI Testing
 Facing a computer controlled opponent is common in games,
but less common in other types of software
 Testing the different attributes and of the AI, and how it reacts in
different situations is critical in many games
 How well it mimics human behavior
 Survival instinct – such as looking for cover in a fire fight
 Hunting state – not waiting to react to the player’s actions, but
proactively performing actions
 Infighting – how it reacts to other AI
 Pathfinding

26. +
Multiplayer / Network Testing
 Playing against multiple other opponents is quite common in games, and
even though other software also communicates with other users and
servers through different mediums, it is often to a lesser extent, and less
sensitive
 Many things can go wrong in multiplayer
 Failed connections
 Dropped connections
 Lag
 Invisible players
 Scoring errors
 Unaccepted invitations

27. +
Audio Testing
 Obviously almost all software has some kind of audio
 However in games it is often more detrimental to the user experience if something
goes wrong
 Often much more complex in games, with many sounds playing simultaneously
 Many things can go wrong
 Audio drop
 Skipping
 Distortion
 Missing sound effects
 Volume level

28. +
Physics Testing
 Some games have physics engines, which affect both
gameplay and animations
 It requires a specific skill to spot physics bugs
 Breakable geometry must be tested to assure that it is
destroyed in a way consistent with the desirable physics
 Dynamic behavior such as boxes moving when you walk into
them is another example of physics in action

29. +
Realism Testing
 This type of testing is also related to how the game managed to mimic the
real world in a desirable way
 Is the car handling like a real car?
 Does the weapon feel like a real weapon?
 Running animations and jumping must have the right look and feel
 This type of testing requires a lot of domain knowledge – to know if an
airplane is realistic enough, you must know about airplanes

30. +
Modification API Testing
 If the game allows for user to create their own mods, and it is a
competitive game like an mmo this requires a unique approach
 Open APIs are common in software, but competitive games
add a dimension of not allowing mod users to gain unfair
advantages or being able to exploit the game using their mods
 Imagining how the APIs will be used is critical to understanding
if there will be future problems or not

31. +
Player Types [9]
 You can categorize gamers in a slightly different way than
users of other applications and technology in general
Killer Achiever Explorer Socializer
Casual Gamer
Hardcore Gamer
Button Masher
Customizer
Exploiter

32. +
Prioritization based on Player Types
 How you prioritize your tests could be influenced by which
player types you are aiming the game for
 If you want to please all types, then making sure you have run
sufficient tests in each category will go a long way when it
comes to reducing critical bugs

33. +
Combinatorial Testing [9]
 Of course combinatorial testing is nothing unique to games, but
when looking at a large, sprawling 3D game world, one can
imagine the size of the combinatorial explosion
 A larger game world, a large amount of actors, and a large
amount of actions for these actors to perform, all results in a
combinatorial nightmare

34. +
Conclusion
 Software testing in general, and game testing in particular, are
extremely complex activities
 It takes a lot of time and effort
 It requires a lot of experience and knowledge
 It needs to be handled very delicately to not produce waste
 It is necessary if you want to release a game that is valuable to
some person(s)

35. +
References
[1] BBST
http://www.testingeducation.org/BBST/
[2] The A Word
https://leanpub.com/TheAWord
[3] Context-Driven Testing
http://context-driven-testing.com/
[4] How Google Tests Software
http://www.amazon.com/Google-Tests-Software-James-Whittaker/dp/0321803027
[5] How We Test Software at Microsoft
http://www.amazon.com/How-We-Test-Software-Microsoft/dp/0735624259/
[6] ISO29119
http://www.softwaretestingstandard.org/
[7] Testing & Checking
http://www.satisfice.com/blog/archives/856
[8] Game Development Essentials: Game QA & Testing
http://www.amazon.com/Game-Development-Essentials-QA-Testing/dp/1435439473
[9] Game Testing: All on One
http://www.amazon.com/Game-Testing-Second-Charles-Schultz/dp/1936420163/
[10] Artificial Intelligence (Video Games)
http://en.wikipedia.org/wiki/Artificial_intelligence_(video_games)
[11] ISO 25010
http://www.iso.org/iso/catalogue_detail.htm?csnumber=35733
[12] Gerald Weinberg
http://secretsofconsulting.blogspot.se/
[13] Software Testing
http://en.wikipedia.org/wiki/Software_testing
[14] Heuristic Risk-based Testing
http://www.satisfice.com/articles/hrbt.pdf
[15] Test-driven Development
http://en.wikipedia.org/wiki/Test-driven_development
[19] Crowdsourced Testing
http://en.wikipedia.org/wiki/Crowdsourced_testing
[20] Test Heuristics Cheat Sheet
http://testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf
[21] Lessons Learned in Software Testing
http://www.testingeducation.org/BBST/testdesign/KanerBachPettichord_Lessons_Learned_in_SW_testingCh3-1.pdf
[22] Agile Testing Quadrants
http://www.developsense.com/presentations/2014-06-Dublin-RSTAgileTesting.pdf
[23] To Combine … or not
http://angryweasel.com/blog/to-combine-or-not/