Quality Assurance 1: Why Quality Matters

Marc Miquel
Marc MiquelPhD User Experience & Lecturer em Universitat Pompeu Fabra
Practical Lesson 1:
The testing process
What is a bug?
Quality is the absence of bugs.
Bug is a behavior not in someone else’s expectations.
Different Sorts of Bugs
• Quality is no bugs, but, what is a bug or defect?
A bug is caused by a mistake, it may imply an error (in values), a lack of accordance to
an specification/standard, and (it may) cause a system failure.
1. By classifying bugs into categories, we can understand their importance and learn
how to spot them.
2. By classifying bugs into types, we can understand where they could be originated.
3. By classifying bugs into severity levels, we can understand their priority in the
development process.
Make sure to investigate invisible walls very closely to avoid being NAB’d (when a
developer re-classifies your “bug” as “Not a Bug”). By the way, this also implies you
don’t know how to do your job!)
Per Olin classified bugs according to categories (Levy et al. 2009, Game Development
Essentials: Game QA & Testing; p. 77).
Visual: missing texture.
[https://steamcommunity.com/sharedfiles/filedetails/?id=135782335]
Physics: unrealistic gravity.
[http://www.smosh.com/smosh-pit/photos/10-funny-video-game-glitches-gifs]
Visual: clipping.
[http://www.smosh.com/smosh-pit/photos/10-funny-video-game-glitches-gifs]
Orthogonal Defect Classification (ODC) system developed by IBM allows to classify
defects into types in order to reveal how they were introduced, be found or avoided
(Schultz, 2011, Game Testing All in One; p. 42).
• A Function error is one that affects a game capability or how the user experiences
the game. The code providing this function is missing or incorrect in some or all
instances where it is required.
• A Assignment type error is the result of incorrectly setting or initializing a value used
by the program or when a required value assignment is missing.
• A Checking defect type occurs when the code fails to properly validate data before it
is used. For instance, the quantity of ‘ammo’ in a shooter.
• Timing defects have to do with the management of shared and real-time resources.
• Build/package/merge or, simply Build defects are the result of mistakes in using the
game source code library system, managing changes to game files, or identifying and
controlling which versions get built.
• Algorithm defects include efficiency or correctness problems that result from some
calculation or decision process.
• Documentation defects occur in the fixed data assets that go into the game. This
includes text, audio, and graphics file content.
• Interface defects occurs where information is being transferred or exchanged.
Bug severity levels (Per Olin)
It can also be tagged with a number on the range 1 to 4.
Normal bugs vs. Tough bugs (Per Olin)
Advices to deal with a ‘tough bug’:
• Once a tough bug appears, take a screenshot with the dev menu, debug data, etc.
• Immediately talk to the lead tester.
• Tell everybody to reproduce this (the exact steps).
The lifecycle of a Build
A testing process aims at finding bugs at every build (or version) of the game.
A basic game testing process in a QA/QC dept. consists of the following steps:
1. Plan and design the test.
2. Prepare for testing.
3. Perform the test.
4. Report the results.
5. Repair the bug.
6. Return to step 1 and re-test.
The lead tester, primary tester, or any other tester tasked with test creation should
draft these documents prior to the distribution of the build.
They tell the tester “go for X build version and Y devices and take test suite Z”.
Sometimes when a test suite is generalist a test case “does not apply”.
This is a test suite for minesweeper
Lead Tester receives the build…
Builds submitted to testing that don’t meet the basic entry criteria are likely to waste
the time of both testers and programmers.
The countdown to testing should stop until the test “launch” criteria are sufficiently
met. These are the criteria:
• The game code should be built without compiler errors. Any new compiler
warnings that occur are analyzed and discussed with the test team.
• The code release notes should be complete and provide the detail that testers
need to plan which tests to run or re-run for this build.
• Defect records for any bugs closed in the new release should be updated so they
can be used by testers to make decisions about how much to test in the new build.
• Tests and builds should be properly version controlled, as described in the
following sidebar (version control is for everyone, developers and testers).
• When you are sufficiently close to the end of the project, you also want to receive
the game on the media that it will ship on. Check that the media provided
contains all of the files that would be provided to your customer.
Configuration Preparation (config prep)
Before the test team can work with the new build, some housekeeping is in order.
The test equipment must be readied for a new round of testing.
• Wiping the “old” build in the computer. You need a clean computer. Save the old
files in a server or common space.
• The latest drivers for all components of the computer. This not only includes the
obvious video card and sound card drivers, but also chipset drivers, motherboard
drivers, ethernet card drivers, and so on.
• The latest versions of any “helper apps” or middleware the game requires to run.
These can range from Microsoft’s DirectX multimedia libraries to multiplayer
matchmaking software such as GameSpy Arcade.
The only other software on the computer should be the new build.
Ready, tester? Wait a second.
The next step after accepting a new build and preparing to test it is to certify
that the build is worthwhile to formally test.
• Smoke Testing. At a minimum, it should consist of a “load & launch,” that is,
the lead or primary tester should launch the game, enter each module from
the main menu, and spend a minute or two playing each module.
• So the build is distributed. Time to test for new bugs, right? Not just yet.
Before testing can take a step forward, you must take a step backward and
verify that the bugs the development team claims to have fixed in this build
are indeed fixed. They are in the “knockdown list”. This process is known as
Regression testing.
Sometimes critical bugs are not retired from the regression testing and the
regression testing may last for the entire development.
Reporting a bug
Possible audience of a bug: lead tester, project manager, marketing, third
parties, customer service,… obviously other testers.
In fact, the primary tester should be allowed to modify all fields in the bug
database except for Priority, Status, Assigned To, and Developer Comments.
How to write good reporting?
• Only facts. “The guard’s hat should be blue” or “the AI is weak”.
• Brief Description or title. The first sentence should contain a brief and clear
description of the bug:
“Crash to desktop when choosing Options from Main Menu”. YES
“Game crashed after I killed all the guards and doubled back through the level to
get all the pick-ups and killed the first re-spawned guard.” NO. too much detail.
“Game crashed after guards respawned”. YES
Advice: write the full description first and then the brief description (title).
• Full description. This provides all the necessary details. Rather than a prose
discussion of the defect, the full description should be written as a series of
brief instructions so that anyone can follow the steps and reproduce the bug.
The steps should be written in second person imperative, as though you were
telling someone what to do. The last step is a sentence (or two) describing the bad
result.
1. Launch game.
2. Choose Multiplayer.
3. Choose Skirmish.
4. Choose “Sorrowful Shoals” map.
5. Choose two players.
6. Start game.
Or much better:
1. Start a two-player skirmish game on “Sorrowful Shoals.”
EXAMPLE:
1. Create a multiplayer game.
2. Click Game Settings.
3. Using the mouse, click any map on the list. Remember the map you clicked on.
4. Press up or down directional keys on your keyboard.
5. Notice the highlight changes. Highlight any other map.
6. Click Back.
7. Click Start Game.
Expected Result: Game loads map you chose with the keyboard and mouse.
Actual Result: Game loads map you chose with the mouse.
Advice 2: Consider introducing the expected results as it may not be obvious for the
programmer.
It is also a good reminder of how test cases are produced.
Programmers should be educated in the testing process.
Other things we MUST include:
• Components versions. Build version, specific drivers versions, etcetera.
• Bug priority. The tester sets the possible priority (1-4 or minor, major to critical).
• Bug category. The tester should use the category “audio”, “artificial intelligence”,
“physics”, etc, so it narrows the scope
• Bug possible origin. The tester can include it in a line near the beginning, so it may
help the programmer in understanding the bug. The tester may say it comes from
“function”, “assignment”, wrong “build”, etc.
Things to avoid:
• Humor. A In a high-stress environment it may fuel the tempers.
• Jargon! Write in a clear language. Testers should avoid very technical terms
(source of confusion if not used properly).
EXERCISE: check in the Internet for a bug found by a player on a game already
released. Where is it reported? Is it reported properly according to all the fields and
recommendations?
Some Common Mistakes
· Non-summing summary line. "Found a bug." No kidding! Like all these other bugs
in the database aren't bugs that somebody found? That's like sending an email with
the subject line, "Hey" - or "I wrote an email" - or "From me." Sum up the problem!
Say what the problem is. Just like when you write an email, you give the recipient an
idea what the email is about before he or she opens it.
· Too-long summary line. "The slithy toves gyred and gimbled in the wabe when the
borogoves were all mimsy and the mome raths outgrabe." Dude, just say "The slithy
toves gyred and gimbled." That condenses the essence of the problem. You can give
us the details about all the excessively mimsy borogoves in the body of the report.
· Tester as designer. "The slithy toves need to have a 5-second cooldown after gyring
so they don't gimble too soon." No, don't tell the developer how to fix it. Just say
what the problem is. It's the designer's job to figure out what's the best way to
balance the slithy toves.
· Not giving step by step instructions. Tell the developer what to do, so he can see
the problem himself. Don't just say "I did this, then I did that." Tell him, "do this, then
do that." Give step-by-step instructions, in detail, in the second person (imperative).
[http://www.sloperama.com/advice/faq75.htm]
· Unclear basis for an expectation. "Usually, pressing X causes the door to open."
What do you mean, "usually"? Do you mean that's how one opens doors in other
games? Do you mean that's how one opens other doors in this game? Do you mean
doors in your home open when you press an X button? What does "usually" mean??
Be specific, dude!
· Confusing "player" with "player character". The player is the human who's holding
the game controller. That digital being on the screen is a character. Don't use the
terms interchangeably.
· Wishy-washy observation step. "Then watch to see if the problem happens or not."
Wrong. Tell the developer he will see the problem. Tell him to observe that
it does happen.
· Inappropriate use of the word "should" as in: "After you follow these steps, you
should see the bug happen." Um, what? The bug should not happen -- that's why it's
a bug! If it was supposed to happen, then it wouldn't be a bug. So you shouldn't say
"should" in this way. Just say "after following these steps, observe that the bug
happens."
[http://www.sloperama.com/advice/faq75.htm]
True or False:
• A “Verify Fix” status on a reported bug means it will remain in at least one more
test cycle.
• A tester should write as many steps as possible when reporting a bug to ensure
the bug can be reliably re-created.
Lesson 1: Why Quality Matters
Third year course in Quality Assurance and Game Balance
Bachelor Degree in Video Game Design and Production
Third term, April 2019 Dr. Marc Miquel Ribé
Overview of the Lesson
In this lesson we will see the following topics:
What is Quality in Games
1. Quality Matters
2. How Can We Improve Quality
3. Different Sorts of Testing
4. Approaches: Blackbox and Whitebox
5. Being a Black-box Tester.
QA and QM: Team, Phases and Documentation
1. TDD and Test plan
2. Testing phases
3. Quality management measurements
What is Quality in Games?
What is quality?
• Quality is ”how well we do something”
• Quality is “being reliable”
• Quality is “being special”
• Quality is ”being superior”
Quality may mean many things ‘in general’…
Some questions about quality ‘in particular’:
• What is quality in software? Does quality matter in software?
• What is quality in video games? Does quality matter in video games?
• How does user experience relate to video game quality?
• How do features relate to video game quality?
Think about a Space shuttle, think about Microsoft Word, think about a car.
• Is quality the same important for them?
In certain services or products, no quality means risking people’s lives or work.
Quality is “managing expectations”.
In video games, new mechanics and new experiences may matter more than quality…
But still, in a competing market, defects can ruin experiences.
We need to work for quality. Mainly, we need to test quality.
[https://www.gamesradar.com/top-7-horrendously-buggy-games-we-loved-anyway/]
Quality is “managing expectations”.
Quality is about working properly according to someone else’s expectations: game
console manufacturer, customer, etcetera.
Quality is the degree in which some characteristics are adapted to the requirements
put into a product or service (ISO 9000:2000), i.e. (in games: the mechanics, the visuals
requirements, etc.).
Quality is ‘what is required’: functionalities, expectations and the absence of bugs
(anomalies or non-defined behaviors).
“The general aim of testing is to affirm the quality of software systems by
systematically ex- ercising the software in carefully controlled circumstances” (1981).
E. F. Miller. Introduction to Software Testing Technology. Second edition.
Quality is the absence of bugs.
Bug is a behavior not in someone else’s expectations.
In the world of project management, quality, cost and time are key constraints that
shape any project. They are the “Iron triangle”.
Every project is a fight between:
1. what is required (quality)
2. the date for delivery
3. the agreed budget/cost
QUESTION: Which of the following puts pressure on a game project?
a) New funding b) Taking people away to have them work on another game c) A
requirement to add more levels to be comparable to a competitor’s recently
announced title d) b and c.
Why Do We Ship Buggy Games? - A Look Behind the Scenes - Extra Credits
[https://www.youtube.com/watch?v=s1_50T5GwZ8]
@TesterGame_com (QA agency for
Software, Games & Apps. Game
Testing Specialists) says:
“Sólo un 34.8% de las empresas
españolas considera el testing como
un perfil imprescindible en el
desarrollo de un videojuego
#libroblancoDEV #gamedev #indiedev
#indiegame #indiegamedev #qa”
These are results from the survey
“Libro Blanco de los Vídeojuegos”
from 2017.
[https://twitter.com/TesterGame_com/
status/951411246296903682]
1. Quality Matters
ALERT:
§ Quality is a key service aspect. Quality is part of the management, as project
success depends on it. However, while quality is important, it is sometimes
considered “unoriginal” or the “last part”.
It is easy that games have bugs, but quality matters.
• Game software is complex.
• Software tools are used to produce games and these tools are not perfect.
• Games must work on multiple platforms/devices with a variety of configurations.
• Games can be played by a large number of people.
Online reputation depends on critics opinions and quality matters.
Quality must be planned and tested.
§ Coding and testing are different enough to justify the quality department.
As we get more and more professional, we can locate the quality responsibility into
one department.
• Trust is at the base of (video game) software development, and so every
stakeholder (product managers, press, etc.) asks for deliveries and expects each
others’ good performance. But… reality is not as beautiful as expected.
• The quality department does not know if something works until tested. They
cannot trust anyone.
• The quality department is given “no time” and “no resources” but still is asked
quality. Tension.
• How do you balance quality, cost and time?
In other words, how do you balance trust and testing? This is where QA comes in.
§ Testing is the process of verifying that something works properly. Testing is
the most essential part required to ensure quality.
What do they always tell to the tester?
Trust no one.
Less trust implies more testing.
Tester: “Please, Trust No One”
A general form of statements to watch out for is “X happened, so (only/don’t) do Y.”
Here are some examples:
• “Only a few lines of code have changed, so don’t inspect any other lines.”
• “The new audio subsystem works the same as the old one, so you only need to
• run your old tests.”
• “We added foreign language strings for the dialogs, so just check a few of them in
one of the languages and the rest should be okay too.”
• “We only made small changes so don’t worry about testing <insert feature name
here>.”
• “You can just run one or two tests on this and let me know if it works.”
• “We’ve gotta get this out today so just ....”
WRONG! Quality control needs to test whatever considered relevant, sometimes
not listening to the recommendation and constraints coming from other
departments. This is the real meaning of “Trust no one”.
Tester: “Don’t stress: You are in a rational world”.
So, as a tester, you should follow these advice:
• Know your role on the team based on the responsibilities assigned to you.
• Execute your tasks aggressively and accurately.
• Do the most important tests first (priority).
• Do the tests most likely to find defects often.
• Make emotion-free and objective decisions to the best extent possible.
It is not always your fault as a tester if there are still some bugs after the final release
(they might be introduced late, they could be hidden because you spent time on other
bugs, etc.). Remember the ‘iron triangle’.
Conclusion: Testing is not enough to guarantee quality.
3. Different Sorts of Testing
Remember: when they say quality, you may ask: ‘according to what or who?’.
I want you to structure the topic in questions, as it is easier to remember.
• Depending on the “Who” and the “What“ (requirements) with which they define
quality, we may find a different sort of testing.
• Functionality testing, Item testing (game design, producer)
• Compliance testing (platform)
• Localization testing (customer environment)
• Compatibility testing (components configurations)
• Depending on the rest of questions, other testing may appear:
• “How”: Blackbox and Whitebox testing.
• “When”: Smoke testing, regression testing, beta testing.
Let’s leave aside Usability testing and Playtesting, as they are User Experience field:
• Usability testing is aimed at finding aspects which block the ease of using an interface.
• Playtesting is aimed at understanding the user experience and game aspects (e.g. balance)
• Compliance
For ensuring that your game complies with the standards and guidelines of major first
party developers like Apple, Google, Amazon, Microsoft, Sony and Nintendo.
• Guidelines are known as TRC (Technical Requirements Checklist) because of Sony.
While Microsoft calls it TCR (Technical Certification Requirements), and Nintendo
calls it “Lot Check” since 1980 with Super Mario Bros.
Some usual compliance requirements:
• The game must not display any religious images.
• The game should pause once the controllers are not attached.
• Before formatting a memory card or a hard drive, a message warns the user that all the data
will be erased.
• All copyrighted brands or names are accompanied by their respective copyright or trademark
signs.
• If data gets corrupted, the game warns the user and suggests reformatting the memory card
or hard drive.
• System setting for a language.
Guidelines are at all levels: design features, fluency, defects, compatibility, etcetera.
• QA Team from the manufacturer will test it against their TRC again to be make
certain the game complies with platform conventions. The answer is binary: yes/no.
• Some companies publishes job offers with “supervise some TRCs” as some of the
tasks, as it may be interesting to dedicate someone’s job to be familiar with each
platform.
• As said, anything may be included in the TRC: the manufacturer branding is also
dependent on that.
4. Approaches: Blackbox and Whitebox
Depending on the approach, testing is either Blackbox or Whitebox. This is extensible
to software testing.
1. White-box testing – focuses on the architectural, integration and systemic aspects
of mobile game: how third-party components, databases, social media/external
entities, as well as graphics/game engines, audio play and so on are integrated in to
your game.
2. Black-box testing – focuses on the functional and overall playability aspects of the
game. Menus, graphical elements, special effects, animations, and the actual
gameplay are those under test with Black-box approach.
Why does Blackbox exists? Programmers don’t fully test their own games (anymore).
They don’t have time to, and even if they did, it’s not a good idea.
Job segmentation allows programmers to be more professionals and focus on their
task.
Blackbox is the only testing that guarantees “real conditions”.
White-box Testing
White-box testing gives the tester an opportunity to exercise the source code directly
in ways no end user ever could. It takes into account he internal mechanism of a system
or component.
Tests performed by developers prior to submitting new code for integration with the
rest of the game.
Testing code modules that will become part of a reusable library across multiple games
and/or platforms; modules that are parts of a game engine or middleware product;
modules that can be used by third-party developers or “modders”.
• Unit testing: it is the testing of individual hardware or software units or groups of
related Units. Testers (usually developers) verify that the code does what it is
intended; it is usually done within a class or a component.
o Unit testing checks a single assumptions about the behavior of one system.
• Integration testing: it is the testing in which software components, hardware
components, or both are combined and tested to evaluate the interaction between
them. Although it is also done in black-box testing, software developers verify that
units work together and check whether they give some error or not (data might get
lost across an interface, messages might not get passed properly, or interfaces might
not be implemented as specified).
o Integration testing checks multiple systems are correctly connected.
• Automated testing: it is the testing in which certain configurations or tests are run
automatically in order to create situations that are difficult to repeat at large scale
by manual testers. Some of the purposes of doing automated testing are to test
loading times, graphics performance, server loading, etcetera.
QUESTION: Why does White-box testing is more considered QA than QC?
What kind of tests might have been performed to this in order not to detect the flaw?
2 unit tests. 0 integration tests.
Black-box Testing
By their nature, black-box test cases are designed and run by people who do not see
the inner workings of the code. They play as if they were players.
• Test Case: it is a set of inputs, the expected results and the actual results (outputs).
• Test Suite: it is a set of test cases.
• Test Plan: it is a document with the test suites and their test cases.
The minimal ’unit’ of black-box testing is the Test Case
IMPORTANT: Test Case can be defined as an statement ”placement of an object is that”
to be resolved as “passes/fails” or as a description of the test case, expected results
and actual results (and “passes/fails”).
You can create a test case as:
• Description (situation and order) > Expected Results > Actual Results > Pass/Fails.
The Expected Result is the way the game should work according to its design
specification.
The Actual Result is anomalous behavior observed when you used the game, caused by
a software defect.
Or you can create it as:
• Description (order) with the expected results > Pass/Fails > Description.
Test Suite for Monopoly “Getting in Jail”
Requirements according to
some Monopoly rules:
When a user lands on the “Go
to Jail” cell, the player goes
directly to jail, does not pass
go, does not collect $200.
On the next turn, the player
must pay $50 to get out of jail
and does not roll the dice or
advance.
If the player does not have
enough money, he or she is
out of the game.
Test cases and suites will be very different depending on the game genre. In a saga, test
suites can be reused. You can generate a test case for the previous ‘characteristics’ you
already know what to expect from.
Again:
• A test plan defines the overall structure of the testing cycle.
• A test case is one specific question or condition the code is operated and evaluated
against.
• A test suite is a group of test cases.
PROPOSED EXERCISE:
Level Test Checklist. Develop this into a test suite.
Check the placement and behavior of objects in the level
Check the placement and behavior of NPCs
Check the fit and form of each unique tile, mesh, and texture used in the level
Check the function and load times of transitions from one level to another
5. Being a Black-box Tester
• Being a black-box tester is being in a loop. Playing only happens once. In a ‘test
suite’, it is not playing, and once you found the first ‘defect’, you are in the middle of
the loop. The loop is called “Piano TV”.
1. Play to crash it, not to have fun.
2. Identify the bug. It may be not important, blocking, etc. Finding bugs is this part.
3. Amplifying the bug. This means that the tester will narrow the possibilities in which
the bug appears. You may have found a bug in a place, but it may occur more than you
think. Amplifying implies testing again (look for all the places which “relate”).
4. Notifying the team. Once the bug is found, you need to report it using a defect
tracking system, describing the most useful information (category and type of bug) and
extra things you can do to help without being too verbose.
The replicability is the most fundamental part (just like science). It is important to put a
clear title and set the priority according to the severity. It is useful to include server
logs, screenshots, saved files, etcetera.
5. Testify to others. In this moment, the tester sends the report and acknowledges the
different teams (lead tester and developer). Remember: testers get emotionally
attached to their found defect (they are sort of medals in the testing sport).
6. Verify the fix. Once the bug is fixed by the developer, the tester needs to try to
reproduce it. Once it is fixed, the tester may close it.
Depending on your personality, you may be one type of tester or another.
If you are a Judger, you prefer a structured, ordered, and fairly predictable
environment, where you can make decisions and have things settled.
If you are a Perceiver, you prefer to experience as much of the world as possible.
You like to keep your options open and are most comfortable adapting.
Judger
Runs the tests for...
Conventional game playing
Repetitive testing
User manual, script testing
Factual accuracy of game
Step-by-step or checklist-based
May rely too much on test details
Concerned about game contents
Perceiver
Finds a way to...
Unconventional game playing
Testing variety
Realistic experience of game
Open-ended or outline-based testing
May stray from the original test purpose
Concerned about game context
QA and QM:
Team, Phases and
Documentation
Testing is the execution. It is the method and the process by which QC ensures
that all the defects are found.
1. How Can We Improve Quality?
Testing may not be enough. We need to plan for quality. We
need three departments.
These are the quality ‘roles and departments’ in Project
Management:
• Quality Control & Testing – controlling a process to ensure
that outcomes are as expected. Quality control is the lower
layer: it includes testing.
• Quality Assurance – obtaining confidence that a product or
service will be satisfactory, as well as planning for it.
• Quality Management – directing an organization so that it
optimizes its performance through analysis and
improvement.
Why is there quality assurance if there is already quality control and testing?
In order to focus on quality earlier in the process.
If we could test on forever, there would be no need for planning. Planning is putting
priorities and doing things to optimize results.
Defining the testing and planning the testing are part of Quality Assurance.
Quality Assurance developed from the realization that quality could be improved by
looking 'further up the line'. It is aimed at preventing nonconformities/defects.
There are two kinds of defects that justify QA:
1. The one which is defined according to the design requirements.
2. The one which is perceived by the customer and was not even foreseen by the
development team.
The two must be taken into account by Quality Assurance in order to improve the game
and evaluate the quality.
2. Team and roles
Game teams come in different sizes, shapes, locations, and skill sets. They can vary by
company, game title, or platform.
• It is usual to find a leader defined hierarchy: development lead, build lead, etc. So
responsibilities are assigned.
• QA department may have a leg in the development team and in testing team.
The testing team may be dynamic and grow according to the project needs.
• Lead tester (QA lead) is assigned according to a game.
o She defines the “testability” requirements, procedures and standards.
o She represents the test team in planning and meetings.
o She may do some of the game testing (in small teams).
• Testers (QC). They are the group of final testers who execute the test plan.
• Test Engineer: breaks stuff while programmers try to make it work (basically through
White-box testing). She works tightly with programmers.
• Beta testers (optionally). “Volunteer army” in case of doing pre-releases.
Outside the testing team:
• Art Team (animators, artists, etc.), Game Designers (level designers, etc.), Sound
Team (sound engineer, music producer).
• User Experience (responsible for playtesting, surveys, interviews; creating
hypothesis; evaluating the GUI).
• Product Manager / Producer: getting the game done on time and within budget.
Evaluate risks. More and more. The management team and production is QM.
The decision-makers are: Producer (QM), QA Manager and Lead Tester.
• Producer Questions: how do we keep the calendar? How do we fit the budget?
• QA Manager Questions: how big is the game? (platforms, map size, etc.) how much
time do we have? How much money do we have?
• Lead tester Questions: What test suites are most important? How can all testers
work on something important? How many bugs are open and which category?
The main responsibilities of the Lead testers are:
managing the test team, designing and implementing the overall test plan, “owning”
the bug database.
The Lead tester must be very a ‘team’ focused person.
Defining documents is (mainly) for the three decision-makers.
3. Documentation
• Quality does not start at testing but on the planning phase. Good Quality
Assurance and Management requires creating a good set of documents.
• Good planning is saving resources (development, testing). Therefore, it is important
to determine the Scope of testing the project will require.
• Testing better earlier can get problems fixed sooner and cheaper. Earlier means
white-box and testing specific parts. It is about balance.
• Documentation is key to coordinate. The most important documents for the test
manager are the game design document (GDD), the test design document (TDD),
software quality assurance plan (SQAP), and the the test plan document (TPD).
For whom is documentation most important?
• GDD—everyone.
• TDD—tech lead, art director, producer, project manager.
• SQAP—project manager, producer, development lead, test lead, QA lead, and
engineer(s). à This document is Quality Management.
• Test plan document—project manager, producer, development lead, key testers.
• Test suites—feature developer, testers.
(QA & QM)
Technical Design Document (TDD)
The technical design document sets out the way your tech lead plans to transform the
game design from words on a page to software on a machine.
• It describes the core tools that will be used to build the game, and whether they are
already in the house (not to be bought or created).
• It documents the list of software and hardware, and changes in the organization
infrastructure (storage capacity, backup plans, network speed).
• TDD is sometimes part of the GDD (Game Design Document). When both
documents exist, it is recommended that they do not overlap each other in content.
Ø GDD is more focused on story, characters, gameplay, art, UI (what and why),
instead of the “how” to achieve that.
• It is important to keep track of the versions of the document, and each change
should be documented in a log along with its creator (as a table).
• Different companies may have different TDD outline but the same content.
• Its Outline should be something like this:
1. Technical Team: Tech Lead, developers, etcetera.
2. Executive Summary: project overview from a tech. perspective, delivery
platforms, etc.
3. Engine Evaluation (internal solutions, external solutions, middleware)
4. Platform-specific issues (delivery platform #1, #2)
5. Development plan (use cases, game mechanics, main loop, data structures,
data flow, physics, AI, graphics, collision, GUI, fonts, audio/video, especial
requirements)
6. Coding standards (programming standards, style guide, code review
procedures)
7. Scheduling (programming tasks, man-month scheduling, calendar month
scheduling, milestone schedule and deliverables, major event planning).
8. Recruitment (personnel, hiring, risk plan for handling delays in acquiring
resources).
9. Equipment budget and costs (personnel hardware sets, team wide tools,
software tools)
10. Miscellaneous
Game Design, Second Edition. Bob Bates, 2004. p. 209
Software Quality Assurance Plan (SQAP)
An SQAP is strictly concerned with the independent monitoring and correction of
product and process quality issues. It is a QM document.
It should address the following topics:
• QA personnel
• Standards to be used
• Reviews and audits that will be conducted
• QA records and reports that will be generated
• QA problem reporting and correction
• QA tools, techniques, and methods
• QA metrics
• Supplier control
• QA records collection
• QA training required
• QA risks
Test Plan Document (TPD)
A test plan acts as the playbook for the test team. It identifies the test team’s goals
along with the resources (staff, time, tools, and equipment) and methods that are
necessary to achieve them.
It should contain:
1. QA Team (QA Lead, internal testers, external testers, etc.)
2. Testing Procedures (Daily Activities such as Generate a daily build, important
regression tests, etc.; Integration of reports, different kinds of testing – test suites)
3. Testing Requirements Origin (generated in meetings, generated by producer).
4. Bug Tracking Software (package, instructions how-to, etc.)
5. Bug Classification and Bug Tracking (Who classifies, who assigns, what happens
when fixed, what happens when the fix is verified)
6. Scheduling and Loading (rotation plan, when testers are in and out, how many
testers needed in total)
7. Equipment budget and costs (team members’ hardware, software tools needed,
console debug kit, etc.)
The Test Plan Document is as detailed as the Technical Design Document. Again, it
should not overlap each other.
Example: [https://sites.google.com/site/knightsvszombies/test-plan]
4. Testing Phases and Development
No matter what size the game and how long the production schedule, the testing of
the game should always follow the same basic structure: Pre-production, Kickoff, Alpha,
Beta, Gold, Release and Post-release.
The interesting things are “how these phases are defined” and the “test kickoff”.
In each game, a Lead Tester is assigned and its up to him to specify the criteria for each
phase of testing (in an ideal world, in a real one, other issues matter). On large teams,
more than one primary tester may be assigned (e.g. for multiplayer mode, etc.).
Each phase needs: an entry criteria (set of tests that a build must pass before being
able to tell the game is already at that phase) and the target date for it.
Test Kickoff (After Pre-Production)
Test kickoff activities are broken into two parts: tester preparation and the kickoff
meeting, which is conducted according to the kickoff agenda.
For the test kickoff meeting, the tester prepares in the following ways:
• Reads the requirements and/or documentation for the game feature being tested
• Gathers equipment, files, and programs needed for the test
• Reads through the tests
Test kickoff meeting is very important
1. Giving a feature overview
2. Addressing feature questions
3. Bringing up any special test instructions
4. Addressing any test execution questions or issues
• It prepares the teams, it sets ready the equipment, it familiarizes the tester, it
resolves test instruction conflicts, it provides a forum for test improvement.
Alpha Phase Entry Criteria (usual ones)
1. All major game features exist and can be tested.
2. A tester can navigate the game along some path from start to finish.
3. The code passes at least 50% of platform TRC.
4. Basic interface is complete and preliminary documentation is available to QA.
5. First-party controllers work.
6. Online multiplayer can be tested.
7. Minimal Performance (e.g. 30 fps)
Beta Phase Entry Criteria (usual ones)
1. All features are implemented.
2. The game can be navigated in all paths.
3. The game passes the 100% of the TRC.
4. The entire user interface is final.
5. The game logic and AI is final.
6. All controllers work.
7. Final artwork is implemented.
The design lock (or feature lock) happens when this phase is concluded.
Gold Testing
In all gold testing, the game should be in a similar state to this:
1. All known Severity 1 bugs (crashes, hangs, major function failures)
2. Greater than 90% of all known Severity 2 bugs are fixed.
3. Greater than 85% of all known Severity 3 bugs are fixed.
4. Release-level performance (60-fps frame rate)
Upon these criteria, the game is declared to be at “code lock” and the final versions are
release candidate or gold master candidates (GMCs).
The game is sent to the platform manufacturer, which conducts its own testing on the
TRC. They may find “showstopper bugs”, which need to be fix and resubmitted.
Post-release testing. Patches.
5. Six-sigma and Phase Containment (PCE)
• Quality Management and Quality Assurances aim at improving the testing process,
as it is a way of improving the resources efficiency and product quality.
• Game Quality Measurements can help us in determining “how good” is game
software according to the number of defects it has. They are managed in the SQAP.
Two measures are “Sigma Level” and the “Phase Containment”.
1. Six-sigma is a set of techniques and tools for process improvement. It is based on
statistical methods. The “Sigma Level” of code is computed according to the
number of errors per line (in assembly-equivalent lines of code, AELOC).
2. Phase containment is the ability to detect faults in the project phase in which they
were introduced. Phase containment effective (PCE) coefficient is a measure of
how well the development and testing process is being done.
Both are simple calculations which are fast to compute and easy to understand.
(QM & QA)
Sigma Level
Sigma-Level is based on the Gaussian distribution. For instance, 38 errors for 250.000
lines of code is equivalent to a 5.1 Sigma Value.
Released Defects per (AELOC) Sigma
20.000 100.000 250.000 1.000.000 Value
124 621 1552 6.210 4
93 466 1165 4.660 4,1
69 347 867 3.470 4,2
51 256 640 2.560 4,3
37 187 467 1.870 4,4
27 135 337 1.350 4,5
19 96 242 968 4,6
13 68 171 687 4,7
9 48 120 483 4,8
6 33 84 337 4,9
4 23 58 233 5
3 15 39 159 5,1
2 10 27 108 5,2
1 7 18 72 5,3
0 4 12 48 5,4
0 3 8 32 5,5
0 2 5 21 5,6
0 1 3 13 5,7
0 0 2 9 5,8
0 0 1 5 5,9
0 0 0 3,4 6
In this process, the defects being counted must
include both the game defects you know about
that have not been fixed, whatever defects your
customers have already found, and your
projection of defects that remain in the software
which haven’t been discovered yet.
Phase Containment (PCE)
Faults that are found in the phase in which they are introduced are known as in-phase
faults or “errors.” If found later, they are “defects” or bugs.
Calculate PCE by dividing the number of in-phase faults by the sum of faults found in all phases to
come up with the PCE for that phase. When new testing phases are added, the PCE can only be
reduced.
The higher the PCE, the better, because it means that the bugs created at one phase
are also found in the same phase.
QUESTION: What strategies can be useful to have a higher PCE?
- Improve knowledge of the subject matter and provide relevant training.
- Have successful team members provide mentoring to less-successful members.
- Document methods used by successful individuals and deploy them throughout the
team.
- Increase compliance with existing methods and standards.
- Add standards which, by design, help prevent faults.
- Add checking tools that run during the creation process, such as color-coded and
syntax-aware editors.
- Add checking tools that run after the creation process, such as stronger compilers and
memory leak checkers.
6. Testers Effectiveness and Performance
• Quality Management uses data regarding the testing process (the editors’ work) in
order to understand how well it fits the needs and expectations of the overall
project (in terms of duration, new hiring, etc.).
• It is useful to contrast planned and actual testing execution data (about the number
of tests suite performed), globally and at a tester level. This is contained in the SQAP.
(QM)
It is important to take into account the tester “overhead” (trainings, meetings,
preparing for demos, etc.) so, only counting a 75% of the time at best performance, 50
at average performance and so on.
It is useful to count the tests in number of days. This way it is possible to calculate in an
Excel file how many extra testers at part-time or full-time can help in pushing the
project forward.
• It usual that testing teams grow at peak moments of the development process.
Monitoring the team to stay on a short-term commitment help in maintaining long-
term goals on track. However, monitoring is not all good.
Pros:
- Some people will do better if they know they are being “watched”.
- It provides a measurable basis to detect who are the “elite” and avoid favoritism.
- Testers seeking better numbers may interact more with developers to prevent bugs.
Cons:
- Some testers may not be perceived as valuable as they really are (they mentor, etc.).
- It may trigger some jealousy among the team if not channeled properly.
- Testers may argue over who gets the credit for certain defects.
Testing Effectiveness
In a similar way to Phase Containment, testing effectiveness is a measurement which
aims at improving the quality control. In this case, it provides a ratio between the
number of defects according to the number of tests performed in a release.
PCE could be used as a reference.
For instance, if you know a PCE coefficient should be 0.60 and 30 tests are performed,
then, at least 2 defects should be found.
Coders may delay a code release until more defects are found.
Testing Effectiveness
Testers can be also measured according to their effectiveness.
However, you see “what” is happening, but you do not understand “why” is happening
that way. You do not know the ‘severity’ of the bugs found by each tester.
REWIND: Some innocent questions about the test phases
• All bugs MUST be fixed before a game can be certified as GMC. True or false?
• Online multiplayer features can be tested in Alpha. True or false?
• Feature lock should happen in Alpha. True or false?
• The Beta build is the version that will be sent to manufacturing. True or false?
• Describe whether each of the following is an appropriate topic to discuss during a
test execution kick-off, and why:
a) Possible contradictions in the feature requirements
b) Ideas for new tests
c) Company stock prices
d) Identical tests already being run in other test suites
e) How “buggy” this feature was in the previous release
f) Recent changes to the game data file formats
g) Lack of detail in the test case documentation.
Summary
• Quality is ‘what is required’: functionalities, expectations and the absence of
bugs (anomalies or non-defined behaviors).
• Quality Management, Quality Assurance and Quality Control are the three
areas which deal with quality in the game development process.
• Testing is the execution of quality control with the aim of finding defects or
bugs. They can be described according to their external characteristics, possible
cause and severity.
• Quality is not coding better or solving bugs, it is detecting what is not working
properly or in best case scenario planning the method to detect. Solving bugs is
development.
• White-box and Black-box testing are two approaches to bug detection usually in
different phases. One is from ”the inside”, as a coder, and the other is “from the
outside”, as a player.
• Black-box testers cannot trust anyone, and need to plan test suites and perform
a rigorous and documented process in order to find bugs, report them and
verify them.
1 de 79

Recomendados

Testing concepts [3] - Software Testing Techniques (CIS640) por
Testing concepts [3] - Software Testing Techniques (CIS640)Testing concepts [3] - Software Testing Techniques (CIS640)
Testing concepts [3] - Software Testing Techniques (CIS640)Venkatesh Prasad Ranganath
554 visualizações33 slides
How to report bugs por
How to report bugsHow to report bugs
How to report bugsMahmoud Asadi
906 visualizações15 slides
Introduce Game Testing And QA por
Introduce Game Testing And QAIntroduce Game Testing And QA
Introduce Game Testing And QAPham Anh Tuan
2K visualizações13 slides
Katalon Studio - Successful Test Automation for both Testers and Developers por
Katalon Studio - Successful Test Automation for both Testers and DevelopersKatalon Studio - Successful Test Automation for both Testers and Developers
Katalon Studio - Successful Test Automation for both Testers and DevelopersKatalon Studio
3K visualizações33 slides
Alpha beta and acceptance testing por
Alpha beta and acceptance testing Alpha beta and acceptance testing
Alpha beta and acceptance testing shah baadshah
2.1K visualizações11 slides
Software Testing por
Software TestingSoftware Testing
Software TestingSengu Msc
869 visualizações26 slides

Mais conteúdo relacionado

Mais procurados

Defects in software testing por
Defects in software testingDefects in software testing
Defects in software testingsandeepsingh2808
4.3K visualizações7 slides
Katalon Studio - Best automation solution for software testing team por
Katalon Studio - Best automation solution for software testing teamKatalon Studio - Best automation solution for software testing team
Katalon Studio - Best automation solution for software testing teamKatalon Studio
9.1K visualizações13 slides
Presentation On Software Testing Bug Life Cycle por
Presentation On Software Testing Bug Life CyclePresentation On Software Testing Bug Life Cycle
Presentation On Software Testing Bug Life CycleRajon
2.5K visualizações8 slides
Manual testing interview questions and answers por
Manual testing interview questions and answersManual testing interview questions and answers
Manual testing interview questions and answerskaranmca
21.3K visualizações20 slides
Manual testing ppt por
Manual testing pptManual testing ppt
Manual testing pptSantosh Maranabasari
35.8K visualizações15 slides
Gray box testing por
Gray box testingGray box testing
Gray box testingDasun Eranthika
7.7K visualizações10 slides

Mais procurados(20)

Defects in software testing por sandeepsingh2808
Defects in software testingDefects in software testing
Defects in software testing
sandeepsingh28084.3K visualizações
Katalon Studio - Best automation solution for software testing team por Katalon Studio
Katalon Studio - Best automation solution for software testing teamKatalon Studio - Best automation solution for software testing team
Katalon Studio - Best automation solution for software testing team
Katalon Studio9.1K visualizações
Presentation On Software Testing Bug Life Cycle por Rajon
Presentation On Software Testing Bug Life CyclePresentation On Software Testing Bug Life Cycle
Presentation On Software Testing Bug Life Cycle
Rajon 2.5K visualizações
Manual testing interview questions and answers por karanmca
Manual testing interview questions and answersManual testing interview questions and answers
Manual testing interview questions and answers
karanmca21.3K visualizações
Manual testing ppt por Santosh Maranabasari
Manual testing pptManual testing ppt
Manual testing ppt
Santosh Maranabasari35.8K visualizações
Gray box testing por Dasun Eranthika
Gray box testingGray box testing
Gray box testing
Dasun Eranthika7.7K visualizações
Bug life cycle por BugRaptors
Bug life cycleBug life cycle
Bug life cycle
BugRaptors5.6K visualizações
SOFTWARE TESTING por Priyanka Karancy
SOFTWARE TESTINGSOFTWARE TESTING
SOFTWARE TESTING
Priyanka Karancy7.1K visualizações
Automation Testing por AbdulImrankhan7
Automation TestingAutomation Testing
Automation Testing
AbdulImrankhan7367 visualizações
Basic Guide to Manual Testing por Hiral Gosani
Basic Guide to Manual TestingBasic Guide to Manual Testing
Basic Guide to Manual Testing
Hiral Gosani1.3K visualizações
Test Case Management Tools por Malang QA Community
Test Case Management ToolsTest Case Management Tools
Test Case Management Tools
Malang QA Community553 visualizações
Dart and Flutter Basics.pptx por DSCVSSUT
Dart and Flutter Basics.pptxDart and Flutter Basics.pptx
Dart and Flutter Basics.pptx
DSCVSSUT1.1K visualizações
Mobile Application Testing por SWAAM Tech
Mobile Application TestingMobile Application Testing
Mobile Application Testing
SWAAM Tech24.5K visualizações
Manual testing interview question by INFOTECH por Pravinsinh
Manual testing interview question by INFOTECHManual testing interview question by INFOTECH
Manual testing interview question by INFOTECH
Pravinsinh305.4K visualizações
Mobile Application Testing por Noor Orfahly
Mobile Application TestingMobile Application Testing
Mobile Application Testing
Noor Orfahly1.2K visualizações
Testing por Sonali Chauhan
TestingTesting
Testing
Sonali Chauhan3.2K visualizações
Software testing.ppt por Komal Garg
Software testing.pptSoftware testing.ppt
Software testing.ppt
Komal Garg5.9K visualizações
Software testing - basics por Prasad Gali
Software testing - basicsSoftware testing - basics
Software testing - basics
Prasad Gali1.3K visualizações
Automation Testing with KATALON Cucumber BDD por RapidValue
Automation Testing with KATALON Cucumber BDDAutomation Testing with KATALON Cucumber BDD
Automation Testing with KATALON Cucumber BDD
RapidValue480 visualizações

Similar a Quality Assurance 1: Why Quality Matters

PHP - Introduction to PHP Bugs - Debugging por
PHP -  Introduction to  PHP Bugs - DebuggingPHP -  Introduction to  PHP Bugs - Debugging
PHP - Introduction to PHP Bugs - DebuggingVibrant Technologies & Computers
790 visualizações83 slides
debugging (1).ppt por
debugging (1).pptdebugging (1).ppt
debugging (1).pptjerlinS1
267 visualizações83 slides
Testing & should i do it por
Testing & should i do itTesting & should i do it
Testing & should i do itMartin Sykora
283 visualizações13 slides
SE - Lecture 8 - Software Testing State Diagram.pptx por
SE - Lecture 8 - Software Testing  State Diagram.pptxSE - Lecture 8 - Software Testing  State Diagram.pptx
SE - Lecture 8 - Software Testing State Diagram.pptxTangZhiSiang
11 visualizações46 slides
Lewis brady engine_terminology (edited version) por
Lewis brady engine_terminology (edited version)Lewis brady engine_terminology (edited version)
Lewis brady engine_terminology (edited version)LewisB2013
199 visualizações7 slides
Y1 gd engine_terminology por
Y1 gd engine_terminologyY1 gd engine_terminology
Y1 gd engine_terminologydavidhall1415
387 visualizações13 slides

Similar a Quality Assurance 1: Why Quality Matters(20)

debugging (1).ppt por jerlinS1
debugging (1).pptdebugging (1).ppt
debugging (1).ppt
jerlinS1267 visualizações
Testing & should i do it por Martin Sykora
Testing & should i do itTesting & should i do it
Testing & should i do it
Martin Sykora283 visualizações
SE - Lecture 8 - Software Testing State Diagram.pptx por TangZhiSiang
SE - Lecture 8 - Software Testing  State Diagram.pptxSE - Lecture 8 - Software Testing  State Diagram.pptx
SE - Lecture 8 - Software Testing State Diagram.pptx
TangZhiSiang11 visualizações
Lewis brady engine_terminology (edited version) por LewisB2013
Lewis brady engine_terminology (edited version)Lewis brady engine_terminology (edited version)
Lewis brady engine_terminology (edited version)
LewisB2013199 visualizações
Y1 gd engine_terminology por davidhall1415
Y1 gd engine_terminologyY1 gd engine_terminology
Y1 gd engine_terminology
davidhall1415387 visualizações
TDD Best Practices por Attila Bertók
TDD Best PracticesTDD Best Practices
TDD Best Practices
Attila Bertók517 visualizações
Bug best practice por gaoliang641
Bug best practiceBug best practice
Bug best practice
gaoliang641775 visualizações
Lewis brady engine terminology (edited version) por LewisB2013
Lewis brady engine terminology (edited version)Lewis brady engine terminology (edited version)
Lewis brady engine terminology (edited version)
LewisB2013181 visualizações
Y1 gd engine_terminology por kieranowens1997
Y1 gd engine_terminologyY1 gd engine_terminology
Y1 gd engine_terminology
kieranowens1997259 visualizações
Putting the pro in programmer por Switch Systems Ltd
Putting the pro in programmerPutting the pro in programmer
Putting the pro in programmer
Switch Systems Ltd595 visualizações
Reverse engineering – debugging fundamentals por Eran Goldstein
Reverse engineering – debugging fundamentalsReverse engineering – debugging fundamentals
Reverse engineering – debugging fundamentals
Eran Goldstein862 visualizações
From SLO to GOTY por ScyllaDB
From SLO to GOTYFrom SLO to GOTY
From SLO to GOTY
ScyllaDB1.2K visualizações
5-Ways-to-Revolutionize-Your-Software-Testing por Mary Clemons
5-Ways-to-Revolutionize-Your-Software-Testing5-Ways-to-Revolutionize-Your-Software-Testing
5-Ways-to-Revolutionize-Your-Software-Testing
Mary Clemons158 visualizações
Role of tester in gaming por Rahul S Singh
Role of tester in gamingRole of tester in gaming
Role of tester in gaming
Rahul S Singh304 visualizações
Software testing and quality assurance por Benjamin Baumann
Software testing and quality assuranceSoftware testing and quality assurance
Software testing and quality assurance
Benjamin Baumann592 visualizações
Y1 gd engine_terminology (1) por RehanaWhiteley
Y1 gd engine_terminology (1)Y1 gd engine_terminology (1)
Y1 gd engine_terminology (1)
RehanaWhiteley149 visualizações
179 black-box-software-testing-copyright-2003-cem-kaner1652 por ngothanhtungth
179 black-box-software-testing-copyright-2003-cem-kaner1652179 black-box-software-testing-copyright-2003-cem-kaner1652
179 black-box-software-testing-copyright-2003-cem-kaner1652
ngothanhtungth179 visualizações
Bug Advocacy por Deepu S Nath
Bug AdvocacyBug Advocacy
Bug Advocacy
Deepu S Nath539 visualizações
Testing fundamentals por Abdul Basit
Testing fundamentalsTesting fundamentals
Testing fundamentals
Abdul Basit1.3K visualizações

Mais de Marc Miquel

User Experience 8: Business, Ethics and More por
User Experience 8: Business, Ethics and MoreUser Experience 8: Business, Ethics and More
User Experience 8: Business, Ethics and MoreMarc Miquel
324 visualizações83 slides
User Experience 7: Quantitative Methods, Questionnaires, Biometrics and Data ... por
User Experience 7: Quantitative Methods, Questionnaires, Biometrics and Data ...User Experience 7: Quantitative Methods, Questionnaires, Biometrics and Data ...
User Experience 7: Quantitative Methods, Questionnaires, Biometrics and Data ...Marc Miquel
433 visualizações92 slides
User Experience 6: Qualitative Methods, Playtesting and Interviews por
User Experience 6: Qualitative Methods, Playtesting and InterviewsUser Experience 6: Qualitative Methods, Playtesting and Interviews
User Experience 6: Qualitative Methods, Playtesting and InterviewsMarc Miquel
924 visualizações95 slides
User Experience 5: User Centered Design and User Research por
User Experience 5: User Centered Design and User ResearchUser Experience 5: User Centered Design and User Research
User Experience 5: User Centered Design and User ResearchMarc Miquel
253 visualizações60 slides
User Experience 4: Usable User Interface por
User Experience 4: Usable User InterfaceUser Experience 4: Usable User Interface
User Experience 4: Usable User InterfaceMarc Miquel
1.6K visualizações225 slides
User Experience 3: User Experience, Usability and Accessibility por
User Experience 3: User Experience, Usability and AccessibilityUser Experience 3: User Experience, Usability and Accessibility
User Experience 3: User Experience, Usability and AccessibilityMarc Miquel
1.6K visualizações109 slides

Mais de Marc Miquel(19)

User Experience 8: Business, Ethics and More por Marc Miquel
User Experience 8: Business, Ethics and MoreUser Experience 8: Business, Ethics and More
User Experience 8: Business, Ethics and More
Marc Miquel324 visualizações
User Experience 7: Quantitative Methods, Questionnaires, Biometrics and Data ... por Marc Miquel
User Experience 7: Quantitative Methods, Questionnaires, Biometrics and Data ...User Experience 7: Quantitative Methods, Questionnaires, Biometrics and Data ...
User Experience 7: Quantitative Methods, Questionnaires, Biometrics and Data ...
Marc Miquel433 visualizações
User Experience 6: Qualitative Methods, Playtesting and Interviews por Marc Miquel
User Experience 6: Qualitative Methods, Playtesting and InterviewsUser Experience 6: Qualitative Methods, Playtesting and Interviews
User Experience 6: Qualitative Methods, Playtesting and Interviews
Marc Miquel924 visualizações
User Experience 5: User Centered Design and User Research por Marc Miquel
User Experience 5: User Centered Design and User ResearchUser Experience 5: User Centered Design and User Research
User Experience 5: User Centered Design and User Research
Marc Miquel253 visualizações
User Experience 4: Usable User Interface por Marc Miquel
User Experience 4: Usable User InterfaceUser Experience 4: Usable User Interface
User Experience 4: Usable User Interface
Marc Miquel1.6K visualizações
User Experience 3: User Experience, Usability and Accessibility por Marc Miquel
User Experience 3: User Experience, Usability and AccessibilityUser Experience 3: User Experience, Usability and Accessibility
User Experience 3: User Experience, Usability and Accessibility
Marc Miquel1.6K visualizações
User Experience 2: Psychology Concepts por Marc Miquel
User Experience 2: Psychology ConceptsUser Experience 2: Psychology Concepts
User Experience 2: Psychology Concepts
Marc Miquel402 visualizações
User Experience 1: What is User Experience? por Marc Miquel
User Experience 1: What is User Experience?User Experience 1: What is User Experience?
User Experience 1: What is User Experience?
Marc Miquel320 visualizações
Quality Assurance 2: Searching for Bugs por Marc Miquel
Quality Assurance 2: Searching for BugsQuality Assurance 2: Searching for Bugs
Quality Assurance 2: Searching for Bugs
Marc Miquel211 visualizações
Game Balance 3: Interesting Strategies por Marc Miquel
Game Balance 3: Interesting StrategiesGame Balance 3: Interesting Strategies
Game Balance 3: Interesting Strategies
Marc Miquel548 visualizações
Game Balance 3: Player Equality and Fairness por Marc Miquel
Game Balance 3: Player Equality and FairnessGame Balance 3: Player Equality and Fairness
Game Balance 3: Player Equality and Fairness
Marc Miquel525 visualizações
Game Balance 2: Sustained Uncertainty por Marc Miquel
Game Balance 2: Sustained UncertaintyGame Balance 2: Sustained Uncertainty
Game Balance 2: Sustained Uncertainty
Marc Miquel421 visualizações
Game Balance 1: What is Game Balance por Marc Miquel
Game Balance 1: What is Game BalanceGame Balance 1: What is Game Balance
Game Balance 1: What is Game Balance
Marc Miquel2.6K visualizações
public presentation of "Calçotada Wars" the card game por Marc Miquel
public presentation of "Calçotada Wars" the card gamepublic presentation of "Calçotada Wars" the card game
public presentation of "Calçotada Wars" the card game
Marc Miquel130 visualizações
public presentation of "La Puta i la Ramoneta" the card game por Marc Miquel
public presentation of "La Puta i la Ramoneta" the card gamepublic presentation of "La Puta i la Ramoneta" the card game
public presentation of "La Puta i la Ramoneta" the card game
Marc Miquel191 visualizações
Towards a User-Centered Wikipedia - Viquitrobada, 26 de novembre 2016, València por Marc Miquel
Towards a User-Centered Wikipedia - Viquitrobada, 26 de novembre 2016, ValènciaTowards a User-Centered Wikipedia - Viquitrobada, 26 de novembre 2016, València
Towards a User-Centered Wikipedia - Viquitrobada, 26 de novembre 2016, València
Marc Miquel129 visualizações
Cultural Identities in Wikipedia (Wikimania 2016) por Marc Miquel
Cultural Identities in Wikipedia (Wikimania 2016)Cultural Identities in Wikipedia (Wikimania 2016)
Cultural Identities in Wikipedia (Wikimania 2016)
Marc Miquel455 visualizações
Happiness Has To Do With Clarity - World Information Architecture Day '15 por Marc Miquel
Happiness Has To Do With Clarity - World Information Architecture Day '15Happiness Has To Do With Clarity - World Information Architecture Day '15
Happiness Has To Do With Clarity - World Information Architecture Day '15
Marc Miquel606 visualizações
The Elements of Videogambling Experience por Marc Miquel
The Elements of Videogambling ExperienceThe Elements of Videogambling Experience
The Elements of Videogambling Experience
Marc Miquel3K visualizações

Último

MVP and prioritization.pdf por
MVP and prioritization.pdfMVP and prioritization.pdf
MVP and prioritization.pdfrahuldharwal141
37 visualizações8 slides
Igniting Next Level Productivity with AI-Infused Data Integration Workflows por
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Safe Software
317 visualizações86 slides
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas... por
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...Bernd Ruecker
48 visualizações69 slides
Microsoft Power Platform.pptx por
Microsoft Power Platform.pptxMicrosoft Power Platform.pptx
Microsoft Power Platform.pptxUni Systems S.M.S.A.
61 visualizações38 slides
Info Session November 2023.pdf por
Info Session November 2023.pdfInfo Session November 2023.pdf
Info Session November 2023.pdfAleksandraKoprivica4
15 visualizações15 slides
Piloting & Scaling Successfully With Microsoft Viva por
Piloting & Scaling Successfully With Microsoft VivaPiloting & Scaling Successfully With Microsoft Viva
Piloting & Scaling Successfully With Microsoft VivaRichard Harbridge
13 visualizações160 slides

Último(20)

MVP and prioritization.pdf por rahuldharwal141
MVP and prioritization.pdfMVP and prioritization.pdf
MVP and prioritization.pdf
rahuldharwal14137 visualizações
Igniting Next Level Productivity with AI-Infused Data Integration Workflows por Safe Software
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Safe Software317 visualizações
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas... por Bernd Ruecker
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...
Bernd Ruecker48 visualizações
Microsoft Power Platform.pptx por Uni Systems S.M.S.A.
Microsoft Power Platform.pptxMicrosoft Power Platform.pptx
Microsoft Power Platform.pptx
Uni Systems S.M.S.A.61 visualizações
Info Session November 2023.pdf por AleksandraKoprivica4
Info Session November 2023.pdfInfo Session November 2023.pdf
Info Session November 2023.pdf
AleksandraKoprivica415 visualizações
Piloting & Scaling Successfully With Microsoft Viva por Richard Harbridge
Piloting & Scaling Successfully With Microsoft VivaPiloting & Scaling Successfully With Microsoft Viva
Piloting & Scaling Successfully With Microsoft Viva
Richard Harbridge13 visualizações
Ransomware is Knocking your Door_Final.pdf por Security Bootcamp
Ransomware is Knocking your Door_Final.pdfRansomware is Knocking your Door_Final.pdf
Ransomware is Knocking your Door_Final.pdf
Security Bootcamp66 visualizações
Democratising digital commerce in India-Report por Kapil Khandelwal (KK)
Democratising digital commerce in India-ReportDemocratising digital commerce in India-Report
Democratising digital commerce in India-Report
Kapil Khandelwal (KK)20 visualizações
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f... por TrustArc
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...
TrustArc72 visualizações
HTTP headers that make your website go faster - devs.gent November 2023 por Thijs Feryn
HTTP headers that make your website go faster - devs.gent November 2023HTTP headers that make your website go faster - devs.gent November 2023
HTTP headers that make your website go faster - devs.gent November 2023
Thijs Feryn26 visualizações
STPI OctaNE CoE Brochure.pdf por madhurjyapb
STPI OctaNE CoE Brochure.pdfSTPI OctaNE CoE Brochure.pdf
STPI OctaNE CoE Brochure.pdf
madhurjyapb14 visualizações
【USB韌體設計課程】精選講義節錄-USB的列舉過程_艾鍗學院 por IttrainingIttraining
【USB韌體設計課程】精選講義節錄-USB的列舉過程_艾鍗學院【USB韌體設計課程】精選講義節錄-USB的列舉過程_艾鍗學院
【USB韌體設計課程】精選講義節錄-USB的列舉過程_艾鍗學院
IttrainingIttraining69 visualizações
Future of AR - Facebook Presentation por ssuserb54b561
Future of AR - Facebook PresentationFuture of AR - Facebook Presentation
Future of AR - Facebook Presentation
ssuserb54b56122 visualizações
"Running students' code in isolation. The hard way", Yurii Holiuk por Fwdays
"Running students' code in isolation. The hard way", Yurii Holiuk "Running students' code in isolation. The hard way", Yurii Holiuk
"Running students' code in isolation. The hard way", Yurii Holiuk
Fwdays24 visualizações
Case Study Copenhagen Energy and Business Central.pdf por Aitana
Case Study Copenhagen Energy and Business Central.pdfCase Study Copenhagen Energy and Business Central.pdf
Case Study Copenhagen Energy and Business Central.pdf
Aitana17 visualizações
Five Things You SHOULD Know About Postman por Postman
Five Things You SHOULD Know About PostmanFive Things You SHOULD Know About Postman
Five Things You SHOULD Know About Postman
Postman38 visualizações
Evolving the Network Automation Journey from Python to Platforms por Network Automation Forum
Evolving the Network Automation Journey from Python to PlatformsEvolving the Network Automation Journey from Python to Platforms
Evolving the Network Automation Journey from Python to Platforms
Network Automation Forum17 visualizações

Quality Assurance 1: Why Quality Matters

  • 1. Practical Lesson 1: The testing process
  • 2. What is a bug?
  • 3. Quality is the absence of bugs. Bug is a behavior not in someone else’s expectations.
  • 4. Different Sorts of Bugs • Quality is no bugs, but, what is a bug or defect? A bug is caused by a mistake, it may imply an error (in values), a lack of accordance to an specification/standard, and (it may) cause a system failure. 1. By classifying bugs into categories, we can understand their importance and learn how to spot them. 2. By classifying bugs into types, we can understand where they could be originated. 3. By classifying bugs into severity levels, we can understand their priority in the development process. Make sure to investigate invisible walls very closely to avoid being NAB’d (when a developer re-classifies your “bug” as “Not a Bug”). By the way, this also implies you don’t know how to do your job!)
  • 5. Per Olin classified bugs according to categories (Levy et al. 2009, Game Development Essentials: Game QA & Testing; p. 77).
  • 9. Orthogonal Defect Classification (ODC) system developed by IBM allows to classify defects into types in order to reveal how they were introduced, be found or avoided (Schultz, 2011, Game Testing All in One; p. 42). • A Function error is one that affects a game capability or how the user experiences the game. The code providing this function is missing or incorrect in some or all instances where it is required. • A Assignment type error is the result of incorrectly setting or initializing a value used by the program or when a required value assignment is missing. • A Checking defect type occurs when the code fails to properly validate data before it is used. For instance, the quantity of ‘ammo’ in a shooter. • Timing defects have to do with the management of shared and real-time resources. • Build/package/merge or, simply Build defects are the result of mistakes in using the game source code library system, managing changes to game files, or identifying and controlling which versions get built. • Algorithm defects include efficiency or correctness problems that result from some calculation or decision process. • Documentation defects occur in the fixed data assets that go into the game. This includes text, audio, and graphics file content. • Interface defects occurs where information is being transferred or exchanged.
  • 10. Bug severity levels (Per Olin) It can also be tagged with a number on the range 1 to 4.
  • 11. Normal bugs vs. Tough bugs (Per Olin) Advices to deal with a ‘tough bug’: • Once a tough bug appears, take a screenshot with the dev menu, debug data, etc. • Immediately talk to the lead tester. • Tell everybody to reproduce this (the exact steps).
  • 12. The lifecycle of a Build A testing process aims at finding bugs at every build (or version) of the game. A basic game testing process in a QA/QC dept. consists of the following steps: 1. Plan and design the test. 2. Prepare for testing. 3. Perform the test. 4. Report the results. 5. Repair the bug. 6. Return to step 1 and re-test.
  • 13. The lead tester, primary tester, or any other tester tasked with test creation should draft these documents prior to the distribution of the build. They tell the tester “go for X build version and Y devices and take test suite Z”. Sometimes when a test suite is generalist a test case “does not apply”. This is a test suite for minesweeper
  • 14. Lead Tester receives the build… Builds submitted to testing that don’t meet the basic entry criteria are likely to waste the time of both testers and programmers. The countdown to testing should stop until the test “launch” criteria are sufficiently met. These are the criteria: • The game code should be built without compiler errors. Any new compiler warnings that occur are analyzed and discussed with the test team. • The code release notes should be complete and provide the detail that testers need to plan which tests to run or re-run for this build. • Defect records for any bugs closed in the new release should be updated so they can be used by testers to make decisions about how much to test in the new build. • Tests and builds should be properly version controlled, as described in the following sidebar (version control is for everyone, developers and testers). • When you are sufficiently close to the end of the project, you also want to receive the game on the media that it will ship on. Check that the media provided contains all of the files that would be provided to your customer.
  • 15. Configuration Preparation (config prep) Before the test team can work with the new build, some housekeeping is in order. The test equipment must be readied for a new round of testing. • Wiping the “old” build in the computer. You need a clean computer. Save the old files in a server or common space. • The latest drivers for all components of the computer. This not only includes the obvious video card and sound card drivers, but also chipset drivers, motherboard drivers, ethernet card drivers, and so on. • The latest versions of any “helper apps” or middleware the game requires to run. These can range from Microsoft’s DirectX multimedia libraries to multiplayer matchmaking software such as GameSpy Arcade. The only other software on the computer should be the new build.
  • 16. Ready, tester? Wait a second. The next step after accepting a new build and preparing to test it is to certify that the build is worthwhile to formally test. • Smoke Testing. At a minimum, it should consist of a “load & launch,” that is, the lead or primary tester should launch the game, enter each module from the main menu, and spend a minute or two playing each module. • So the build is distributed. Time to test for new bugs, right? Not just yet. Before testing can take a step forward, you must take a step backward and verify that the bugs the development team claims to have fixed in this build are indeed fixed. They are in the “knockdown list”. This process is known as Regression testing. Sometimes critical bugs are not retired from the regression testing and the regression testing may last for the entire development.
  • 17. Reporting a bug Possible audience of a bug: lead tester, project manager, marketing, third parties, customer service,… obviously other testers. In fact, the primary tester should be allowed to modify all fields in the bug database except for Priority, Status, Assigned To, and Developer Comments. How to write good reporting? • Only facts. “The guard’s hat should be blue” or “the AI is weak”. • Brief Description or title. The first sentence should contain a brief and clear description of the bug: “Crash to desktop when choosing Options from Main Menu”. YES “Game crashed after I killed all the guards and doubled back through the level to get all the pick-ups and killed the first re-spawned guard.” NO. too much detail. “Game crashed after guards respawned”. YES
  • 18. Advice: write the full description first and then the brief description (title). • Full description. This provides all the necessary details. Rather than a prose discussion of the defect, the full description should be written as a series of brief instructions so that anyone can follow the steps and reproduce the bug. The steps should be written in second person imperative, as though you were telling someone what to do. The last step is a sentence (or two) describing the bad result. 1. Launch game. 2. Choose Multiplayer. 3. Choose Skirmish. 4. Choose “Sorrowful Shoals” map. 5. Choose two players. 6. Start game. Or much better: 1. Start a two-player skirmish game on “Sorrowful Shoals.”
  • 19. EXAMPLE: 1. Create a multiplayer game. 2. Click Game Settings. 3. Using the mouse, click any map on the list. Remember the map you clicked on. 4. Press up or down directional keys on your keyboard. 5. Notice the highlight changes. Highlight any other map. 6. Click Back. 7. Click Start Game. Expected Result: Game loads map you chose with the keyboard and mouse. Actual Result: Game loads map you chose with the mouse. Advice 2: Consider introducing the expected results as it may not be obvious for the programmer. It is also a good reminder of how test cases are produced. Programmers should be educated in the testing process.
  • 20. Other things we MUST include: • Components versions. Build version, specific drivers versions, etcetera. • Bug priority. The tester sets the possible priority (1-4 or minor, major to critical). • Bug category. The tester should use the category “audio”, “artificial intelligence”, “physics”, etc, so it narrows the scope • Bug possible origin. The tester can include it in a line near the beginning, so it may help the programmer in understanding the bug. The tester may say it comes from “function”, “assignment”, wrong “build”, etc. Things to avoid: • Humor. A In a high-stress environment it may fuel the tempers. • Jargon! Write in a clear language. Testers should avoid very technical terms (source of confusion if not used properly). EXERCISE: check in the Internet for a bug found by a player on a game already released. Where is it reported? Is it reported properly according to all the fields and recommendations?
  • 21. Some Common Mistakes · Non-summing summary line. "Found a bug." No kidding! Like all these other bugs in the database aren't bugs that somebody found? That's like sending an email with the subject line, "Hey" - or "I wrote an email" - or "From me." Sum up the problem! Say what the problem is. Just like when you write an email, you give the recipient an idea what the email is about before he or she opens it. · Too-long summary line. "The slithy toves gyred and gimbled in the wabe when the borogoves were all mimsy and the mome raths outgrabe." Dude, just say "The slithy toves gyred and gimbled." That condenses the essence of the problem. You can give us the details about all the excessively mimsy borogoves in the body of the report. · Tester as designer. "The slithy toves need to have a 5-second cooldown after gyring so they don't gimble too soon." No, don't tell the developer how to fix it. Just say what the problem is. It's the designer's job to figure out what's the best way to balance the slithy toves. · Not giving step by step instructions. Tell the developer what to do, so he can see the problem himself. Don't just say "I did this, then I did that." Tell him, "do this, then do that." Give step-by-step instructions, in detail, in the second person (imperative). [http://www.sloperama.com/advice/faq75.htm]
  • 22. · Unclear basis for an expectation. "Usually, pressing X causes the door to open." What do you mean, "usually"? Do you mean that's how one opens doors in other games? Do you mean that's how one opens other doors in this game? Do you mean doors in your home open when you press an X button? What does "usually" mean?? Be specific, dude! · Confusing "player" with "player character". The player is the human who's holding the game controller. That digital being on the screen is a character. Don't use the terms interchangeably. · Wishy-washy observation step. "Then watch to see if the problem happens or not." Wrong. Tell the developer he will see the problem. Tell him to observe that it does happen. · Inappropriate use of the word "should" as in: "After you follow these steps, you should see the bug happen." Um, what? The bug should not happen -- that's why it's a bug! If it was supposed to happen, then it wouldn't be a bug. So you shouldn't say "should" in this way. Just say "after following these steps, observe that the bug happens." [http://www.sloperama.com/advice/faq75.htm]
  • 23. True or False: • A “Verify Fix” status on a reported bug means it will remain in at least one more test cycle. • A tester should write as many steps as possible when reporting a bug to ensure the bug can be reliably re-created.
  • 24. Lesson 1: Why Quality Matters Third year course in Quality Assurance and Game Balance Bachelor Degree in Video Game Design and Production Third term, April 2019 Dr. Marc Miquel Ribé
  • 25. Overview of the Lesson In this lesson we will see the following topics: What is Quality in Games 1. Quality Matters 2. How Can We Improve Quality 3. Different Sorts of Testing 4. Approaches: Blackbox and Whitebox 5. Being a Black-box Tester. QA and QM: Team, Phases and Documentation 1. TDD and Test plan 2. Testing phases 3. Quality management measurements
  • 26. What is Quality in Games?
  • 27. What is quality? • Quality is ”how well we do something” • Quality is “being reliable” • Quality is “being special” • Quality is ”being superior” Quality may mean many things ‘in general’… Some questions about quality ‘in particular’: • What is quality in software? Does quality matter in software? • What is quality in video games? Does quality matter in video games? • How does user experience relate to video game quality? • How do features relate to video game quality?
  • 28. Think about a Space shuttle, think about Microsoft Word, think about a car. • Is quality the same important for them? In certain services or products, no quality means risking people’s lives or work.
  • 29. Quality is “managing expectations”. In video games, new mechanics and new experiences may matter more than quality… But still, in a competing market, defects can ruin experiences. We need to work for quality. Mainly, we need to test quality. [https://www.gamesradar.com/top-7-horrendously-buggy-games-we-loved-anyway/]
  • 30. Quality is “managing expectations”. Quality is about working properly according to someone else’s expectations: game console manufacturer, customer, etcetera. Quality is the degree in which some characteristics are adapted to the requirements put into a product or service (ISO 9000:2000), i.e. (in games: the mechanics, the visuals requirements, etc.). Quality is ‘what is required’: functionalities, expectations and the absence of bugs (anomalies or non-defined behaviors). “The general aim of testing is to affirm the quality of software systems by systematically ex- ercising the software in carefully controlled circumstances” (1981). E. F. Miller. Introduction to Software Testing Technology. Second edition.
  • 31. Quality is the absence of bugs. Bug is a behavior not in someone else’s expectations.
  • 32. In the world of project management, quality, cost and time are key constraints that shape any project. They are the “Iron triangle”. Every project is a fight between: 1. what is required (quality) 2. the date for delivery 3. the agreed budget/cost QUESTION: Which of the following puts pressure on a game project? a) New funding b) Taking people away to have them work on another game c) A requirement to add more levels to be comparable to a competitor’s recently announced title d) b and c. Why Do We Ship Buggy Games? - A Look Behind the Scenes - Extra Credits [https://www.youtube.com/watch?v=s1_50T5GwZ8]
  • 33. @TesterGame_com (QA agency for Software, Games & Apps. Game Testing Specialists) says: “Sólo un 34.8% de las empresas españolas considera el testing como un perfil imprescindible en el desarrollo de un videojuego #libroblancoDEV #gamedev #indiedev #indiegame #indiegamedev #qa” These are results from the survey “Libro Blanco de los Vídeojuegos” from 2017. [https://twitter.com/TesterGame_com/ status/951411246296903682]
  • 34. 1. Quality Matters ALERT: § Quality is a key service aspect. Quality is part of the management, as project success depends on it. However, while quality is important, it is sometimes considered “unoriginal” or the “last part”. It is easy that games have bugs, but quality matters. • Game software is complex. • Software tools are used to produce games and these tools are not perfect. • Games must work on multiple platforms/devices with a variety of configurations. • Games can be played by a large number of people. Online reputation depends on critics opinions and quality matters. Quality must be planned and tested. § Coding and testing are different enough to justify the quality department. As we get more and more professional, we can locate the quality responsibility into one department.
  • 35. • Trust is at the base of (video game) software development, and so every stakeholder (product managers, press, etc.) asks for deliveries and expects each others’ good performance. But… reality is not as beautiful as expected. • The quality department does not know if something works until tested. They cannot trust anyone. • The quality department is given “no time” and “no resources” but still is asked quality. Tension. • How do you balance quality, cost and time? In other words, how do you balance trust and testing? This is where QA comes in.
  • 36. § Testing is the process of verifying that something works properly. Testing is the most essential part required to ensure quality. What do they always tell to the tester? Trust no one. Less trust implies more testing.
  • 37. Tester: “Please, Trust No One” A general form of statements to watch out for is “X happened, so (only/don’t) do Y.” Here are some examples: • “Only a few lines of code have changed, so don’t inspect any other lines.” • “The new audio subsystem works the same as the old one, so you only need to • run your old tests.” • “We added foreign language strings for the dialogs, so just check a few of them in one of the languages and the rest should be okay too.” • “We only made small changes so don’t worry about testing <insert feature name here>.” • “You can just run one or two tests on this and let me know if it works.” • “We’ve gotta get this out today so just ....” WRONG! Quality control needs to test whatever considered relevant, sometimes not listening to the recommendation and constraints coming from other departments. This is the real meaning of “Trust no one”.
  • 38. Tester: “Don’t stress: You are in a rational world”. So, as a tester, you should follow these advice: • Know your role on the team based on the responsibilities assigned to you. • Execute your tasks aggressively and accurately. • Do the most important tests first (priority). • Do the tests most likely to find defects often. • Make emotion-free and objective decisions to the best extent possible. It is not always your fault as a tester if there are still some bugs after the final release (they might be introduced late, they could be hidden because you spent time on other bugs, etc.). Remember the ‘iron triangle’. Conclusion: Testing is not enough to guarantee quality.
  • 39. 3. Different Sorts of Testing Remember: when they say quality, you may ask: ‘according to what or who?’. I want you to structure the topic in questions, as it is easier to remember. • Depending on the “Who” and the “What“ (requirements) with which they define quality, we may find a different sort of testing. • Functionality testing, Item testing (game design, producer) • Compliance testing (platform) • Localization testing (customer environment) • Compatibility testing (components configurations) • Depending on the rest of questions, other testing may appear: • “How”: Blackbox and Whitebox testing. • “When”: Smoke testing, regression testing, beta testing. Let’s leave aside Usability testing and Playtesting, as they are User Experience field: • Usability testing is aimed at finding aspects which block the ease of using an interface. • Playtesting is aimed at understanding the user experience and game aspects (e.g. balance)
  • 40. • Compliance For ensuring that your game complies with the standards and guidelines of major first party developers like Apple, Google, Amazon, Microsoft, Sony and Nintendo. • Guidelines are known as TRC (Technical Requirements Checklist) because of Sony. While Microsoft calls it TCR (Technical Certification Requirements), and Nintendo calls it “Lot Check” since 1980 with Super Mario Bros. Some usual compliance requirements: • The game must not display any religious images. • The game should pause once the controllers are not attached. • Before formatting a memory card or a hard drive, a message warns the user that all the data will be erased. • All copyrighted brands or names are accompanied by their respective copyright or trademark signs. • If data gets corrupted, the game warns the user and suggests reformatting the memory card or hard drive. • System setting for a language. Guidelines are at all levels: design features, fluency, defects, compatibility, etcetera.
  • 41. • QA Team from the manufacturer will test it against their TRC again to be make certain the game complies with platform conventions. The answer is binary: yes/no. • Some companies publishes job offers with “supervise some TRCs” as some of the tasks, as it may be interesting to dedicate someone’s job to be familiar with each platform. • As said, anything may be included in the TRC: the manufacturer branding is also dependent on that.
  • 42. 4. Approaches: Blackbox and Whitebox Depending on the approach, testing is either Blackbox or Whitebox. This is extensible to software testing. 1. White-box testing – focuses on the architectural, integration and systemic aspects of mobile game: how third-party components, databases, social media/external entities, as well as graphics/game engines, audio play and so on are integrated in to your game. 2. Black-box testing – focuses on the functional and overall playability aspects of the game. Menus, graphical elements, special effects, animations, and the actual gameplay are those under test with Black-box approach. Why does Blackbox exists? Programmers don’t fully test their own games (anymore). They don’t have time to, and even if they did, it’s not a good idea. Job segmentation allows programmers to be more professionals and focus on their task. Blackbox is the only testing that guarantees “real conditions”.
  • 43. White-box Testing White-box testing gives the tester an opportunity to exercise the source code directly in ways no end user ever could. It takes into account he internal mechanism of a system or component. Tests performed by developers prior to submitting new code for integration with the rest of the game. Testing code modules that will become part of a reusable library across multiple games and/or platforms; modules that are parts of a game engine or middleware product; modules that can be used by third-party developers or “modders”.
  • 44. • Unit testing: it is the testing of individual hardware or software units or groups of related Units. Testers (usually developers) verify that the code does what it is intended; it is usually done within a class or a component. o Unit testing checks a single assumptions about the behavior of one system. • Integration testing: it is the testing in which software components, hardware components, or both are combined and tested to evaluate the interaction between them. Although it is also done in black-box testing, software developers verify that units work together and check whether they give some error or not (data might get lost across an interface, messages might not get passed properly, or interfaces might not be implemented as specified). o Integration testing checks multiple systems are correctly connected. • Automated testing: it is the testing in which certain configurations or tests are run automatically in order to create situations that are difficult to repeat at large scale by manual testers. Some of the purposes of doing automated testing are to test loading times, graphics performance, server loading, etcetera. QUESTION: Why does White-box testing is more considered QA than QC?
  • 45. What kind of tests might have been performed to this in order not to detect the flaw?
  • 46. 2 unit tests. 0 integration tests.
  • 47. Black-box Testing By their nature, black-box test cases are designed and run by people who do not see the inner workings of the code. They play as if they were players.
  • 48. • Test Case: it is a set of inputs, the expected results and the actual results (outputs). • Test Suite: it is a set of test cases. • Test Plan: it is a document with the test suites and their test cases. The minimal ’unit’ of black-box testing is the Test Case
  • 49. IMPORTANT: Test Case can be defined as an statement ”placement of an object is that” to be resolved as “passes/fails” or as a description of the test case, expected results and actual results (and “passes/fails”). You can create a test case as: • Description (situation and order) > Expected Results > Actual Results > Pass/Fails. The Expected Result is the way the game should work according to its design specification. The Actual Result is anomalous behavior observed when you used the game, caused by a software defect. Or you can create it as: • Description (order) with the expected results > Pass/Fails > Description.
  • 50. Test Suite for Monopoly “Getting in Jail” Requirements according to some Monopoly rules: When a user lands on the “Go to Jail” cell, the player goes directly to jail, does not pass go, does not collect $200. On the next turn, the player must pay $50 to get out of jail and does not roll the dice or advance. If the player does not have enough money, he or she is out of the game.
  • 51. Test cases and suites will be very different depending on the game genre. In a saga, test suites can be reused. You can generate a test case for the previous ‘characteristics’ you already know what to expect from. Again: • A test plan defines the overall structure of the testing cycle. • A test case is one specific question or condition the code is operated and evaluated against. • A test suite is a group of test cases. PROPOSED EXERCISE: Level Test Checklist. Develop this into a test suite. Check the placement and behavior of objects in the level Check the placement and behavior of NPCs Check the fit and form of each unique tile, mesh, and texture used in the level Check the function and load times of transitions from one level to another
  • 52. 5. Being a Black-box Tester • Being a black-box tester is being in a loop. Playing only happens once. In a ‘test suite’, it is not playing, and once you found the first ‘defect’, you are in the middle of the loop. The loop is called “Piano TV”. 1. Play to crash it, not to have fun. 2. Identify the bug. It may be not important, blocking, etc. Finding bugs is this part.
  • 53. 3. Amplifying the bug. This means that the tester will narrow the possibilities in which the bug appears. You may have found a bug in a place, but it may occur more than you think. Amplifying implies testing again (look for all the places which “relate”). 4. Notifying the team. Once the bug is found, you need to report it using a defect tracking system, describing the most useful information (category and type of bug) and extra things you can do to help without being too verbose. The replicability is the most fundamental part (just like science). It is important to put a clear title and set the priority according to the severity. It is useful to include server logs, screenshots, saved files, etcetera. 5. Testify to others. In this moment, the tester sends the report and acknowledges the different teams (lead tester and developer). Remember: testers get emotionally attached to their found defect (they are sort of medals in the testing sport). 6. Verify the fix. Once the bug is fixed by the developer, the tester needs to try to reproduce it. Once it is fixed, the tester may close it.
  • 54. Depending on your personality, you may be one type of tester or another. If you are a Judger, you prefer a structured, ordered, and fairly predictable environment, where you can make decisions and have things settled. If you are a Perceiver, you prefer to experience as much of the world as possible. You like to keep your options open and are most comfortable adapting. Judger Runs the tests for... Conventional game playing Repetitive testing User manual, script testing Factual accuracy of game Step-by-step or checklist-based May rely too much on test details Concerned about game contents Perceiver Finds a way to... Unconventional game playing Testing variety Realistic experience of game Open-ended or outline-based testing May stray from the original test purpose Concerned about game context
  • 55. QA and QM: Team, Phases and Documentation
  • 56. Testing is the execution. It is the method and the process by which QC ensures that all the defects are found. 1. How Can We Improve Quality? Testing may not be enough. We need to plan for quality. We need three departments. These are the quality ‘roles and departments’ in Project Management: • Quality Control & Testing – controlling a process to ensure that outcomes are as expected. Quality control is the lower layer: it includes testing. • Quality Assurance – obtaining confidence that a product or service will be satisfactory, as well as planning for it. • Quality Management – directing an organization so that it optimizes its performance through analysis and improvement.
  • 57. Why is there quality assurance if there is already quality control and testing? In order to focus on quality earlier in the process. If we could test on forever, there would be no need for planning. Planning is putting priorities and doing things to optimize results. Defining the testing and planning the testing are part of Quality Assurance. Quality Assurance developed from the realization that quality could be improved by looking 'further up the line'. It is aimed at preventing nonconformities/defects. There are two kinds of defects that justify QA: 1. The one which is defined according to the design requirements. 2. The one which is perceived by the customer and was not even foreseen by the development team. The two must be taken into account by Quality Assurance in order to improve the game and evaluate the quality.
  • 58. 2. Team and roles Game teams come in different sizes, shapes, locations, and skill sets. They can vary by company, game title, or platform. • It is usual to find a leader defined hierarchy: development lead, build lead, etc. So responsibilities are assigned. • QA department may have a leg in the development team and in testing team. The testing team may be dynamic and grow according to the project needs. • Lead tester (QA lead) is assigned according to a game. o She defines the “testability” requirements, procedures and standards. o She represents the test team in planning and meetings. o She may do some of the game testing (in small teams). • Testers (QC). They are the group of final testers who execute the test plan. • Test Engineer: breaks stuff while programmers try to make it work (basically through White-box testing). She works tightly with programmers. • Beta testers (optionally). “Volunteer army” in case of doing pre-releases.
  • 59. Outside the testing team: • Art Team (animators, artists, etc.), Game Designers (level designers, etc.), Sound Team (sound engineer, music producer). • User Experience (responsible for playtesting, surveys, interviews; creating hypothesis; evaluating the GUI). • Product Manager / Producer: getting the game done on time and within budget. Evaluate risks. More and more. The management team and production is QM.
  • 60. The decision-makers are: Producer (QM), QA Manager and Lead Tester. • Producer Questions: how do we keep the calendar? How do we fit the budget? • QA Manager Questions: how big is the game? (platforms, map size, etc.) how much time do we have? How much money do we have? • Lead tester Questions: What test suites are most important? How can all testers work on something important? How many bugs are open and which category? The main responsibilities of the Lead testers are: managing the test team, designing and implementing the overall test plan, “owning” the bug database. The Lead tester must be very a ‘team’ focused person. Defining documents is (mainly) for the three decision-makers.
  • 61. 3. Documentation • Quality does not start at testing but on the planning phase. Good Quality Assurance and Management requires creating a good set of documents. • Good planning is saving resources (development, testing). Therefore, it is important to determine the Scope of testing the project will require. • Testing better earlier can get problems fixed sooner and cheaper. Earlier means white-box and testing specific parts. It is about balance. • Documentation is key to coordinate. The most important documents for the test manager are the game design document (GDD), the test design document (TDD), software quality assurance plan (SQAP), and the the test plan document (TPD). For whom is documentation most important? • GDD—everyone. • TDD—tech lead, art director, producer, project manager. • SQAP—project manager, producer, development lead, test lead, QA lead, and engineer(s). à This document is Quality Management. • Test plan document—project manager, producer, development lead, key testers. • Test suites—feature developer, testers. (QA & QM)
  • 62. Technical Design Document (TDD) The technical design document sets out the way your tech lead plans to transform the game design from words on a page to software on a machine. • It describes the core tools that will be used to build the game, and whether they are already in the house (not to be bought or created). • It documents the list of software and hardware, and changes in the organization infrastructure (storage capacity, backup plans, network speed). • TDD is sometimes part of the GDD (Game Design Document). When both documents exist, it is recommended that they do not overlap each other in content. Ø GDD is more focused on story, characters, gameplay, art, UI (what and why), instead of the “how” to achieve that. • It is important to keep track of the versions of the document, and each change should be documented in a log along with its creator (as a table). • Different companies may have different TDD outline but the same content.
  • 63. • Its Outline should be something like this: 1. Technical Team: Tech Lead, developers, etcetera. 2. Executive Summary: project overview from a tech. perspective, delivery platforms, etc. 3. Engine Evaluation (internal solutions, external solutions, middleware) 4. Platform-specific issues (delivery platform #1, #2) 5. Development plan (use cases, game mechanics, main loop, data structures, data flow, physics, AI, graphics, collision, GUI, fonts, audio/video, especial requirements) 6. Coding standards (programming standards, style guide, code review procedures) 7. Scheduling (programming tasks, man-month scheduling, calendar month scheduling, milestone schedule and deliverables, major event planning). 8. Recruitment (personnel, hiring, risk plan for handling delays in acquiring resources). 9. Equipment budget and costs (personnel hardware sets, team wide tools, software tools) 10. Miscellaneous Game Design, Second Edition. Bob Bates, 2004. p. 209
  • 64. Software Quality Assurance Plan (SQAP) An SQAP is strictly concerned with the independent monitoring and correction of product and process quality issues. It is a QM document. It should address the following topics: • QA personnel • Standards to be used • Reviews and audits that will be conducted • QA records and reports that will be generated • QA problem reporting and correction • QA tools, techniques, and methods • QA metrics • Supplier control • QA records collection • QA training required • QA risks
  • 65. Test Plan Document (TPD) A test plan acts as the playbook for the test team. It identifies the test team’s goals along with the resources (staff, time, tools, and equipment) and methods that are necessary to achieve them. It should contain: 1. QA Team (QA Lead, internal testers, external testers, etc.) 2. Testing Procedures (Daily Activities such as Generate a daily build, important regression tests, etc.; Integration of reports, different kinds of testing – test suites) 3. Testing Requirements Origin (generated in meetings, generated by producer). 4. Bug Tracking Software (package, instructions how-to, etc.) 5. Bug Classification and Bug Tracking (Who classifies, who assigns, what happens when fixed, what happens when the fix is verified) 6. Scheduling and Loading (rotation plan, when testers are in and out, how many testers needed in total) 7. Equipment budget and costs (team members’ hardware, software tools needed, console debug kit, etc.) The Test Plan Document is as detailed as the Technical Design Document. Again, it should not overlap each other. Example: [https://sites.google.com/site/knightsvszombies/test-plan]
  • 66. 4. Testing Phases and Development No matter what size the game and how long the production schedule, the testing of the game should always follow the same basic structure: Pre-production, Kickoff, Alpha, Beta, Gold, Release and Post-release. The interesting things are “how these phases are defined” and the “test kickoff”. In each game, a Lead Tester is assigned and its up to him to specify the criteria for each phase of testing (in an ideal world, in a real one, other issues matter). On large teams, more than one primary tester may be assigned (e.g. for multiplayer mode, etc.). Each phase needs: an entry criteria (set of tests that a build must pass before being able to tell the game is already at that phase) and the target date for it.
  • 67. Test Kickoff (After Pre-Production) Test kickoff activities are broken into two parts: tester preparation and the kickoff meeting, which is conducted according to the kickoff agenda. For the test kickoff meeting, the tester prepares in the following ways: • Reads the requirements and/or documentation for the game feature being tested • Gathers equipment, files, and programs needed for the test • Reads through the tests Test kickoff meeting is very important 1. Giving a feature overview 2. Addressing feature questions 3. Bringing up any special test instructions 4. Addressing any test execution questions or issues • It prepares the teams, it sets ready the equipment, it familiarizes the tester, it resolves test instruction conflicts, it provides a forum for test improvement.
  • 68. Alpha Phase Entry Criteria (usual ones) 1. All major game features exist and can be tested. 2. A tester can navigate the game along some path from start to finish. 3. The code passes at least 50% of platform TRC. 4. Basic interface is complete and preliminary documentation is available to QA. 5. First-party controllers work. 6. Online multiplayer can be tested. 7. Minimal Performance (e.g. 30 fps) Beta Phase Entry Criteria (usual ones) 1. All features are implemented. 2. The game can be navigated in all paths. 3. The game passes the 100% of the TRC. 4. The entire user interface is final. 5. The game logic and AI is final. 6. All controllers work. 7. Final artwork is implemented. The design lock (or feature lock) happens when this phase is concluded.
  • 69. Gold Testing In all gold testing, the game should be in a similar state to this: 1. All known Severity 1 bugs (crashes, hangs, major function failures) 2. Greater than 90% of all known Severity 2 bugs are fixed. 3. Greater than 85% of all known Severity 3 bugs are fixed. 4. Release-level performance (60-fps frame rate) Upon these criteria, the game is declared to be at “code lock” and the final versions are release candidate or gold master candidates (GMCs). The game is sent to the platform manufacturer, which conducts its own testing on the TRC. They may find “showstopper bugs”, which need to be fix and resubmitted. Post-release testing. Patches.
  • 70. 5. Six-sigma and Phase Containment (PCE) • Quality Management and Quality Assurances aim at improving the testing process, as it is a way of improving the resources efficiency and product quality. • Game Quality Measurements can help us in determining “how good” is game software according to the number of defects it has. They are managed in the SQAP. Two measures are “Sigma Level” and the “Phase Containment”. 1. Six-sigma is a set of techniques and tools for process improvement. It is based on statistical methods. The “Sigma Level” of code is computed according to the number of errors per line (in assembly-equivalent lines of code, AELOC). 2. Phase containment is the ability to detect faults in the project phase in which they were introduced. Phase containment effective (PCE) coefficient is a measure of how well the development and testing process is being done. Both are simple calculations which are fast to compute and easy to understand. (QM & QA)
  • 71. Sigma Level Sigma-Level is based on the Gaussian distribution. For instance, 38 errors for 250.000 lines of code is equivalent to a 5.1 Sigma Value. Released Defects per (AELOC) Sigma 20.000 100.000 250.000 1.000.000 Value 124 621 1552 6.210 4 93 466 1165 4.660 4,1 69 347 867 3.470 4,2 51 256 640 2.560 4,3 37 187 467 1.870 4,4 27 135 337 1.350 4,5 19 96 242 968 4,6 13 68 171 687 4,7 9 48 120 483 4,8 6 33 84 337 4,9 4 23 58 233 5 3 15 39 159 5,1 2 10 27 108 5,2 1 7 18 72 5,3 0 4 12 48 5,4 0 3 8 32 5,5 0 2 5 21 5,6 0 1 3 13 5,7 0 0 2 9 5,8 0 0 1 5 5,9 0 0 0 3,4 6 In this process, the defects being counted must include both the game defects you know about that have not been fixed, whatever defects your customers have already found, and your projection of defects that remain in the software which haven’t been discovered yet.
  • 72. Phase Containment (PCE) Faults that are found in the phase in which they are introduced are known as in-phase faults or “errors.” If found later, they are “defects” or bugs. Calculate PCE by dividing the number of in-phase faults by the sum of faults found in all phases to come up with the PCE for that phase. When new testing phases are added, the PCE can only be reduced.
  • 73. The higher the PCE, the better, because it means that the bugs created at one phase are also found in the same phase. QUESTION: What strategies can be useful to have a higher PCE? - Improve knowledge of the subject matter and provide relevant training. - Have successful team members provide mentoring to less-successful members. - Document methods used by successful individuals and deploy them throughout the team. - Increase compliance with existing methods and standards. - Add standards which, by design, help prevent faults. - Add checking tools that run during the creation process, such as color-coded and syntax-aware editors. - Add checking tools that run after the creation process, such as stronger compilers and memory leak checkers.
  • 74. 6. Testers Effectiveness and Performance • Quality Management uses data regarding the testing process (the editors’ work) in order to understand how well it fits the needs and expectations of the overall project (in terms of duration, new hiring, etc.). • It is useful to contrast planned and actual testing execution data (about the number of tests suite performed), globally and at a tester level. This is contained in the SQAP. (QM)
  • 75. It is important to take into account the tester “overhead” (trainings, meetings, preparing for demos, etc.) so, only counting a 75% of the time at best performance, 50 at average performance and so on. It is useful to count the tests in number of days. This way it is possible to calculate in an Excel file how many extra testers at part-time or full-time can help in pushing the project forward. • It usual that testing teams grow at peak moments of the development process. Monitoring the team to stay on a short-term commitment help in maintaining long- term goals on track. However, monitoring is not all good. Pros: - Some people will do better if they know they are being “watched”. - It provides a measurable basis to detect who are the “elite” and avoid favoritism. - Testers seeking better numbers may interact more with developers to prevent bugs. Cons: - Some testers may not be perceived as valuable as they really are (they mentor, etc.). - It may trigger some jealousy among the team if not channeled properly. - Testers may argue over who gets the credit for certain defects.
  • 76. Testing Effectiveness In a similar way to Phase Containment, testing effectiveness is a measurement which aims at improving the quality control. In this case, it provides a ratio between the number of defects according to the number of tests performed in a release. PCE could be used as a reference. For instance, if you know a PCE coefficient should be 0.60 and 30 tests are performed, then, at least 2 defects should be found. Coders may delay a code release until more defects are found.
  • 77. Testing Effectiveness Testers can be also measured according to their effectiveness. However, you see “what” is happening, but you do not understand “why” is happening that way. You do not know the ‘severity’ of the bugs found by each tester.
  • 78. REWIND: Some innocent questions about the test phases • All bugs MUST be fixed before a game can be certified as GMC. True or false? • Online multiplayer features can be tested in Alpha. True or false? • Feature lock should happen in Alpha. True or false? • The Beta build is the version that will be sent to manufacturing. True or false? • Describe whether each of the following is an appropriate topic to discuss during a test execution kick-off, and why: a) Possible contradictions in the feature requirements b) Ideas for new tests c) Company stock prices d) Identical tests already being run in other test suites e) How “buggy” this feature was in the previous release f) Recent changes to the game data file formats g) Lack of detail in the test case documentation.
  • 79. Summary • Quality is ‘what is required’: functionalities, expectations and the absence of bugs (anomalies or non-defined behaviors). • Quality Management, Quality Assurance and Quality Control are the three areas which deal with quality in the game development process. • Testing is the execution of quality control with the aim of finding defects or bugs. They can be described according to their external characteristics, possible cause and severity. • Quality is not coding better or solving bugs, it is detecting what is not working properly or in best case scenario planning the method to detect. Solving bugs is development. • White-box and Black-box testing are two approaches to bug detection usually in different phases. One is from ”the inside”, as a coder, and the other is “from the outside”, as a player. • Black-box testers cannot trust anyone, and need to plan test suites and perform a rigorous and documented process in order to find bugs, report them and verify them.