Disclaimer: the slide deck does not contain all the content, this was a mostly spoken/hands-on talk.
Code used in the talk: http://github.com/inequation/ggd
Video games are highly complex and indeterministic systems working closely to hardware. This means that sometimes the regular troubleshooting measure of setting up breakpoints just doesn't cut it.
How to filter out noise, when our breakpoint is hit dozens of thousands of times per second? Or if our bug, despite perfect reproduction rate, manifests itself in a different way each time? How to find the culprit overwriting our precious data?
This talk will present some interesting cases from the life of a game programmer along with tips and tricks to have up your sleeve when fighting them.
2. Code snippets
Code snippets
All code used in the talk available online:
github.com/inequation/ggd
2
2
inequation.org
inequation.org
3. Agenda
Agenda
●
Who is this guy?
●
Why the talk?
●
A taste of gamedev debugging
●
Right tool for the job
●
Noise filtering
●
Memory corruption
●
Questions
3
3
inequation.org
inequation.org
4. Who is this guy?
Who is this guy?
Freelance Programmer
●
●
●
(Sep 2013 – onwards)
inequation.org
Painkiller Hell & Damnation
Linux port finalization (2013)
Unannounced project
Generalist Programmer,
The Farm 51
●
●
●
●
4
4
(Mar 2010 – Aug 2013)
thefarm51.com
Painkiller Hell & Damnation
(2012-2013; Win/Linux/X360/PS3)
Deadfall Adventures
(2011-2013; Win/X360)
A few unannounced projects
inequation.org
inequation.org
5. Agenda
Agenda
●
Who is this guy?
●
Why the talk?
●
A taste of gamedev debugging
●
Right tool for the job
●
Noise filtering
●
Memory corruption
●
Questions
5
5
inequation.org
inequation.org
6. Why the talk?
Why the talk?
Because THIS has happened to me:
http://imgur.com/yBa1OGm
6
6
inequation.org
inequation.org
7. Why the talk?
Why the talk?
Intern: I've read* at all the code and
still don't see the bug.
Me:
So just debug it!
*read, as in „stare without actually running it”
7
7
inequation.org
inequation.org
9. Why the talk?
Why the talk?
The three uses of debugging
●
Bug hunting (doh)
●
●
9
9
inequation.org
inequation.org
10. Why the talk?
Why the talk?
The three uses of debugging
Bug hunting (doh)
● Reverse engineering
●
●
10
10
inequation.org
inequation.org
11. Why the talk?
Why the talk?
The three uses of debugging
Bug hunting (doh)
● Reverse engineering
● Testing a new feature
●
11
11
inequation.org
inequation.org
12. Agenda
Agenda
●
Who is this guy?
●
Why the talk?
●
A taste of gamedev debugging
●
Right tool for the job
●
Noise filtering
●
Memory corruption
●
Questions
12
12
inequation.org
inequation.org
13. A taste of gamedev debugging
A taste of gamedev debugging
Code: 01-taste
13
13
inequation.org
inequation.org
14. A taste of gamedev debugging
A taste of gamedev debugging
Can you spot the culprit?
// class declaration
class Crasher extends ActorComponent;
var int DummyArray[1024];
// in ammo consumption code
Crash = new class'Crasher';
Comp = new class'ActorComponent' (Crash);
14
14
inequation.org
inequation.org
15. Agenda
Agenda
●
Who is this guy?
●
Why the talk?
●
A taste of gamedev debugging
●
Right tool for the job
●
Noise filtering
●
Memory corruption
●
Questions
15
15
inequation.org
inequation.org
16. Right tool for the job
Right tool for the job
„Fix a bug for an intern, they will get stuck on
the next one.
Teach them debugging, they will fix most bugs
they encounter on their own.”
– Paulo Coelho
16
16
inequation.org
inequation.org
17. Right tool for the job
Right tool for the job
Debugging tools are essential to this
profession!
When joining a new team or starting development for a
new platform, demand debugging tools
● Ask senior teammates
● If they don't know, there must be documentation
● Be proactive!
● Don't give up until the debugger is fully working
● No tools? Roll your own!
● You are a coder after all, right?
●
17
17
inequation.org
inequation.org
18. Right tool for the job
Right tool for the job
Not all bugs are in the code
Animation graphs
● Flow graphs/visual scripts
● Post-process effects
●
Still need a way to debug them
18
18
inequation.org
inequation.org
19. Right tool for the job
Right tool for the job
Code: 02-tools
19
19
inequation.org
inequation.org
20. Agenda
Agenda
●
Who is this guy?
●
Why the talk?
●
A taste of gamedev debugging
●
Right tool for the job
●
Noise filtering
●
Memory corruption
●
Questions
20
20
inequation.org
inequation.org
21. Noise filtering
Noise filtering
There are parts of code executed thousands
of times each frame
Object transformation
● Collision detection
●
Also rarer, but still impractical to track
Setting materials on objects
● Attaching and detaching of components
●
21
21
inequation.org
inequation.org
23. Agenda
Agenda
●
Who is this guy?
●
Why the talk?
●
A taste of gamedev debugging
●
Right tool for the job
●
Noise filtering
●
Memory corruption
●
Questions
23
23
inequation.org
inequation.org
24. Memory corruption
Memory corruption
Look closely:
// class declaration
class Crasher extends ActorComponent;
var int DummyArray[1024];
// in ammo consumption code
Crash = new class'Crasher';
Comp = new class'ActorComponent' (Crash);
24
24
inequation.org
inequation.org
25. Memory corruption
Memory corruption
Look closely:
// class declaration
class Crasher extends ActorComponent;
var int DummyArray[1024];
// in ammo consumption code
Crash = new class'Crasher';
Comp = new class'ActorComponent' (Crash);
25
25
inequation.org
inequation.org
28. Memory corruption
Memory corruption
But this can happen anywhere! How to find it?
Use a memory fence
● Many related techniques
● Allocate additional space in front and behind actual
allocations
● Then protect them from writing...
● Or write a byte pattern and periodically assert its
consistency
● Also it's useful to log stack traces
● Memory and CPU overhead!
● Use a debug memory allocator (dmalloc)
● Use a memory debugger (Valgrind)
● Use a memory analysis tool (HeapInspector)
●
28
28
inequation.org
inequation.org
30. Takeaway
Takeaway
You can't be an effective programmer without
debugging tools
● If there are no tools, make some
● Noise filtering techniques save your time
● Time is not only money – nerves are just as important!
● Know your machine (physical or virtual) down to the
metal
● Instruction opcodes, registers etc. come in handy
● Tons of resources available
● Random crashes and/or content glitches may indicate
memory corruption
● Memory corruption can be defeated!
●
30
30
inequation.org
inequation.org