4Developers 2015: Gamedev-grade debugging - Leszek Godlewski
•Transferir como ODP, PDF•
0 gostou•244 visualizações
Leszek Godlewski Language: English Video games are complex and non-deterministic systems. So complex, in fact, that some days the everyday breakpoint just doesn't cut it when you're looking for that next bug. Drawing from the experience of deploying three large titles to four platforms, this talk will discuss the different approaches and borderline magical tricks to debugging different parts of a game: noise filtering when our breakpoint is hit way too often, memory stomping, time-dependent bugs, rendering artifacts… Story of a game programmer's life.
Recomendados
Mais conteúdo relacionado
Destaque
Destaque (18)
Semelhante a 4Developers 2015: Gamedev-grade debugging - Leszek Godlewski
Semelhante a 4Developers 2015: Gamedev-grade debugging - Leszek Godlewski (20)
Último
Último (20)
4Developers 2015: Gamedev-grade debugging - Leszek Godlewski
- 1. Gamedev-grade debugging Leszek Godlewski, The Astronauts Source: http://igetyourfail.blogspot.com/2009/01/reaching-out-tale-of-failed-skinning.html
- 2. ● Engine Programmer, The Astronauts (Nov 2014 – present) – PS4 port of The Vanishing of Ethan Carter ● Programmer, Nordic Games (early 2014 – Nov 2014) ● Freelance Programmer (Sep 2013 – early 2014) ● Generalist Programmer, The Farm 51 (Mar 2010 – Aug 2013) Who is this guy?
- 3. Agenda ● How is gamedev different? ● Bug species ● Case studies ● Conclusions
- 4. StartStart Exit?Exit? EndEnd Yes No UpdateUpdate DrawDraw How is gamedev different?
- 5. 33 milliseconds ● How much time you have to get shit done™ – 30 Hz → 33⅓ ms per frame – 60 Hz → 16⅔ ms per frame EditorEditor Level toolsLevel tools Asset toolsAsset tools EngineEngine PhysicsPhysics RenderingRendering AudioAudio NetworkNetwork PlatformPlatform InputInput Network back-end Network back-end GameGame UIUI LogicLogic AIAI
- 6. Interdisciplinary working environment ● Designers – Game, Level, Quest, Audio… ● Artists – Environment, Character, 2D, UI, Concept… ● Programmers – Gameplay, Engine, Tools, UI, Audio… ● Writers ● Composers ● Actors ● Producers ● PR & Marketing Specialists ● … } Tightly woven teams
- 7. Severe, fixed hardware constraints ● Main reason for extensive use of native code
- 9. Indeterminism & complexity ● Leads to poor testability – Parts make no sense in isolation – What exactly is correct? – Performance regressions? Source: https://github.com/memononen/recastnavigation
- 10. Aversion to general software engineering ● Modelling ● Object-Oriented Programming ● Design patterns ● C++ STL ● Templates in general ● …
- 11. Agenda ● How is gamedev different? ● Bug species ● Case studies ● Conclusions
- 13. General programming bugs ● Memory access violations ● Memory stomping/buffer overflows ● Infinite loops ● Uninitialized variables ● Reference cycles ● Floating point precision errors ● Out-Of-Memory/memory fragmentation ● Memory leaks ● Threading errors
- 14. Bad maths ● Incorrect transform order – Matrix multiplication not commutative – AB ≠ BA ● Incorrect transform space Source: http://leadwerks.com/wiki/index.php?title=TFormQuat
- 15. Temporal bugs ● Incorrect update order – for (int i = 0; i < entities.size(); ++i) entities[i].update(); ● Incorrect interpolation/blending – Bad alpha term – Bad blending mode (additive/modulate) ● Deferred effects – After n frames – After n times an action happens – n may be random, indeterministic
- 16. Graphical glitches ● Incorrect render state ● Shader code bugs ● Precision Source: http://igetyourfail.blogspot.com/2009/01/visit-lake-fail-this-weekend.html
- 17. Content bugs ● Incorrect scripts ● Buggy assets Source: http://www.polycount.com/forum/showpost.php?p=1263124&postcount=10466
- 18. Worst part? ● Most cases are two or more of the aforementioned, intertwined
- 19. Agenda ● How is gamedev different? ● Bug species ● Case studies ● Conclusions
- 20. Most material captured by Case studies
- 21. Video settings not updating
- 22. Incorrect weapon after demon mode foreshadowing
- 25. Corpses teleported on death ● In normal gameplay, pawns have simplified movement – Sweep the actor's collision primitive through the world – Slide along slopes, stop against walls Source: http://udn.epicgames.com/Three/PhysicalAnimation.html
- 26. Corpses teleported on death ● Upon death, pawns switch to physics-based movement (ragdoll) Source: http://udn.epicgames.com/Three/PhysicalAnimation.html
- 27. Corpses teleported on death (cont.) ● Physics bodies have separate state from the game actor – Actor does not drive physics bodies, unless requested – If actor is driven by physics simulation, their location is synchronized to the hips bone body's Source: http://udn.epicgames.com/Three/PhysicalAnimation.html
- 28. Corpses teleported on death (cont.) ● Idea: breakpoint in FarMove()? – One function because world octree is updated – Function gets called a gazillion times per frame � – Terrible noise ● Breakpoint condition? – Teleport from arbitrary point A to arbitrary point B – Distance? ● Breakpoint sequence? – Break on death instead – When breakpoint hit, break in FarMove()
- 29. Corpses teleported on death (cont.) ● Cause: physics body driving the actor with out- of-date state ● Fix: request physics body state synchronization to animation before switching to ragdoll
- 30. Weapons floating away from the player
- 31. Weapons floating away from the player
- 32. Weapons floating away from the player ● Extremely rare, only encountered on consoles – Reproduction rate somewhere at 1 in 50 attempts – And never on developer machines � ● Player pawn in a special state for the rollercoaster ride – Many things could go wrong ● For the lack of repro, sprinkled the code with debug logs
- 33. Weapons floating away from the player (cont.) ● Cause: incorrect update order – for (int i = 0; i < entities.size(); ++i) entities[i].update(); – Player pawn forced to update after rollercoaster car – Possible for weapons to be updated before player pawns ● Fix: enforce weapon update after player pawns
- 35. Characters with “rapiers” ● UE3 has ”content cooking” as part of game build pipeline – Redistributable builds are ”cooked” builds ● Artifact appears only in cooked builds
- 36. Characters with “rapiers” (cont.) ● Logs contained assertions for ”out-of-bounds vertices” ● Mesh vertex compression scheme – 32-bit float → 16-bit short int (~50% savings) – Find bounding sphere for all vertices – Normalize all vertices to said sphere radius – Map [-1; 1] floats to [-32768; 32767] 16-bit integers ● Assert condition – for (int i = 0; i < 3; ++i) assert(v[i] >= -1.f && v[i] <= 1.f, ”Out-of-bound vertex!”);
- 37. Characters with “rapiers” (cont.) ● v[i] was NaN – Interesting property of NaN: all comparisons fail – Even with itself ● float f = nanf(); bool b = (f == f); // b is false ● How did it get there?! ● Tracked the NaN all the way down to the raw engine asset!
- 38. Characters with “rapiers” (cont.) ● Cause: ??? ● Fix: re-export the mesh from 3D software – Magic!
- 40. Undeniable assertion ● Happened while debugging ”rapiers” ● Texture compression library without sources ● Flood of non-critical assertions – For almost every texture – Could not ignore in bulk � – Terrible noise ● Solution suggestion taken from [SINILO12]
- 41. Undeniable assertion (cont.) ● Enter disassembly
- 42. Undeniable assertion (cont.) ● Locate assert message function call instruction
- 43. Undeniable assertion (cont.) ● Enter memory view and look up the adress – 0xE8 is the CALL opcode – 4-byte address argument
- 44. Undeniable assertion (cont.) ● NOP it out! – 0x90 is the NOP opcode
- 48. Incorrect player movement ● Recreating player movement from one engine in another (Pain Engine → Unreal Engine 3) ● Different physics engines (Havok vs PhysX) ● Many nuances – Air control – Jump and fall heights – Slope & stair climbing & sliding down
- 49. Incorrect player movement (cont.) ● Main nuance: capsule vs cylinder
- 50. Incorrect player movement (cont.) ● Switching our pawn collision to capsule-based was not an option ● Emulate by sampling the ground under the cylinder instead ● No clever way to debug, just make it ”bug out” and break in debugger
- 51. Incorrect player movement (cont.) ● Situation when getting stuck ● Cause: vanilla UE3 code sent a player locked between non-walkable surfaces into the ”falling” state ● Fix: keep the player in the “walking” state
- 52. Incorrect player movement (cont.) ● Situation when moving without player intent ● Added visualization of sampling, turned on collision display ● Cause: undersampling ● Fix: increase radial sampling resolution 1) 2)
- 54. Blinking full-screen damage effects ● Post-process effects are organized in one-way chains
- 55. Blinking full-screen damage effects (cont.) ● No debugger available to observe the PP chain ● Rolled my own overlay that walked and dumped the chain contents MaterialEffect 'Vignette' Param 'Strength' 0.83 [IIIIIIII ] MaterialEffect 'FilmGrain' Param 'Strength' 0.00 [ ] UberPostProcessEffect 'None' SceneHighLights (X=0.80,Y=0.80,Z=0.80) SceneMidTones (X=0.80,Y=0.80,Z=0.80) … MaterialEffect 'Blood' Param 'Strength' 1.00 [IIIIIIIIII]
- 56. Blinking full-screen damage effects (cont.) ● Cause: entire PP chain override – Breakpoint in chain setting revealed the level script as the source – Overeager level designer ticking one checkbox too many when setting up thunderstorm effects ● Fix: disable chain overriding altogether – No use case for it in our game anyway
- 60. Incorrect animation states ● Animation in UE3 is done by evaluating a tree – Branches are weight-blended (either replacement or additive blend) – Sequences (raw animations) for whole-skeleton poses – Skeletal controls for fine-tuning of individual bones Source: http://udn.epicgames.com/Three/AnimTreeEditorUserGuide.html
- 61. Incorrect animation states (cont.) ● Prominent case for domain-specific debuggers ● No tools for that in UE3, rolled my own visualizer – Walks the animation tree and dumps active branches – Allows inspection of states, but not transitions – Conventional debugging still required, but greatly narrowed down
- 62. Incorrect animation states (cont.) ● Animation bug “checklist” ● Inspect the animation state in slow motion – Is the correct blending mode used? ● Inspect the AI and cutscene state – Capable of full animation overrides ● Inspect the assets (animation sequences) – Is the root bone correctly oriented? – Is the root bone motion correct? – Are inverse kinematics targets present and correctly placed? – Is the mesh skeleton complete and correct?
- 63. Incorrect animation states (cont.) ● Incorrect blend of reload animation – Cause: bad root bone orientation in animation sequence ● Left hand off the weapon – Cause: left hand inverse kinematics was off – Fix: revise IK state control code ● Left hand incorrectly oriented – Cause: bad IK target marker orientation on weapon mesh
- 64. Viewport stretched when portals are in view
- 65. Viewport stretched when portals are in view ● Graphics debugging is: – Tracing & recording graphics API calls – Replaying the trace – Reviewing the renderer state and resources ● Trace may be somewhat unreadable at first…
- 66. Viewport stretched when portals are… (cont.) ● Traces may be annotated for clarity – Direct3D: ID3DUserDefinedAnnotation – OpenGL: GL_KHR_debug (more info: [GODLEWSKI01])
- 67. Viewport stretched when portals are… (cont.) ● Quick renderer state inspection revealed that viewport dimensions were off – 1024x1024, 1:1 aspect ratio instead of 1280x720, 16:9 – Looks like shadow map resolution? ● Found the latest glViewport() call – Shadow map code indeed ● Why wasn't the viewport updated for main scene rendering?
- 68. Viewport stretched when portals are… (cont.) ● Renderer state changes are expensive – New state needs to be validated – Modern graphics APIs are asynchronous – State reading may requrie synchronization → stalls ● Cache the current renderer state to avoid redundant calls – Cache ↔ state divergence → bugs!
- 69. Viewport stretched when portals are… (cont.) ● Cause: cache ↔ state divergence – Difference between Direct3D and OpenGL: viewport dimensions as part of render target state, or global state ● Fix: tie viewport dimensions to render target in the cache
- 70. Black artifacts
- 71. Black artifacts
- 72. Black artifacts
- 73. Black artifacts
- 74. Black artifacts
- 75. Black artifacts ● First thing to do is to inspect the state ● Nothing suspicious found, turned to shaders ● On OpenGL 4.2+, shaders could be debugged in NSight… ● OpenGL 2.1, so had to resort to early returns from shader with debug colours – Shader equivalent of debug logs, a.k.a. ”Your Mum's Debugger” ● ”Shotgun debugging” with is*() functions – isnan(), isinf() ● isnan() returned true!
- 76. Black artifacts (cont.) ● Cause: undefined behaviour in NVIDIA's pow() implementation – Results are undefined if x < 0. Results are undefined if x = 0 and y <= 0. [GLSL120] – Undefined means the implementation is free to do whatever ● NVIDIA returns QNaN the Barbarian (displayed as black, poisoning all involved calculations) ● Other vendors usually return 0 ● Fix: for all pow() calls, clamp either: – Arguments to their proper ranges – Output to [0; ∞)
- 77. Mysterious crash
- 78. Mysterious crash ● Game in content lock (feature freeze) for a while ● Playstation 3 port nearly done ● Crash ~3-5 frames after entering a specific room ● First report included a perfectly normal callstack but no obvious reason ● QA reassigned to another task, could not pursue more ● Concluded it must've been an OOM crash
- 79. Mysterious crash (cont.) ● Bug comes back, albeit with wildly different callstack ● Asked QA to reproduce mutliple times, including other platforms – No crashes on X360 & Windows! ● Totally different callstack each time ● Confusion! – OOM? Even in 512 MB developer mode (256 MB in retail units)? – Bad content? – Console OS bug? – Audio thread? – ???
- 80. Mysterious crash (cont.) ● Reviewed a larger sample of callstacks ● Most ended in dlmalloc's integrity checks – Assertions triggered upon allocations and frees ● Memory stomping…? Could it be…?
- 81. Mysterious crash (cont.) ● Started researching memory debugging – No tools provided by Sony ● Tried using debug allocators (dmalloc et al.) – Most use the concept of memory fences – Difficult to hook up to UE3 malloc Regular allocation Fenced allocation malloc
- 82. Mysterious crash (cont.) ● Found and integrated a community-developed tool, Heap Inspector [VANDERBEEK14] – Memory analyzer – Focused on consumption and usage patterns monitoring – Records callstacks for allocations and frees ● Several reproduction attempts revealed a correlation – Crash adress – Construction of a specific class ● Gotcha!
- 83. Mysterious crash (cont.) // class declaration class Crasher extends ActorComponent; var int DummyArray[1024]; // in ammo consumption code Crash = new class'Crasher'; Comp = new class'ActorComponent' (Crash);
- 84. Mysterious crash (cont.) ● Cause: buffer overflow vulnerability in UnrealScript VM – No manifestation on X360 & Windows due to larger allocation alignment (8 vs 16 bytes) ● Fix: make copy-construction fail when template is a subclassed object ● I wish I had Valgrind! [GODLEWSKI02]
- 85. Agenda ● How is gamedev different? ● Bug species ● Case studies ● Conclusions
- 86. Takeaway ● Time is of the essence! ● Always on a tight schedule ● Constantly in motion – Temporal visualization is key – Custom, domain-specific tools ● Complex and indeterministic – Difficult to automate testing – Wide knowledge required ● Prone to bugs outside the code – Custom, domain-specific tools, again
- 87. Takeaway (cont.) ● Rendering is a whole separate beast – Absolutely custom tools in isolation from the rest of the game – Still far from ideal usability ● Good to know your machine down to the metal ● Good memory debugging tools make a world's difference ● You are never safe, not even in managed languages!
- 88. Questions? @ leszek.godlewski@theastronauts.com t @TheIneQuation K www.inequation.org
- 89. Thank you!
- 90. References ● SINILO12 – Sinilo, M. ”Coding in a debugger” [link] ● GODLEWSKI01 – Godlewski, L. ”OpenGL (ES) debugging” [link] ● GLSL120 – Kessenich, J. ”The OpenGL® Shading Language”, Language Version: 1.20, Document Revision: 8, p. 57 [link] ● VANDERBEEK14 – van der Beek, J. ”Heap Inspector” [link] ● GODLEWSKI02 – Godlewski, L. ”Advanced Linux Game Programming” [link]
Notas do Editor
- So here&apos;s the agenda for our meeting today. It appears a bit lacklustre, but the devil is in the details. I&apos;d like to lay down just a little bit of context before diving into the case studies.
- This is a schematic for a typical game game loop. The algorithm in real application may be slightly different, but that&apos;s the gist of it: until the exit condition is met, there is an endless game state update and display loop. The “drawing” here is an umbrella term for generating all of the game output – video, audio, tactile/haptic feedback, everything. &lt;number&gt;
- Apologies for the S-bomb, but that&apos;s just calling it out for what it is. ;) This is a simplified structural diagram of a modern game. As you can see, there are many components that need to do their own jobs in that short time.
- Programmers almost never work without interaction with non-technical team-mates. Usually you&apos;ll be working in tightly woven teams of people of diverse skills.
- Especially on consoles. PCs are obviously more lax. Not unlike embedded development. Most prominent reason for extensive use of native code.
- Primary balance between different factors.
- Emergent gameplay – unscripted interaction of game systems with interesting results Crysis series Watch_Dogs Nemesis in Shadow of Mordor How do you define correctness? People have different tastes, difficult to measure fun Even in technical parts, e.g. rendering – per-pixel comparison makes little sense (hardware – rasterization/derivation patterns, shader compilers, approximations etc.) What really caused the regression? CPU/GPU/IO load Configuration New content
- There is a prevailing attitude that special problems need special, not generic solutions. Of course there is a lot of ego, elitist bullshit and the NIH (Not Invented Here) syndrome in this, so take it with a pinch of salt. ;)
- Not sure what this game is, just found this GIF on the net, gave me a chuckle. ;) The categorization here is nothing formal or canonical, it&apos;s just something that helps me make sense of my work. &lt;number&gt;
- Memory access violations Dangling or uninitialized pointers Use-after-free Threading errors Race conditions Deadlocks
- Especially in linear algebra. Also calculus, if you&apos;re doing sufficiently advanced stuff like physics or graphics programming.
- This is a huge thing in game development. Most things in games are in some kind of motion and so their state changes over time. Again, this entity update code is just a schematic; specific implementations may differ in details, but that&apos;s the gist of it. If you access the entity #i + n (where n &gt; 0) from entity #i&apos;s update, that&apos;s reading the old state from the previous frame. That&apos;s why update order matters.
- Incorrect render state Resource (texture, shader, buffer) binding Blending mode Tests (depth, stencil, alpha, backface cull) Shader code bugs Regular programming Precision Low-precision texture formats Transcendental function approximations
- Scripts may be code or graphs, like the illustration shows With assets I mean not just in substantively incorrect things like a human body with bad proportions, a dog with 5 legs etc. They may be technically invalid, illegal: Mesh vertices Texture formats Texture colour space Texture compression formats Animations Root bone orientation Intended blend mode (replace/add/modulate) Compression Custom, domain-specific debugging comes into play here
- It&apos;s almost never just one thing; you&apos;ll get content bugs that manifest (or not) because of rendering glitches, temporal bugs coming out of bad maths, threading issues resulting from bad content updated in a specific order or all the other ways around, all the while sprinkled with generic programming errors.
- Most of the cases I&apos;m going to discuss are illustrated by material captured by Quantic Lab, who were the QA contractors for those projects. Not all bugs have been documented so well, though, so some don&apos;t have fancy screens or videos. I have vowed to better document the bugs of my next games, however, so that blooper rolls can be put together. I had a hard time trying to come up with an order to show the cases in. Areas are not a good choice because of overlap; chronology doesn&apos;t make much sense either; so I just tried to guesstimate how interesting they would appear to the audience and sorted them on that key in ascending order. &lt;number&gt;
- &lt;number&gt;
- &lt;number&gt;
- &lt;number&gt;
- Just a screenshot of the game mode, not an illustration of the case. The mode was programmed mostly by an intern and had some custom death logic so that living players can help and revive those who died. When a player really died (i.e. could not be revived anymore), they would be teleported into a seemingly random point in the map. Some investigation proved it was always very close to the spawn point where their pawn was originally inserted into the game. &lt;number&gt;
- Again, just to explain the context in a whim. This will actually be a box in UE3, it&apos;s a contrived illustration.
- This one is the real thing, though.
- Actor actually does drive some part of the physics state, but that&apos;s outside the scope of this talk.
- &lt;number&gt;
- &lt;number&gt;
- Strangely enough, the debug logs worked!
- I noticed that the printouts in bugged runs of the level had an inverted update order. Mechanisms to enforce update order are beyond the scope of this talk.
- &lt;number&gt;
- Upon inspection, cooking logs included warnings about “out-of-bounds vertices”. This led me to the mesh vertex compression code. The savings are around 50% because while most vertex attributes get cut by half, some may not be floats, and also you need to store another single float for the sphere radius. This isn&apos;t literally the code I worked on, but a close paraphrase.
- What was the cause? Hell if I know. Re-exporting fixed it. Magic!
- Now this one isn&apos;t something that we get all the time, just once in a while, but it proves it&apos;s good to know the low-level stuff. &lt;number&gt;
- Maciek has some more examples of that technique on his blog, so if this is interesting, go check it out.
- Of course, this only lasts until the end of the debugging session. See [SINILO12] for more cool examples.
- Painkiller HD was a remake. I was tasked with closely replicating the player movement physics in the new engine. The first versions of the code didn&apos;t perform well in tight spaces, getting the player stuck… &lt;number&gt;
- …or moving without player&apos;s intent. &lt;number&gt;
- Before I discuss the debugging of this, let me lay out some context. Different game engines meant, for instance, different measurements of distance and velocity. It may sound absurd if you think of physics as a strict science, but different physics simulation engines will behave absolutely differently with the same data. That is beyond the scope of this talk, though.
- Since a capsule has half-spheres on both of its ends, you can probably imagine how it performs more smoothly than a cylinder on uneven terrain.
- Every sample that would not hit the ground would contribute to a force vector. This vector would then be applied when the player does not express intent to move. This would emulate the effects of the capsule closely enough to achieve a similar player movement experience to the original. I&apos;ve found no clever way to debug movement code, just reproduce the bug, pause the game and break in debugger. It&apos;s easier to do when time is slowed down.
- Going back to the first of the observed bugs: when debugged – got stuck, then just put a breakpoint in the movement function – the situation turned out to be as in the illustration. Blue lines are surface normals; green is the player&apos;s collision primitive. Both surface normals are “unwalkable” (i.e. too steep) by the configurable minimum normal Z (45 degrees by default). Vanilla UE3 would switch to “falling” state when no walkable floor was found, and since our “falling” logic was already modified to disallow air manouvering and the surfaces were otherwise level, the player would be stuck.
- Simple visualization – coloured line segments reflecting the traces made when sampling – aptly revealed that undersampling was to blame. Red arrows indicate “ground found”, green ones – “no ground”. Black arrow is “slip force”. You can probably see how this situation ends in oscillation. Simply adding more sample points (rings) fixed it.
- There was one level in the entire game where the blood effects would periodically disappear for a second or two. &lt;number&gt;
- They are simply applied from left to right. Left-most node is the raw scene texture, right-most node&apos;s output gets drawn onto the screen.
- This is not the real deal but a contrived illustration, as I can&apos;t really get ahold of the sources right now, but it&apos;s very close to what I got. It would just sit there in the corner, using the reflection mechanism to inspect the effects&apos; parameters and drawing progress bars where it makes sense. The text is in red because my testing showed that surprisingly, the colour red was used very sparingly in the game&apos;s pallette.
- The tool made it evident that when blood disappeared, it was something overriding the entire PP chain. The way we&apos;ve been managing PP FX, we had no use case for the option anyway.
- Pay attention to that character from the waist up. &lt;number&gt;
- The screenshot has been brightened to make sure it&apos;s legible on a projector, the colours are not as blown out in the actual game. &lt;number&gt;
- Ditto. &lt;number&gt;
- Blend weights and other parameters that made sense that way were drawn with progress bars. The most useful part of this is that you could observe how the state changes over time. Running the game in slow motion helped a lot, too.
- Animation state usually gave a good hint on where to start debugging, like locomotion state. Also, NPCs in cover were another programmer&apos;s responsibility as a whole, so this step served as a global “if” when assigning bug tickets. Additive blending of regular sequences and vice versa look horribly wrong (long neck syndrome anyone?). Root bone is a virtual bone that is the root to the entire skeleton hierarchy. Bad root bone transforms can miserably break otherwise perfectly correct animations.
- This one is slightly newer, it only happened this year. :) There&apos;s a section in Darksiders where portals can be used for fast travelling and puzzle solving. &lt;number&gt;
- The exact debugging workflow varies depending on the tool. Especially the granularity: some capture entire API call streams, some capture single frames. The principle remains the same, though.
- And this is basically what you do. Shameless plug about OpenGL debugging!
- Some context again.
- In OpenGL, viewport dimensions are a parameter of the global state. In Direct3D, it&apos;s part of render target state. So if you switch render targets in OpenGL, viewport dimensions don&apos;t change, but in Direct3D they do. This port used to assume the Direct3D way, which was not correct.
- But that wasn&apos;t so bad, graphics bugs can get much worse. This was on the PS3. &lt;number&gt;
- This as well. &lt;number&gt;
- This was NVIDIA on Linux. &lt;number&gt;
- NVIDIA on Linux. &lt;number&gt;
- So that you don&apos;t think only PKHD was this broken – here&apos;s Darksiders on NVIDIA on Linux. &lt;number&gt;
- NVIDIA also provided the PS3 GPU. Poisoning in this context means that all calculations in which the NaN is an operand result in NaN. Undefined is “do whatever”, including formatting setting your machine on fire. Other vendors – AFAIK. Former is mathematically correct, latter is simpler and cheaper on most GPUs. Thankfully, max(0.0, NaN) returns 0.0. It&apos;s defined as if (x &lt; y) {return y;} else {return x;} and any comparisons involving NaN always fail.
- This place in PKHD gave me the creeps for weeks. We had a very mysterious crash upon entering that cellar door. &lt;number&gt;
- And I thought no more of it.
- I asked colleagues for opinion, but we could only take shots in the dark.
- After the initial confusion blew over, I figured it would be best to review the hard data we had, so I took to a larger sample of callstacks. Could it be memory stomping? A real bug for real men? ;)
- I&apos;ve never done memory debugging before, so I needed research. Memory fences work by reserving additional area between allocation that should somehow raise errors when that area is overwritten. I tried hooking several debug allocators up (dmalloc, Electric Fence etc.) but they either couldn&apos;t be easily integrated because of the PS3 OS API, were too memory-hungry (e.g. reserved an entire page for each and every allocation) and so on.
- Heap Inspector isn&apos;t really the best tool for the job, but it was easy to integrate and captured callstacks reasonably fast.
- This is not the literal code, but a paraphrase. It&apos;s UnrealScript, so a managed language in a VM. Can you spot the culprit? There&apos;s a complete class declaration at the top. At the bottom, we first instantiate the Crasher class, and then copy-construct an object of its base class with the new Crasher object as template.
- Our culprit class only had 2 or 3 new fields; the Windows/X360 allocation alignment was large enough to hide the problem. Shameless plug again – GDCE slides discuss Valgrind.
- Personally, I miss three things in graphics debugging: Speed – waiting for state resolution drives me nuts “Touched pixels/vertices” counter – to warn about the dreaded “my draw call has no effect” bug Full source-level shader code debugging Make a game large enough and you will eventually get down to disassembly.
- &lt;number&gt;