Does your code work? Probably not. The libraries you're using probably don't work either. If you're lucky, the OS does, but even then you'll probably find something wrong if you look hard enough. Debugging is the reason that the last 20% of shipping a product usually accounts for 80% of the time. And yet, there are a million blog posts and talks about writing code, but very few about figuring out why it doesn't work right once you have. So, how do you find bugs? In this talk I'll explore a set of tools and techniques that have helped me diagnose defects in everything from php code to malloc implementations. One time I even used this strategy to diagnose an outage in a codebase I'd never seen that was written in a language I barely knew and a framework I'd never heard of - in less than 5 minutes. You'll walk away with this talk with everything you need to learn how to debug anything. Video: https://www.youtube.com/watch?v=VV7b7fs4VI8

1. How to Debug
Anything
@jamesgolick
well ok, not anything, but most stuff on
unixy operating systems that have
the tooling i’m going to talk
about today

2. https://packagecloud.io
“better than google”
- PC Weekly

3. Everything is
Terrible

4. Everything is
Broken

5. “Correct Code”™

6. “If you want to deploy high
quality software that performs,
you should expect to fix bugs at
every level.”
- me

7. “I don’t understand
how this is possible.”
- every programmer ever

8. 0. php
a blind debugging session

9. the website is down

10. what we have to work with
• The source code. (nope)
• Knowledge of the system. (nope)
• Familiarity with the programming language.
(nope)
• SSH Access. (yup)

11. logging in the real world
(often useless)

12. #cool

13. find a pid

14. sudo strace -ff -s 2048 -p 22935

15. write(1, "hin", 3) = 3
function name arguments return value
how to read strace output

16. man 2 write
learn more about your favorite system calls

17. {sa_family=AF_INET, sin_port=htons(50318), sin_addr=inet_addr("192.168.212.2")}, [16]) = 12
fcntl(12, F_GETFD) = 0
fcntl(12, F_SETFD, FD_CLOEXEC) = 0
getsockname(12, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("192.168.212.182")}, [16]) = 0
fcntl(12, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(12, F_SETFL, O_RDWR|O_NONBLOCK) = 0
read(12, "GET / HTTP/1.1rnHost: localhost:8181rnConnection: keep-alivernCache-Control: max-age=0rnAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8rnUser-Agent:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36rnAccept-Encoding: gzip,deflate,sdchrnAccept-Language: en-US,en;q=0.8rnCookie:
_jsuid=3899596137;
_packages_session=d2NrM3RZMUJXRE8zcFB0aXNGVk83Ny9lRDR3Y09uSVNoRUcrREV0VnF2UjFxRjk1NjAyUzZ5ZG81M1JyczRzUU10ZTBqMXI5QkJXZzFqZnM1RUNmdEdGYmN2eG92SUsvU24wOWhJSlhNZzYrQXdYN2tMYnRZaEhWN3ArbEpiZVpMUWNjWHNRWHc2VjkwQzZ2S
0Y4aGlLeks3MmhoTXBXN2NRWUEwbGFFekpENHdveCtTNXl1MllDUTFzUzZMSU5WZlRqUlQ1aXB2bWVsZDVGVFE1Tlp0UT09LS1vYWdoMk9mZHUvS3U5OWpoME1ZY3pBPT0%3D--8e4ac5c1aebe1e9226063c3d2b83b4176535377arnrn", 8000) = 792
stat("/var/www/", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
stat("/var/www/index.php", {st_mode=S_IFREG|0664, st_size=447, ...}) = 0
setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={60, 0}}, NULL) = 0
rt_sigaction(SIGPROF, {0x7f8a898930c0, [PROF], SA_RESTORER|SA_RESTART, 0x7f8a8c6d14a0}, {0x7f8a898930c0, [PROF], SA_RESTORER|SA_RESTART, 0x7f8a8c6d14a0}, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [PROF], NULL, 8) = 0
umask(077) = 022
umask(022) = 077
getcwd("/", 4095) = 2
chdir("/var/www") = 0
setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={30, 0}}, NULL) = 0
lstat("/var/www/index.php", {st_mode=S_IFREG|0664, st_size=447, ...}) = 0
open("/var/www/index.php", O_RDONLY) = 13
fstat(13, {st_mode=S_IFREG|0664, st_size=447, ...}) = 0
fstat(13, {st_mode=S_IFREG|0664, st_size=447, ...}) = 0
fstat(13, {st_mode=S_IFREG|0664, st_size=447, ...}) = 0
fstat(13, {st_mode=S_IFREG|0664, st_size=447, ...}) = 0
mmap(NULL, 447, PROT_READ, MAP_SHARED, 13, 0) = 0x7f8a8d34f000
munmap(0x7f8a8d34f000, 447) = 0
close(13) = 0
getcwd("/var/www", 4096) = 9
lstat("/var/www/./oh-fuck.php", 0x7fff11342620) = -1 ENOENT (No such file or directory)
lstat("/usr/share/php/oh-fuck.php", 0x7fff11342620) = -1 ENOENT (No such file or directory)
lstat("/usr/share/pear/oh-fuck.php", 0x7fff11342620) = -1 ENOENT (No such file or directory)
lstat("/var/www/oh-fuck.php", 0x7fff11342620) = -1 ENOENT (No such file or directory)
getcwd("/var/www", 4096) = 9
lstat("/var/www/./oh-fuck.php", 0x7fff11342580) = -1 ENOENT (No such file or directory)
lstat("/usr/share/php/oh-fuck.php", 0x7fff11342580) = -1 ENOENT (No such file or directory)
lstat("/usr/share/pear/oh-fuck.php", 0x7fff11342580) = -1 ENOENT (No such file or directory)
lstat("/var/www/oh-fuck.php", 0x7fff11342580) = -1 ENOENT (No such file or directory)
getcwd("/var/www", 4096) = 9
lstat("/var/www/oh-fuck.php", 0x7fff113446e0) = -1 ENOENT (No such file or directory)
open("/var/www/oh-fuck.php", O_RDONLY) = -1 ENOENT (No such file or directory)
chdir("/") = 0
umask(022) = 022
open("/dev/urandom", O_RDONLY) = 13
read(13, "33260300377gK222d", 8) = 8
close(13) = 0
open("/dev/urandom", O_RDONLY) = 13
read(13, "427417x3516336260", 8) = 8
close(13) = 0
open("/dev/urandom", O_RDONLY) = 13
read(13, "&M330225-P340345", 8) = 8
close(13) = 0
setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8a8d34e000
writev(12, [{"HTTP/1.0 500 Internal Server ErrorrnDate: Sat, 14 Jun 2014 17:40:32 GMTrnServer: Apache/2.2.22 (Ubuntu)rnX-Powered-By: PHP/5.3.10-1ubuntu3.11rnVary: Accept-EncodingrnContent-Encoding:
gziprnContent-Length: 20rnConnection: closernContent-Type: text/htmlrnrn", 256}, {"37213100000003", 10}, {"30", 2}, {"00000000", 8}], 4) = 276
write(7, "192.168.212.2 - - [14/Jun/2014:17:40:32 +0000] "GET / HTTP/1.1" 500 276 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/
537.36"n", 199) = 199
times({tms_utime=6, tms_stime=56, tms_cutime=0, tms_cstime=0}) = 1718165756
shutdown(12, 1 /* send */) = 0
poll([{fd=12, events=POLLIN}], 1, 2000) = 1 ([{fd=12, revents=POLLIN|POLLHUP}])
read(12, "", 512) = 0
close(12) = 0
read(4, 0x7fff113490cf, 1) = -1 EAGAIN (Resource temporarily unavailable)
ETOOMUCHOUTPUT

18. writev(12, [{"HTTP/1.0 500 Internal
Server Err"..., 256},
{"37213100000003", 10},
{"30", 2}, {"00000000",
8}], 4) = 276
find failure
always work backwards

19. open("/var/www/db.in.php", O_RDONLY)
=
-1 ENOENT (No such file or directory)
find the cause
hopefully.

20. open("/var/www/index.php", O_RDONLY)
=
13
find the offender

21. prove your hypothesis

22. find the offender

23. fix the bug!

24. #cool

25. “I don’t understand
how this is possible.”
- every programmer ever

26. 0. Forget everything you think you know.

27. 1. Get a third party opinion.

28. third parties
i have known and loved
source: http://www.brendangregg.com/linuxperf.html

29. 1. apt
building a cloud for packages is hard

30. sudo apt-get update

31. sudo strace -ff apt-get update

32. write(1, "Ign http://
192.168.212.136:3000 trusty
Releasen", 62) = 62
find failure
always work backwards

33. read(6, "400 URI FailurenURI:
https://packagecloud-repositories-
dev2.s3.amazonaws.com/1/1/ubuntu/
dists/trusty/Release?
AWSAccessKeyId=AKIAILW54TIPGLUGWOYA&Si
gnature=s/c0fzVQhxpBPbpyTIzCxAfo/
8g=&Expires=1402837136nMessage: Bad
header line nn", 64000) = 230
find the cause

34. confirm your hypothesis

35. #cool

36. apt-get source apt

37. read(6, "400 URI FailurenURI: https://
packagecloud-repositories-
dev2.s3.amazonaws.com/1/1/ubuntu/dists/
trusty/Release?
AWSAccessKeyId=AKIAILW54TIPGLUGWOYA&Sig
nature=s/c0fzVQhxpBPbpyTIzCxAfo/
8g=&Expires=1402837136nMessage: Bad
header line nn", 64000) = 230
locate a hook

38. locate a hook

39. stare at the code

40. confirm your hypothesis

41. > Content-Type:

42. > Content-Type: text/
plain

43. #cool

44. 2. Locate the correct source code.

45. 3. Identify a hard-coded string to grep for.

46. 4. Stare at the code until it makes sense.

47. 5. Fix whatever is broken.

48. How to Debug Anything
!
!
0. Forget everything you think you know.
1. Get a third party opinion.
2. Locate the correct source code.
3. Identify a hard-coded string to grep for.
4. Stare at the code until it makes sense.
5. Fix whatever is broken.

49. Questions?
@jamesgolick
https://packagecloud.io