Investigative Debugging

“Evidently,” said I, “[the] assistant counts for a good deal in this mystery.”
“Not him.”
“What then?”
“The knees of his trousers.”
“And what did you see?”
“What I expected to see.”
Sir Arthur Conan Doyle, The Red-Headed League

Here’s a confession: I’ve been a Sherlock Holmes fan since I was a little kid. I’ve probably read all the Holmes books three times or more.

Sherlock Holmes solved mysteries by thinking very, very hard about the facts at hand, forming a few possible theories that fit all of them, and then performing the minimum of tests or investigation to verify which of those models was right.

Holmes’ approach to sleuthing was very effective, for two reasons:

He had an unparalleled analytical mind, along with encyclopedic knowledge of all topics related to criminal enterprise in London; and
He was a fictional protagonist.

Unfortunately, when I’m debugging my code or trying to understand how a library works, I too often fall into the trap of thinking that I’m Sherlock Holmes. I imagine that I can lean back in my armchair, steeple my fingers, and think very hard about the problem until I come up with a hypothesis which I can then quickly test.

I’ve wasted quite a lot of time this way.

Real detectives gather data

Real-world criminal investigation doesn’t look anything like this. Outside the pages of Victorian fiction, police investigators collect mountains of raw data, and sift through it. They also make use of their special privileges as government agents to dig into criminal databases, subpoena private records, and tap phones.

We can debug programs this way too. Investigative debugging is the practice of using the full diagnostic arsenal at our disposal, along with our special privileges as machine operators, to understand code by surveilling it. Not through cleverness, but through sheer overwhelming empirical evidence. We get to the bottom of the behavior of a dodgy program or library by reading its tax returns, rifling through its mail, and tracing its phone calls.

In this course we’ll look at surveillance tools like:

Containerization to isolate a program’s file dependencies and communications
Language tracing libraries
strace,ltrace, and dtrace to track system and library calls
LD_PRELOAD to inject your own fake functions into programs
Browser dev tools to peer into JavaScript HTTP calls
mitmproxy and Postman to intercept and interpret HTTP traffic
Wireshark to examine network traffic at a lower level
lsof to track file and port usage
ldd and strings for peering into executables

This course is a Graceful.Dev Garden Path, which means it is a guided pathway through a collection of (mostly) self-contained topics. Its status is: growing, meaning that there is a significant amount of material here already, and more is expected in the future.