Characterization tests, one dimension at a time
The term “characterization tests” comes from Michael Feathers’ Working Effectively with Legacy Code.
Refactoring without tests isn’t refactoring. It’s rewriting with optimism. “Looks equivalent” is not “is equivalent.” The subtle behavioral change hiding in an untested path finds its way to production six months later, git blame points at a commit that moved some files around, and nobody can tell what the intended behavior even was.
Capture behavior before you touch it. A normal test says “this is what the function should do.” A characterization test says “this is what the function actually does right now” — you’re not endorsing the behavior, you’re fencing it off so you notice the moment it changes.
Say you inherit this with zero tests and zero context:
func maskAccountNumber(input string) string {
if input == "" {
return ""
}
digits := onlyDigits(input)
switch len(digits) {
case 16:
return "**** **** **** " + digits[12:]
case 10:
return strings.Repeat("*", 6) + digits[6:]
default:
return input // ??? unclear why
}
}
You don’t know if that default case is intentional or a bug. Before touching it, run the function against inputs and write down whatever it actually returns:
// Characterization tests — not a spec, a snapshot
func TestMaskAccountNumber(t *testing.T) {
tests := []struct{ name, input, want string }{
{"masks a 16-digit card number", "4111111111111234", "**** **** **** 1234"},
{"masks a 10-digit account number", "1234567890", "******7890"},
{"returns empty string for empty input", "", ""},
{"returns input unchanged for an unrecognized length — current behavior, not necessarily correct", "12345", "12345"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := maskAccountNumber(tt.input); got != tt.want {
t.Errorf("maskAccountNumber(%q) = %q, want %q", tt.input, got, tt.want)
}
})
}
}
// Now refactor with confidence
That last case is the whole point. You’re not asserting that returning the raw digits for a 5-digit input is right — you’re asserting that it’s what happens today. Now you can restructure the switch, change how digits get extracted, whatever the refactor calls for, and the moment behavior shifts, a test goes red instead of the change slipping through silently.
This still holds even when you do want to change behavior — that’s not a contradiction, it’s the point. Say the real fix is “an unrecognized length should return an error, not silently return the raw input.” You don’t skip the characterization test to get there; you write it first, confirm it’s red for the reason you expect, and then update that one case as part of the behavior-change commit — separate from any refactor commit. The failing test forces a decision point: is this red because you broke something by accident, or because you meant to change it? Either way you find out immediately, not from a bug report six months later. And the diff on that one case — asserting "12345" unchanged becoming asserting an error — becomes the changelog: anyone reading git blame sees exactly which commit changed the behavior, and why, instead of inferring it from a pile of moved files.
The second discipline matters just as much: never mix structure and behavior in the same commit. Rename, move, and fix a bug, and that’s three commits, not one. Each dimension reviews independently, reverts independently, and points to a clear cause when something breaks. A diff that mixes file moves with logic changes can’t be reviewed with any confidence — the reviewer can’t tell which parts were incidental and which were intentional.
Raise the safety net: “Before we restructure this, can we add a few tests that document current behavior? That way anything that changes unintentionally gets caught, and the next person gets a description of what this is supposed to do.”
Raise the mixed commit: “This PR is hard to review because structural changes are mixed in with logic changes. Could we split it, one commit for the rename or move, one for the actual change? The second diff will be much easier to reason about.”
Hypothesis-driven debugging
Debugging by coincidence is making changes until the symptom disappears. The fix “works” in the sense that the error message goes away. Nobody knows the root cause. Two months later the same bug resurfaces in a different shape, and nobody knows why, because nobody ever understood why the first time.
Form a falsifiable hypothesis before you touch code instead: “I believe the bug is in X because Y. If I’m right, evidence Z should be there. If I’m wrong, evidence W should be there instead.” It’s the scientific method applied to debugging, and most engineers skip it in favor of “let me try this and see.”
Being wrong is fine — it’s expected, and it’s still progress, as long as the hypothesis was specific enough to be wrong about something. “I believe the timeout is caused by the retry loop, so I’d expect the logs to show three attempts before the 500” is falsifiable: check the logs, see two attempts, and you’ve just ruled out the retry loop entirely, with evidence, in the time it took to grep a log file. “Maybe it’s something with retries” isn’t falsifiable — no evidence confirms or rules it out, so being wrong about it teaches you nothing and you’re back where you started. The goal was never to be right on the first guess. It’s to make every guess narrow the search.
The longer you can stay outside the code, the sharper the hypothesis you bring back into it. A ticket says “customers report transfers aren’t completing.” The rabbit-hole move is to open TransferService and start reading. The narrowing move is to ask cross-cutting questions that don’t require a single file open yet:
- Every customer, or one segment?
- Every transfer type — wire, ACH, internal — or one specifically?
- Every channel this can be triggered from — mobile, web, branch — or one?
- Started at a specific time, or always been intermittent?
Answer those from a support dashboard or a logs query, not from source, and “transfers aren’t completing” might turn into “wire transfers only, only for accounts opened after last Tuesday’s migration, ACH and internal transfers are fine.” That’s not a vague area to start reading in anymore — it’s a hypothesis: something about the new-account wire path changed on the migration date. You’ve ruled out 90% of TransferService before opening it, because whatever’s wrong has to be true of wires and false of ACH, true of new accounts and false of old ones.
This is harder to stick to now than it used to be, not easier. It takes real discipline to run those cross-cutting checks first when you could paste the ticket straight into an AI coding assistant and have it start proposing diffs in TransferService within seconds. The assistant can only reason from what it can see, and what it can see is code — so it reasons from the code outward, same as you do if you follow it in immediately. The shortcut feels like speed. What it’s actually doing is skipping the ten minutes of narrowing that would have told you which twenty lines of that file actually matter, in favor of confidently investigating all of it.
The minimal reproduction is the most important artifact this discipline produces. Building one forces you to separate what’s essential from what’s incidental:
// Minimal reproduction becomes the regression test
func TestFormatAccountHolderName_NilProfile(t *testing.T) {
customer := Customer{ID: 1, Profile: nil}
got := formatAccountHolderName(customer)
if got != "Unknown" {
t.Errorf("formatAccountHolderName(nil profile) = %q, want %q", got, "Unknown") // was panicking
}
}
The practical test for whether you’re done: can you explain why the fix works? “I’m not sure, but it stopped happening” means the bug isn’t fixed, it’s masked. The same conditions will produce it again. You just haven’t seen it yet.
Cargo cult programming lands you in the same place by a different road: copying a pattern without understanding the assumptions it depends on. The code works in the environment it was copied from. Change the environment — different auth model, different transport, different concurrency assumptions — and it fails, and nobody understands why, because nobody understood the original.
// Copied auth pattern — looks correct, but the CSRF protection assumes
// the API is same-origin. If this wire-transfer endpoint is consumed by
// a mobile app, it's wrong.
router.Use(cookieAuth())
router.Use(csrfProtection()) // copied from a tutorial without checking assumptions
Raise it, coincidence: “Before we close this out, can we make sure we understand why the fix works? If we can’t explain the mechanism, we can’t tell solved from masked. What regression test would have caught the original bug?”
Raise it, cargo cult: “Help me understand why this pattern works here. I want to make sure the assumptions it relies on still hold in our context before we build on top of it.”