Characterization tests, one dimension at a time
The term “characterization tests” comes from Michael Feathers’ Working Effectively with Legacy Code.
Refactoring without tests isn’t refactoring. It’s rewriting with optimism. “Looks equivalent” is not “is equivalent.” The subtle behavioral change hiding in an untested path finds its way to production six months later, git blame points at a commit that moved some files around, and nobody can tell what the intended behavior even was.
Capture behavior before you touch it. A normal test says “this is what the function should do.” A characterization test says “this is what the function actually does right now” — you’re not endorsing the behavior, you’re fencing it off so you notice the moment it changes.
Say you inherit this with zero tests and zero context:
function maskAccountNumber(input: string): string {
if (input === "") return "";
const digits = onlyDigits(input);
switch (digits.length) {
case 16:
return "**** **** **** " + digits.slice(12);
case 10:
return "*".repeat(6) + digits.slice(6);
default:
return input; // ??? unclear why
}
}
You don’t know if that default case is intentional or a bug. Before touching it, run the function against inputs and write down whatever it actually returns:
// Characterization tests — not a spec, a snapshot
describe("maskAccountNumber", () => {
test.each([
["masks a 16-digit card number", "4111111111111234", "**** **** **** 1234"],
["masks a 10-digit account number", "1234567890", "******7890"],
["returns empty string for empty input", "", ""],
["returns input unchanged for an unrecognized length — current behavior, not necessarily correct", "12345", "12345"],
])("%s", (_name, input, want) => {
expect(maskAccountNumber(input)).toBe(want);
});
});
// Now refactor with confidence
That last case is the whole point. You’re not asserting that returning the raw digits for a 5-digit input is right — you’re asserting that it’s what happens today. Now you can restructure the switch, change how digits get extracted, whatever the refactor calls for, and the moment behavior shifts, a test goes red instead of the change slipping through silently.
This still holds even when you do want to change behavior — that’s not a contradiction, it’s the point. Say the real fix is “an unrecognized length should return an error, not silently return the raw input.” You don’t skip the characterization test to get there; you write it first, confirm it’s red for the reason you expect, and then update that one case as part of the behavior-change commit — separate from any refactor commit. The failing test forces a decision point: is this red because you broke something by accident, or because you meant to change it? Either way you find out immediately, not from a bug report six months later. And the diff on that one case — asserting "12345" unchanged becoming asserting an error — becomes the changelog: anyone reading git blame sees exactly which commit changed the behavior, and why, instead of inferring it from a pile of moved files.
The second discipline matters just as much: never mix structure and behavior in the same commit. Rename, move, and fix a bug, and that’s three commits, not one. Each dimension reviews independently, reverts independently, and points to a clear cause when something breaks. A diff that mixes file moves with logic changes can’t be reviewed with any confidence — the reviewer can’t tell which parts were incidental and which were intentional.
Raise the safety net: “Before we restructure this, can we add a few tests that document current behavior? That way anything that changes unintentionally gets caught, and the next person gets a description of what this is supposed to do.”
Raise the mixed commit: “This PR is hard to review because structural changes are mixed in with logic changes. Could we split it, one commit for the rename or move, one for the actual change? The second diff will be much easier to reason about.”
When the function has forty callers
Characterization tests work cleanly on maskAccountNumber because it has one obvious seam: call it, check what comes back. Real legacy code is rarely that isolated — the function you need to change is called from forty places across six modules, with calling conventions that have drifted over three years of different authors adding their own special cases. “Understand it before you touch it” starts to sound like it’s asking for something that takes a month.
It isn’t, because you don’t need to understand every caller — you need to understand the contract at the seam you’re touching, and forty callers almost never means forty distinct behaviors:
Group callers by calling pattern, not by call site. Use your editor’s “Find Usages” or “Call Hierarchy” (built into VS Code, WebStorm) to enumerate every caller, then look at what each one actually passes. Forty call sites often collapse into three or four distinct shapes — settle(tx, false, undefined) shows up at a dozen sites for the same reason. One characterization test per shape, not per caller.
Pin the contract at the seam, not the whole call graph. You don’t need a characterization test for every caller’s caller. You need one for every distinct way the function you’re changing gets used — which is the group from the step above, not the raw count of places it’s called from.
For genuinely tangled fan-in, change the shared function last, not first. This is branch-by-abstraction: introduce the new behavior behind a new function name or a flag, migrate callers to it one at a time, verified by the pinned characterization tests as you go, and only remove the old path once every caller has moved. That way a mistake in one caller’s migration doesn’t take down the other thirty-nine.
None of this changes with an AI coding assistant in the loop — an agent can write the characterization tests for each calling pattern faster than you can by hand, but it still needs you to have grouped the callers and named the contract first. “What must stay true here” is a question about the domain, not the code, and that question doesn’t go away just because the diff gets generated faster. On the coverage question specifically: most editors with a coverage extension can show inline gutters after running vitest --coverage or jest --coverage — worth checking before assuming a function is a total unknown. A green gutter next to the function doesn’t tell you the right things were tested, but a red one tells you nothing was, which is useful information before you start.
Cargo cult programming lands you in the same place by a different road: copying a pattern without understanding the assumptions it depends on. The code works in the environment it was copied from. Change the environment — different auth model, different transport, different concurrency assumptions — and it fails, and nobody understands why, because nobody understood the original.
// Copied auth pattern — looks correct, but the CSRF protection assumes
// the API is same-origin. If this wire-transfer endpoint is consumed by
// a mobile app, it's wrong.
router.use(cookieAuth());
router.use(csrfProtection()); // copied from a tutorial without checking assumptions
Raise it, coincidence: “Before we close this out, can we make sure we understand why the fix works? If we can’t explain the mechanism, we can’t tell solved from masked. What regression test would have caught the original bug?”
Raise it, cargo cult: “Help me understand why this pattern works here. I want to make sure the assumptions it relies on still hold in our context before we build on top of it.”