Measure first, know your bottlenecks
Performance intuition is wrong more often than it’s right. The thing you’re certain is the bottleneck almost never turns out to be. Profile first and let the data decide where effort goes. There’s no reliable shortcut around this.
Most web application code spends its time waiting, not computing. A request handler calls the database, waits, calls another service, waits, maybe hits a cache, waits again — and the CPU sits idle through nearly all of it. That’s why the common bottleneck categories skew so heavily toward I/O, roughly in this order of frequency:
- Database queries, especially N+1 patterns
- Network round trips
- Missing or incorrect caching
- CPU, which is actually rare in a typical web app
The N+1 pattern earns special attention because it’s both the most common performance bug and the most invisible one in development. Ten rows in a test database, loading all transactions with their accounts, costs eleven queries and runs in milliseconds. A hundred thousand rows in production costs a hundred thousand and one queries and grinds the server to a halt.
// N+1: one query per transaction
transactions, err := db.Transactions.FindAll(ctx)
for _, tx := range transactions {
tx.Account, err = db.Accounts.FindByID(ctx, tx.AccountID) // N queries
}
// Fixed: 2 queries total
transactions, err := db.Transactions.FindAll(ctx)
accounts, err := db.Accounts.FindByIDs(ctx, accountIDsOf(transactions))
accountsByID := keyByID(accounts)
for _, tx := range transactions {
tx.Account = accountsByID[tx.AccountID]
}
This is “push fors down” in practice: instead of calling a single-item function N times inside an application loop, push the collection to the function that touches the database and let it batch. The loop disappears, and the database does one round trip regardless of how many rows come back.
Look at tail latency, not averages. p50 is what the median user feels. p99 is your worst-case users. An endpoint sitting at 50ms p50 and 5000ms p99 has a real problem an average will never show you. Watching mean latency and calling performance “fine” is the observability version of premature optimization: you’ve convinced yourself you’re measuring something when you’re not.
Premature optimization is the mirror image: spending time making something fast that was never slow, usually at the cost of readability or correctness. Amdahl’s Law is unforgiving here. Optimize a non-bottleneck and you get a negligible improvement no matter how well you do it.
The profile is supposed to sometimes disagree with you — that’s the entire reason to run it. If you were certain the database was the bottleneck and the flame graph shows 80% of the time in JSON serialization instead, that’s not a wasted profiling session. That’s the session that stopped you from spending a week optimizing queries that were never the problem. The cost of being wrong about the bottleneck is a five-minute profile. The cost of being wrong about the bottleneck without profiling is however long the “optimization” takes, plus however long it takes someone to notice it didn’t help.
Spot it, N+1: query logs in development showing the same query repeated dozens of times, response times that scale linearly with the size of the returned collection, await sitting inside a for loop or a .map() over database results.
Spot it, premature optimization: “I rewrote this to be more performant” with no before/after numbers, caching added before anyone measured whether the uncached version was actually too slow.
Raise it, N+1: “I think we’ve got an N+1 here, a database query per item in the loop. Fine on small data sets, but it’ll get slower linearly as the list grows. There’s a straightforward batch pattern for this, want me to show you?”
Raise it, premature optimization: “What did we measure that told us this was too slow? Want to make sure we’re optimizing the right thing. Could we profile first and target the actual bottleneck?”