The second 100%

Coverage is the number we invented so we could stop thinking.

I spent years being proud of coverage numbers. I want that time back.

Here’s a session check, close to code I’ve actually shipped:

func (s *Store) CanUseSession(t Token, now time.Time) bool {
	return t.SignatureValid() &&
	       now.Before(t.ExpiresAt) &&
	       !s.nonces.Seen(t.Nonce)
}

And here’s the test I’ve written for it a hundred times:

func TestCanUseSession(t *testing.T) {
	store := NewStore()
	if !store.CanUseSession(validToken(), time.Now()) {
		t.Fatal("valid session rejected")
	}
}

100% line coverage. Every condition executed. The report is green.

Now delete the nonce check. Test passes. Delete the expiration check. Test passes. Replace the whole body with return true. Test passes. This test can’t tell the difference between my function and no function. All it proves is that good input produces a good answer — which is also what code with no security checks does.

Branch coverage is the same lie with more steps. Add the case everyone adds — a tampered token gets rejected — and both branches are hit, the report looks even better, and two of the three security checks can still be deleted without a single test noticing. That’s what our coverage metrics measure: whether code ran. Not whether any of it was ever forced to matter. And we hold serious meetings about whether the threshold should be 80% or 90%.

Avionics solved this twice

In the 80s, avionics certification hit exactly this problem, and their answer had two parts. We took neither. I’ll get to the second part later, because it took watching this kind of failure to understand it.

Part one is MC/DC — Modified Condition/Decision Coverage. Ugly name, simple idea: every condition in a decision must independently flip the result, or you haven’t tested the decision.

For my session check, that means four cases:

signature valid   expired   nonce seen   result
yes               no        no           allow
no                no        no           deny    // signature matters
yes               yes       no           deny    // expiration matters
yes               no        yes          deny    // nonce matters

So let’s do it properly this time:

func TestCanUseSession(t *testing.T) {
	store := NewStore()
	now := time.Now()

	if !store.CanUseSession(validToken(), now) {
		t.Fatal("valid session rejected")
	}
	if store.CanUseSession(tamperedToken(), now) {
		t.Fatal("bad signature accepted")
	}
	if store.CanUseSession(expiredToken(), now) {
		t.Fatal("expired token accepted")
	}

	replayed := validToken()
	store.CanUseSession(replayed, now) // first use: allowed
	if store.CanUseSession(replayed, now) {
		t.Fatal("replayed token accepted")
	}
}

100% MC/DC. Every condition forced to matter. Delete the expiration check and a test fails. Flip the nonce logic and a test fails. This is a good test — I’d approve it in review today. It felt like the ceiling of rigor.

Remember that feeling.

The bug that still ships

That test suite is good enough to make the function feel done. The code is simplified, but this is the exact class of bug I’ve watched ship: in-memory state standing in for a guarantee that needed to survive the process. Here’s the incident this design is one deploy away from.

s.nonces is an in-memory map. Every deploy restarts the process, the map comes up empty, and every token captured before the deploy becomes replayable after it. Deploy daily, and replay protection — the thing with a dedicated, passing, well-designed test — effectively doesn’t exist for a window after every single release.

Which test failed? None. Which test could fail? None. Look at the replay test again: it creates a store, uses a token, replays it against the same store, in the same process, in the same millisecond. Within that world, the test is correct. The bug doesn’t live in that world. It lives in the question the test never asked: what remembers the nonces, and for how long?

That’s the part that stings. MC/DC did its job perfectly — every condition in the decision was proven to matter. But it can only interrogate the logic I wrote. It can’t interrogate the logic I didn’t write. The bug wasn’t in a branch. It was in a question:

• What happens to seen nonces across a restart?

• Across two instances behind a load balancer?

• Whose clock is now, and how far do our servers drift?

• What does the client learn from a rejection — does the error message tell an attacker which check failed?

None of these are coverage questions. A suite could hit 100% of anything a tool can measure and never touch one of them. Nobody writes them down, so nobody tests them, so production tests them for us.

A missing test is annoying. A missing question is dangerous.

I eventually got a word for those questions: obligations.

An obligation isn’t a test case. It’s a class of behavior the change must account for. “Replayed token rejected” was a test case, and we had it. “Replay protection survives restarts and failover” was an obligation, and it existed nowhere — not in the ticket, not in the tests, not in review, only in the gap between what the ticket said and what production required.

This is why I now think of coverage as two different numbers. The first 100% is structural: did the code run, was the logic exercised. Line coverage, branch coverage, MC/DC — they all live here, and MC/DC is the honest end of it. The second 100% is semantic: did we cover what the code is responsible for — the requirement, the obligations, the assumptions, the risks. Every tool we have measures the first. Every incident I can remember came from the second.

Not every obligation becomes a test. Some are handled elsewhere, some are accepted risks, some need fuzzing or a runtime assertion or one honest sentence in a doc. Clock skew might be “accepted, ±30 seconds, here’s why” — that’s fine. What’s not fine is the obligation silently not existing, so that the difference between considered and accepted and never thought about is invisible.

And these aren’t exotic cases. Restarts, failover, cache staleness, error messages that say too much — they’re Tuesday. If nobody writes them down, they live in memory: the author’s, the reviewer’s, maybe nobody’s.

Tests are evidence, not truth

This is part two of what avionics knew — the part I skipped earlier. In that world, a test that doesn’t trace to a requirement is evidence of nothing. Every test exists to support a claim; every requirement must have evidence behind it. They understood forty years ago that executing code proves nothing by itself. The industry kept the percentage and threw away both ideas that gave it meaning.

For most of my career, this was my whole model of testing:

code -> tests -> CI -> confidence

Tests were code that exercised other code. More tests, more coverage, more confidence. The replay failure above fits this model perfectly and the model never blinks — the code is exercised, CI is green, confidence is high, and the bug ships anyway.

The model I’d steal from avionics, minus the paperwork:

requirement -> obligations -> evidence -> confidence

Same code, same tests, same CI — but now they sit inside an argument. The requirement says what must be true. The obligations say what must be considered. The tests are demoted from “the thing itself” to what they always actually were: evidence for specific claims.

Here’s my session check under that model:

Requirement:
A session token is accepted only if its signature is valid,
it has not expired, and it has never been used before.

Obligations:                              Evidence:
- forged signature rejected               TestCanUseSession (MC/DC)
- expired token rejected                  TestCanUseSession (MC/DC)
- replayed token rejected, same process   TestCanUseSession (MC/DC)
- replay survives restart                 MISSING — nonce store is in-memory
- replay survives failover                MISSING
- clock skew between servers              accepted risk: ±30s, documented
- malformed token rejected safely         fuzz target, nightly
- rejection reveals which check failed    MISSING

Same function, same tests. But now the failure isn’t a surprise buried in an architecture diagram — it’s a row that says MISSING, visible in review, before the deploy. The test suite didn’t get better. The *argument* got better, and the argument is what caught it.

This changes what review means. “Do we have tests?” becomes “which claim does this test prove?” “What’s the coverage number?” becomes “which obligations have no evidence?” It’s much harder to fake an argument than a number.

Code does not remember why

Engineers love saying code is the source of truth, because code runs and docs rot. Half right. Code is the source of truth for what the system does. It’s a terrible source of truth for what the system was supposed to do.

Code can’t tell you whether a behaviour is intentional or accidental. Whether the weird branch exists because of a customer, a migration, or a 3am incident nobody wrote down. Whether clock skew was accepted or never considered. So every change starts with archaeology: reconstruct intent from the diff, stale tickets, Slack threads, and whoever hasn’t left yet.

This was already failing when humans wrote all the code. AI makes it fail faster, because generating plausible code is now cheap and verifying intent is not. And when the code and the tests come from the same prompt — the same incomplete understanding — they agree with each other perfectly. The replay bug above, generated fluently in seconds: in-memory nonce store, matching in-process replay test, green CI, high coverage, missing obligation still missing. That’s not verification. That’s a model grading its own homework.

Humans always did this too. AI just industrialised the polished misunderstanding.

What the second 100% looks like

Not MC/DC everywhere. Not a requirements document for a button color. Start where being wrong is expensive: auth, billing, deletion, migrations, tenant boundaries, anything that moves money or can’t be un-shipped.

Write the requirement in plain English. List the obligations. Attach evidence — and let MISSING be visible. That’s the entire practice. The obligations table above is not a big artifact; it’s twenty lines that would expose the failure before it became an incident, and it answers questions a coverage report can’t: Which obligations are handled? Which are accepted risks — accepted by a person, on the record? Which decisions need MC/DC instead of a happy path? What goes stale if the requirement changes?

The first 100% covers the structure of the code. The second covers its responsibility. One asks whether the code ran. The other asks whether anyone understood what the code had to protect.

Why I’m building Proof

This bothered me long enough that I’m building a tool around it. Requirements that survive the ticket closing. Obligations that live in the repo instead of someone’s head. Tests that know which claim they support. Code, tests, and requirements that invalidate each other when they drift.

Not compliance theatre. Not another number to gamify. An evidence chain that answers the only question that matters in a review: Why do we trust this change?

Coverage can’t answer that. It can only tell you which lines ran.

The second 100% is everything your coverage tool cannot see — and after enough incidents, you learn that’s exactly where the bugs live.