Recorded diff shapes become self-checking, and a false cost claim comes out

[derek73]

The recorded diff shapes that test_the_recorded_rule_still_wins_each_contested_name had been feeding classify() move into tools/differential/compare.py as _RECORDED_DIFFS, and the run now verifies them. Four of the 35 rows turned out to be wrong on arrival. Along the way, the cost claim that decided a neighbouring decline turned out never to have been measured.

Closes #497. docs/design/decisions.md gains the seventh ledger arc entry, which is the distilled record; this body is the change, not a second copy of it.

What changed

_CROSS_RULE_WINNERS (tests/v2/test_ledger_guards.py) recorded a (name, shape) -> winning rule mapping and handed the shape to classify() as an input. Nothing ever compared that shape to a shape a run makes, so a wrong one that still routed to the same rule stayed green forever — the half only a run can do, because main() has measured every name's real diff by the time it asks and the unit suite installs no wheel at all.

So the shapes move to where both halves exist. _CROSS_RULE_WINNERS keeps what it is for — which rule wins a contested name — and reads the shape from compare._RECORDED_DIFFS. The pair (name, shape) -> winner reconstructs the old dict exactly for every surviving row: no winner moved and no shape changed (checked by parsing both versions and reconstructing, not by reading the diff).

One roster, two checks, two placements, and the asymmetry is the decision rather than where the code fit:

• The absent-name half — a recorded row naming a string no corpus holds any more — refuses pre-worker, beside vacant, because it reads the ledger and the loaded names and nothing the worker produces. It reads the pre-skip corpus_names, because "does any corpus still hold this name?" is a question about the files. Reading the post-skip list instead would tell a contributor to delete a live row; measured at 1.4.0 the two lists differ by three names.
• The measured half prints a MOVED SHAPE block and feeds the exit code, like over_declared_rules. It reads the post-skip list, because "was this name compared?" is a question about the run. It cannot raise mid-report: a raise there would precede dormancy, OVER-DECLARED, UNEXPLAINED and the radar block, so a stale roster row would hide an unexplained diff — the gate's primary output. Measured, a --corpus tools/differential/corpus.jsonl run at 1.4.0 prints 62 EXPLAINED NOTHING lines, and the corrupted copy below prints all 62 and the MOVED SHAPE block; a raise would have printed neither.

A row with no measured shape has two causes — the parser stopped moving the name, or it is compared under a declared order alone — and the two hand the check byte-identical arguments, so the message names both and asserts neither. The README's MOVED SHAPE bullet is corrected for the same reason: it kept "the parser changed what the name does", which is false in the case that motivated the check.

The finding: the worker pass was never multi-minute

"The multi-minute worker pass" entered on 2026-08-05 in 7767ba2, at one site, with no measurement recorded beside it, and was copied outward until eight sites carried it (git grep -n -i minute 514a2a1): four in tools/differential/compare.py (one of them spelling it "costs minutes", so count the claim and not the wording), three in tests/v2/test_differential.py, and a Declined: bullet in docs/design/decisions.md that gave it as "the whole of the reason" for declining precise per-name contest detection.

Measured 2026-09-03 over the whole corpus:

baseline 1.4.0   0.34s wall     worker pass 0.10s
baseline 2.0.0   0.46s          worker pass 0.24s
baseline 2.1.0   0.47s          worker pass 0.26s
baseline 2.2.0   0.55s          worker pass 0.29s
all four back to back            1.97s

The worker pass is a fraction of a second and a whole gate run is seconds, where the phrase said minutes. Read the class, not the digits. (Timing trap, recorded with the figures: wrap compare._run_worker in a timer and call main() — calling _run_worker directly on the loaded entries aborts at a 1.4.0 baseline, since the seven order-bearing shape-4/5 entries have to be dropped by the baseline-minimum skip first.)

Three of the eight sites were making an ordering argument — a refusal before the worker beats one after it — and that argument stands; it is restated on what is actually true, that a late refusal lands below the run's own published baseline: header for a comparison it is about to disown. The Declined bullet keeps its conclusion on coverage, which is true and was the second support all along: a run-time check sees one ledger, for one baseline, and only when somebody invokes the tool, where the static predicate covers a rule from the moment it is written.

Two things transfer past the digits. The claim was never a measurement anyone got wrong — it was a plausible sentence about a subprocess that installs a wheel, and it propagated by copying. And that is how a false number survives a month: an argument with two supports loses one and nothing downstream changes, so no reader ever meets a contradiction.

Removing it was itself hard to do cleanly — review of the commit that removed the phrase found four instances of the very defect that commit exists to remove (figures living verbatim in two dated homes with nothing linking them; a baseline-specific figure quoted at a site that names no baseline; a bullet still selling the static check on the cheapness it had just retired). Fixed in 40c82a7.

What the mechanism caught on arrival

Re-measuring the 35 rows to write the provenance note found four that a run contradicts — two names across the two 2.x ledgers, so two rows each — all four added by #452's work (ee6aaa6, PR #455) under two explicit claims of measurement.

They were deleted rather than corrected, because neither name pins a contest, which is what the roster is for:

• 'Nguyen, Van' diffs at no baseline at all — compared under the default order every time, the tree agreeing with all four wheels — so no run ever asks classify() about it. That is 'Doe,, Jr.''s position, which the 1.4.0 roster gives no row for the same reason.
• 'Jane née and Jones Smith' diffs {family, given, maiden, middle} at 1.4.0/2.0.0/2.1.0 and not at all at 2.2.0, never the {family, maiden, middle} recorded. Its shape was correctable and the rows went anyway: the string is a malformed harvest from corpus_issues.jsonl (radar) whose right parse nobody can state.

The general shape, which is the part worth keeping: a row pinning a race with one runner is never exercised as a contest, so its shape only ever had to agree with itself. At the shape each of these names really produces, exactly one rule admits it. That is why nothing caught them, and it is not a fact about two names.

"Only one rule admits it" is not on its own grounds to delete a row — 13 of the 31 surviving 1.4.0 rows are in that position too and stay, because the shapes they pin are shapes runs actually make. Deleting a row removes a pin, not a name; no corpus changed.

Both 2.x sections stay as explicit empty mappings — the completeness guards require it — and their comments now say emptiness is a position, not an oversight, and say what it is not: measured, 5 of the 247 diffs at 2.0.0 and 1 of the 155 at 2.1.0 do move a shape two or more rules admit, and nobody has argued one of those boundaries yet. #501 asks whether that should change now those six are measured rather than unexamined, and the comments cite it.

Why a shape may be pinned where a count may not

This is what keeps the roster affordable, and it is the line between this and #452's declined explains = N. A diff shape is one name under two parser versions, so adding a name to a corpus cannot move it; an explained-name count moves on every corpus edit. Shapes drift only when the parser changes those names — so the drift and the reason to re-read the roster are the same event, which is exactly what a recorded number usually fails to be.

The prose sweep, and what its own scan got wrong

#451's criterion sorts the stale claims into three treatments, chosen by what the count is doing rather than by how wrong it is: a past-tense count that is the finding stays; a count standing beside the enumeration it counts loses the digit, because the list is self-checking and the digit is the only part that can rot; a live exhaustiveness claim with nothing recomputing it is regrounded on the property it was reaching for. Corrected examples: a widening described as claiming "644 of the 654 unclassified corpus names" where the same expression claims 994 of 1008 today; "all five corpora" where six corpus*.jsonl are on disk and a run globs them all; _MUST_NOT_MATCH's "Six such widenings" over a list of four; "all 31 the regex claims" where it claims 32.

The scan that scoped the sweep was wrong three ways, and it is the scan rather than the three lines it missed that the next sweep will reuse:

1. Two of its 19 machine-readable hits were false positives — a version number, and a hypothesis rather than a measurement — so the real population was 17.
2. Its file list omitted expected_since_2.2.0.toml, which held a further copy of a claim it was already treating in two other ledgers, so the sweep would have shipped two of three siblings corrected.
3. Its predicate matched only lines beginning with #, which made every docstring invisible to it. That is how _reaches_non_vocabulary's figures — taken against a 751-name corpus, against 1116 distinct names today — survived the first pass. A docstring pass found one further stale digit and confirmed three others correct.

Three of the four were fixed in a follow-up commit rather than the first, because the first sweep identified and skipped them; leaving them would have shipped a sweep that skipped known instances of its own issue.

Verification

uv run pytest -q: 6480 passed, 156 skipped, 5 xfailed. uv run ruff check . and bare uv run mypy (111 source files) clean.

The gate at all four baselines, exit 0 at each:

baseline	corpus	intentional	unexplained	radar unclassified	MOVED SHAPE
1.4.0	1113 names (7 skipped)	352	0	0	none
2.0.0	1120	247	0	0	none
2.1.0	1120	155	0	0	none
2.2.0	1120	14	0	0	none

No rule moved. Parsing all four ledgers at 514a2a1 and at HEAD and comparing structurally — not grepping — the rule count and the positional issue order are identical in each (75 / 66 / 59 / 1), and no name_regex, fields or orders value differs. The ledger changes in this branch are comment-only.

The new check is not vacuous. Corrupting one _RECORDED_DIFFS row in a scratch copy ("Andrews, M.D.": ("given", "suffix") → ("given", "middle")) and re-running at 1.4.0 gives exit 1 and, appended after a report that is otherwise byte-identical to the clean run:

MOVED SHAPE expected_since_1.4.0.toml: 1 recorded diff shape(s) disagree with this
run. Each is a FINDING, not a number to update: the winner pinned beside the shape
in _CROSS_RULE_WINNERS (tests/v2/test_ledger_guards.py) was recorded for the OLD
shape, so read both before editing either.
  'Andrews, M.D.'
    _RECORDED_DIFFS records ['given', 'middle']; this run measured ['given', 'suffix']

Under --corpus tools/differential/corpus.jsonl the same corrupted copy prints all 62 EXPLAINED NOTHING lines and the block, which is the placement the check was moved for.

The unit suite is still hermetic. tests/v2/test_differential.py's header says nothing there spawns uv or the network; proved by running both differential modules with subprocess.Popen, subprocess.run and os.system replaced by a raising stub — 269 passed. (Six tests do call the real _run_worker, under a faked Popen; the loose paraphrase "every test monkeypatches _run_worker" was itself one of the claims this branch corrects.)

Related

• Closes Measured claims in the differential comments go stale unnoticed (a reviewer read "751 names" as fact; the corpus holds 1116) #497.
• Six contested diffs in the 2.x ledgers have no pin, and the empty rosters read as if none existed (MD, PHD goes to whichever rule comes first) #501 — filed from this work: six contested 2.x diffs have no pin and the empty rosters read as if none existed. Cited from the comments that state the position.
• A ledger rule's explained set is printed but never checked, so a diff can shrink underneath it and every run stays green #452 — commented, not closed. Its still-open reach-ceiling half is re-measured there: the ceiling needs no worker at all, and The corpus becomes two tiers, and the contract tier becomes chosen names #482's tier split inverts the ranking it ranks by. At 1.4.0 the two comma rules are the outlier by total reach (288 = 25.8%) while fix(#271/#272/#298) is the outlier by contract reach (80 = 24.5% against their 19.6%) — so a ceiling on total reach names the wrong rule as the biggest hazard.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.66%. Comparing base (514a2a1) to head (b3f03cf).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #502   +/-   ##
=======================================
  Coverage   98.66%   98.66%           
=======================================
  Files          45       45           
  Lines        3219     3219           
=======================================
  Hits         3176     3176           
  Misses         43       43           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.