Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/design/decisions.md
Original file line number Diff line number Diff line change
Expand Up @@ -787,6 +787,7 @@ Declined:
- Giving the peel rule a predicate the compound rule fails (2026-09-02, #382 option 1) — narrowing `fix(cjk-comma-honorific-peel)`'s `name_regex` to the honorific-bearing shapes, whose stated effect in #382 was that the pair "becomes order-independent and the original contract holds again". Declined because that effect is UNREACHABLE, and measured rather than argued. The two rules ship a BYTE-IDENTICAL `name_regex` (verified 2026-09-02 by comparing the two strings), and `_rule_reach` computes each rule's names from its OWN pattern, so narrowing the peel rule's regex narrows the peel rule's reach and nothing else: the compound rule goes on reaching every one of the names. The general form, which no regex edit escapes — where the earlier rule's `fields` are a SUBSET of the later one's, order-independence requires the LATER rule to stop reaching the name, and an edit to the earlier rule's own regex cannot cause that. Here the peel rule's `{given, suffix}` is a strict subset of the compound rule's `{family, given, suffix, title}`, so every diff the peel rule admits the compound rule admits too. Implemented to check it: narrowing the peel regex to a `GLUED_HONORIFICS` alternation keeps all seven of the co-matched names whose 1.4.0 diff is exactly `{given, suffix}`, and swapping the two rules in the NARROWED ledger still reattributes all seven. So the pair keeps the arrangement the arc settled on, and that is the answer rather than a cost trade: it is narrow-first, the declaration-free default, so it appears in none of the eleven contests `order_contests` reports and owes no `precedes_narrower` block, and all seven names are pinned by name in `_CROSS_RULE_WINNERS` (23 corpus names reach both regexes; the seven are the ones #375's reorder mutation moves). The cost stands as a second reason and not the first: the narrowing would hand-copy more honorific vocabulary into the pattern and grow the `_HONORIFIC_SOURCES` sync-roster surface (mechanisms.md#CURATED-VOCABULARY-ALTERNATION's second half). Option 2's convention plus a check on the wide-first exceptions to it is what the arc took, and option 3 it REFUSED on a false premise (the finding above). One thing this does not settle, and the ledger comment on the peel rule overstates it: `fields` separates the two only for the UNION rows whose diff includes `family`; for those seven it is file order that decides.
- Scoping the check to contract-tier contests only (2026-09-02) — five of the eleven are contested only over radar names, so this would have cut the file's exemptions by nearly half. Declined because `_CROSS_RULE_WINNERS` already pins radar names — measured, most of the names it pins are radar-tier (22 of 33 today) — so the repo would be inconsistent with itself about whether a radar contest matters. Whether the radar-only CJK comma rules still earn their place after #488's demotion is a real question and is filed as #495; it is a question about those rules, not about the check.
- A shared name-population function for both callers (2026-09-02, the spec's own first sketch) — see the two-populations decision above. `main()` and the unit guard must read different populations, so one function would break `--corpus`; they agree by a docstring that names the function it mirrors instead.
- Deleting the rules whose explained names are all radar (2026-09-02, [#495](https://github.com/derek73/python-nameparser/issues/495), closed as not planned) — filed off #488's demotion on the reasoning that a rule nothing can demand is a rule to remove, and refused on what deleting one actually does. Measured at baseline 1.4.0, 19 of the 72 explaining rules explain only radar names, between them 48 of the 352 intentional diffs — and that set is a SUPERSET of what the issue proposed deleting, not a different set. Three of the five rules #495 tabulated are in it — `fix(#296) a credential-only comma string reads a name and its postnominal`, `fix(cjk-comma-compound)`, and the roman-numeral `fix(suffix-routing)` — and they are exactly the three its option 2 named as having zero contract reach. The other two are outside it for the reason that disqualified them as candidates there too: they explain contract names, fourteen of them for `fix(cjk-glued-honorific-peel)` and `'Smith Jr.'` (corpus_rules.jsonl) for the jr `fix(suffix-routing)`. So the criterion reaches everything the issue wanted gone and more, and what refuses it is the fate of those 48 diffs. Measured by driving `classify()` over each one with its own rule removed: 25 report UNCLASSIFIED and 23 are claimed by ANOTHER rule (26 and 22 with all 19 deleted at once). For nearly half the population, deleting does not leave the diff watched-but-unexplained — it leaves it explained, by a rule whose comment is about something else. Nine of `fix(cjk-comma-compound)`'s eleven land on `fix(cjk-glued-honorific-peel)`; all seven of `fix(cjk-comma-honorific-peel)`'s land on `fix(cjk-comma-compound)`; `'Carod i'` lands on the roman-numeral `fix(suffix-routing)`, and `'Bob Jones, author'` on `fix(comma-precomma-family)`. That is the rule-broader-than-the-diff absorption this file already names twice — `fix(#424)` at decisions.md#H1, going on classifying a diff that shrank underneath it while the fields-only catch-all's heading stepped 14 -> 17 with nothing announcing either, and the fields-only arc's "a fields-only rule silently falsifies OTHER comments" — reached this time on purpose rather than by accident, which is not an improvement. Applied, it deletes `fix(#342)` and `fix(#397)`, the two NOT WANTED rules whose whole job is classifying a reading nobody wants so the gate stays green while the bug is open, and which carry the delete-when-fixed instruction the dormancy check enforces once the fix lands (the fields-only arc above). The cost there is NOT a red gate, and a first draft of this bullet said it was: both names sit in corpus_issues.jsonl, which is radar, and on a radar corpus an unmatched diff prints UNCLASSIFIED (radar) while the run keeps exiting 0 — UNEXPLAINED is the contract tier's word (tools/differential/README.md). Measured with both rules gone, the run reports `unexplained: 0; radar unclassified: 1` and exits 0. The one is `'Aishwarya Rai'`; `'Carod i'` is not reported at all, the numeral rule having taken it — so one of the two open bugs loses its marker not to silence but to a wrong label. Verified end to end against a ledger with `fix(#397)` removed: exit 0, 352 intentional diffs still, no UNCLASSIFIED, no EXPLAINED NOTHING, and `'Carod i'` printed under `fix(suffix-routing) a two-token name ending in a roman numeral`. It also deletes `feat(#273)`, `feat(#269) Arabic بن prefix chains onto family (non-Latin new-recognition)` and `fix(#360) ste moved into the never-given particles with mc` — records of new recognition behavior and of a vocabulary move — and the last two have to be named to the RULE rather than to the issue, as the ledger holds sibling rules under both strings. The `feat(#269)` sibling is the sharp one: `feat(#269) non-Latin titles/conjunctions recognized` does not qualify, explaining the contract-tier `'Хосе И Мария Сантос'` (corpus_rules.jsonl), so a delete by issue prefix takes both and sends that name UNEXPLAINED — measured, and a red gate. The criterion that does justify deleting a rule is that it explains NOTHING, which `dormant` already names and the dormancy check already enforces; a rule explaining eleven radar names is not dormant. What the radar tier is FOR answers the rest: decisions.md#cjk-comma-demotion puts it as giving the differential "somewhere to WATCH a name without promising it", and a rule explaining a radar name IS the watching. The five exemptions citing #495 are rewritten in the same diff rather than left standing: four of them put it as a LIVE question — the two under `fix(#296) a credential-only comma string`, the one on the `fix(cjk-comma-compound)` block, and the one on the `'田中さん II'` numeral contest — and the fifth, on the jr contest, called neither rule a candidate in the present tense. Each records the answer now and keeps every measurement it carried. One of those measurements was wrong and is corrected with them: `fix(comma-family) lone post-comma piece routes to suffix/title, not first` explains eight names of which FIVE are contract, not six — the six having shipped in #499. It is not a mis-map: six is what tiering by FILE MEMBERSHIP gives from a CORRECT `_CORPUS_TIERS`, and the single name it gets wrong is `'John Smith, Dr.'`, the third RECOMPUTE trap below. #496's decline was argued inside the `fix(cjk-comma-compound)` one and stands without it: the twin rule was refused because nothing can demand it, which is untouched by what #495 answered. RECOMPUTE: do not join names to files by hand. `_CORPUS_TIERS` maps FILES, the report prints names with no file, and 24 of the 352 explained names sit in files of BOTH tiers, so a hand join has to reproduce `main()`'s contract-first dedup AND its baseline-minimum skip to get them right. Let the run do it: call `compare.main()` unchanged under two wrappers — one around `_run_worker`, whose `entries` argument is the final list with `tier` and `file` already stamped on every entry, and one around `classify()`, recording (name, order, diff, issue) for each of its 352 calls — then reclassify offline from those records against `rules` minus whichever rule is under test, having first confirmed the wrapped run reproduced the shipped one (352 intentional, 0 unexplained, 0 radar unclassified). Parse no printed repr by stripping quotes: three of the 352 names do not survive it — `"'Smitty' Dr. Jones"`, `"'Smitty' Jones Jr."`, and the spelling of محمد بن سلمان wrapped in RIGHT-TO-LEFT MARKs — and dropping them silently is what produced the wrong count of 20 on the first pass, the first of the three being `fix(#410)`'s only contract name. `ast.literal_eval` reads the printout correctly. The ten-name truncation is NOT the hazard it looks: contract files load first, so any contract name a rule explains sits in the printed prefix, and tiering from the first ten alone gives the same 19. THIRD, and the only one of the three that changes an answer on this tree: a name's tier, in any claim scoped to a BASELINE, is a property of the ENTRY that run compared and not of the files the string appears in. The two part where a string sits in files of both tiers AND its contract copy carries a shape tag, because the dedup key is `(name, order)` — a shape declaring an order leaves TWO entries rather than one contract winner, and the baseline-minimum skip can then drop the contract one. Measured, 2 of those 24 names carry a shape tag and exactly 1 bites at 1.4.0: `'John Smith, Dr.'` is in corpus_issues.jsonl (radar, no shape) and corpus_shapes.jsonl (contract, shape 4, FAMILY_FIRST, min_baseline 2.0.0); both entries survive the dedup, which the 2.0.0 run shows by comparing both, and at 1.4.0 the shape-4 one is among the seven skipped, leaving the radar entry as the only comparison. `'Beethoven, Ludwig van'`'s shape 2 is order-None with a 1.4.0 minimum, so the dedup collapses it and contract wins at every baseline. File-membership tiering gets exactly 1 of the 352 wrong, and that one is what `fix(comma-family)`'s six-versus-five turned on; `entry["tier"]` gets it right for free. The radar-only count of 19 is the same under either tiering. To exercise a ledger edit, point `_allowlist_for` at a scratch file rather than copying the tree — the corpus glob resolves from `HERE` and `_check_tree` refuses a nameparser outside `REPO_ROOT`, so a copy elsewhere cannot run at all.

### comma-suffix-arc — #291/#296/#316 (2026-07-26 → 2026-08-01)

Expand Down
71 changes: 50 additions & 21 deletions tools/differential/expected_since_1.4.0.toml
Original file line number Diff line number Diff line change
Expand Up @@ -431,9 +431,24 @@ corpus_issues.jsonl alone -- radar since #468's tier split, and
untouched by #488, which created corpus_cjk_tolerated.jsonl and holds
neither name -- so neither can demand a rule any more and nothing
fatal turns on this pair. The tier reaches the NAMES, not both rules:
measured, this rule explains two names and both are radar, which makes
it a candidate for #495; the routing rule explains eight of which six
are contract, and is not one."""
measured, this rule explains two names and both are radar, which made
it a candidate for #495; the routing rule explains eight of which FIVE
are contract, and was not one. (Six shipped here in #499 and is what
tiering by FILE MEMBERSHIP gives, with a correct _CORPUS_TIERS: it
counts 'John Smith, Dr.' contract because corpus_shapes.jsonl holds a
copy. The dedup is on (name, order), and that copy carries shape 4,
whose FAMILY_FIRST order makes it a SECOND entry rather than the
dedup winner -- verified at baseline 2.0.0, where the run compares
both -- and shape 4's min_baseline of 2.0.0 puts it among the seven
this baseline skips. So the only entry a 1.4.0 run compares for that
string is the radar one from corpus_issues.jsonl. Tier here is a
property of the ENTRY the run compared, not of the files the string
sits in.) #495 is now closed as not planned and
the radar-only rules keep their place, so both halves of that
measurement are facts about REACH and no longer a question about
either rule's life -- docs/design/decisions.md#differential-ledger's
Declined entry on #495 has the population and why deleting one
mislabels rather than merely loses."""

[[change.precedes_narrower]]
issue = "fix(comma-precomma-family) pre-comma run reads as family, not given"
Expand All @@ -449,10 +464,12 @@ that moves the title and the suffix in the same breath. A string that
is nothing but credentials is not a name plus a comma. Radar tier
since #468 on both names -- corpus_issues.jsonl, which #488 did not
touch -- so this pair is watched rather than enforced, and again only
the wider rule is #495's business.
the wider rule was ever #495's business.
Measured, the precomma rule explains seven names of which three are
contract ('Berg, abdul vd', 'Smith, Dr.', 'Smith, de Mesnil Jean'), so
it outlives the radar tier whatever #495 decides about this one."""
it outlives the radar tier by itself. #495 is closed as not planned,
so the wider rule outlives it too: nothing radar-only is being
deleted, and this declaration turns on neither answer."""

[[change]]
issue = "fix(#325) a split credential followed by another suffix after a one-word family comma reads as suffixes"
Expand Down Expand Up @@ -1320,12 +1337,15 @@ peel took the honorific off it -- so for those three the rule's own
label is wider than the name is. The rule that would describe them,
fix(cjk-comma-honorific-peel) above, covers this shape for a
POST-comma given name and has no family-side twin. Writing one was
weighed as #496 and DECLINED: the three are radar (the paragraph
below), so no gate can demand the rule, and #495 asks whether the
radar-only rules already here still earn their place -- fewer rules
for this corner, not a second one. So holding them here is the least
wrong of the readings available, and this is the entry that says so
out loud rather than letting the rule name imply otherwise.
weighed as #496 and DECLINED on one reason alone: the three are radar
(the paragraph below), so no gate can demand the rule, and a rule
nothing can demand is not one to add. #495 asked the converse --
whether the radar-only rules ALREADY here still earn their place --
and closed as not planned, so they stay. That answer leaves the #496
decline exactly where it was: it turns on nothing being able to demand
a NEW rule, which is not what #495 settled. So holding them here is
the least wrong of the readings available, and this is the entry that
says so out loud rather than letting the rule name imply otherwise.

All seventeen contested names are radar tier, by two warrants and
not one: sixteen sit in corpus_cjk_tolerated.jsonl, demoted there by
Expand All @@ -1338,12 +1358,17 @@ unconditionally, though -- a [[never]] exclusion outranks the tier
(see _CORPUS_TIERS) and none refuses these. That is a fact about the
NAMES and it does not carry to the rules. Measured by deleting each
rule and re-classifying the corpus -- without fix(cjk-comma-compound)
the run still reports 0 unexplained, so it really is radar-only and is
the candidate #495 weighs; without the peel rule TWELVE contract-tier
names go UNEXPLAINED ('Andersonさん', '王先生' and '김민준씨' among
them) and the run fails. Fourteen of the seventeen names the peel rule
explains are contract, which is why #495 already records it as a rule
to keep rather than a deletion candidate."""
the run still reports 0 unexplained, so it really is radar-only and it
was the candidate #495 weighed; without the peel rule TWELVE
contract-tier names go UNEXPLAINED ('Andersonさん', '王先生' and
'김민준씨' among them) and the run fails. Fourteen of the seventeen
names the peel rule explains are contract, which is why #495 recorded
it as a rule to keep rather than a deletion candidate. #495 closed as
not planned, so the compound rule keeps its place as well -- and the
reason is this contest read from the other side: measured, deleting
the compound rule does not leave its eleven diffs UNCLASSIFIED, it
hands NINE of them to the peel rule, whose comment is about a
different shape. Two are left unclassified."""

[[change]]
issue = "fix(cjk-glued-honorific-peel) glued honorific peels into suffix"
Expand Down Expand Up @@ -1394,9 +1419,11 @@ LATENT: measured, the diff is {family, given, suffix} (1.4 read first
`given` is outside the numeral rule's {family, suffix} at any
position. _CROSS_RULE_WINNERS pins the name here. '田中さん II' is
radar tier since #488, so nothing in the gate turns on this contest.
The demotion candidate #495 weighs is the NUMERAL rule -- measured,
The demotion candidate #495 weighed was the NUMERAL rule -- measured,
both names it explains ('John V', 'Mohamad X') are radar -- and not
this one, which explains fourteen contract-tier names."""
this one, which explains fourteen contract-tier names. #495 closed as
not planned, so the numeral rule stays; the measurement stands as the
statement about its reach that it always was."""

[[change.precedes_narrower]]
issue = "fix(suffix-routing) a two-token name ending in the suffix word jr keeps it in `suffix`"
Expand All @@ -1416,9 +1443,11 @@ LATENT, the diff being {family, given, suffix} against the jr rule's
{family, suffix}. _CROSS_RULE_WINNERS pins the name here, on a shape
measured against the wheel -- the corroboration its structural twin
'田中さん II' did not have until #382 re-measured it. The NAME is radar
tier since #488, so nothing fatal turns on it, but neither RULE is a
tier since #488, so nothing fatal turns on it, but neither RULE was a
#495 candidate: the jr rule explains the contract-tier 'Smith Jr.',
and this one fourteen contract names."""
and this one fourteen contract names. #495 closed as not planned, so
nothing radar-only is being deleted either, and this pair outlives the
question from both directions."""

[[change]]
issue = "fix(cjk-honorific-suffix) postnominal honorifics recognized, compounding with the CJK order flip"
Expand Down