diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 3718efab..94a77697 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -413,6 +413,8 @@ Declined (ambiguity kinds for script-resolved names, 2026-07-27): Decided 2026-09-08 (was Open: [#316](https://github.com/derek73/python-nameparser/issues/316), what a trailing title-vocabulary word should do while the comma paths disagreed): the trailing slot reads VOCABULARY, not shape (#H5), and the leading inference here stays unconditional — #316's open question 2, a symmetric leading rule that would read "Esq. Smith" as a suffix, is DECLINED. #109 shipped the leading inference on purpose, the family comma already carries the credential case ("Smith, Esq." → suffix, this rule's own Accepted clause), and reversing S2's "a suffix never opens the string" for period-marked words for the sake of one word is the wrong trade. The two comma paths agree now: "John Smith Prof." and "Smith, Prof." both read title Prof. See #H5. +- 2026-09-10 — the shape declines a word carrying a script with no initials (#323); the argument, the halfwidth-kana limit and the measurements are in #cjk-full-stops. + ### H3 — the title run's floor - 2026-09-08 (the #316/#489 bundle) — the leading title run leaves a name word standing, and a post-nominal is not one. Derek's question is what opened it, verbatim: "Why does `Dr King Jr` need to parse `king` differently than `Dr King`? Jr is a recognized suffix, so it could not count as a following name." The mechanism behind the old reading is an ORDER: the leading peel runs before the trailing suffix peel and its only floor was "a title needs a following piece", so it took `Dr King` whole and left `Jr` to be the name — title `Dr King`, family `Jr`, no suffix at all. The floor added here is a second one, asked of the run's last word: where everything behind the run is a suffix piece and that word is not itself suffix vocabulary, the run gives it back. `Dr King Jr` now reads title `Dr`, family `King`, suffix `Jr` — what v1 wanted (#v1-xfail-triage), what rules.md#S2's descriptive note predicted, and what the comma spelling `King, Dr Jr` has always given. @@ -464,7 +466,7 @@ Decided 2026-09-08 (was Open: [#316](https://github.com/derek73/python-nameparse - 2026-07-27 #271 (decision; shipped in 2.1.0 via PR #294) — Korean division ships as a default: the census surname list is closed, hangul is self-selecting (a hangul entry can only match hangul text), and being unsplit is recoverable while a wrong split is not — which is also why an unrecognized name stays whole. The filed proposal (#271, 2026-07-07) asked for OPT-IN segmentation for Korean too, "like all localization"; default-on is the later refinement, and the census/self-selecting argument above is what justified promoting Korean past the blanket opt-in stance. - 2026-07-29 #272 (the ja amendment; shipped in 2.1.0 via PR #297) — Han division is opt-in per language pack because Han text does not identify its language (高橋一郎 under a Chinese list divides wrongly); a pluggable segmenter takes what the vocabulary declines, so pack + segmenter compose mechanically: a listed surname is a dictionary certainty and wins. -- 2026-07-29 (ja amendment §1a) — script CLASSIFICATION NFC-normalizes its input (NFD katakana carries combining marks outside the block; NFD hangul decomposes to jamo, which would miss the order rule for macOS-origin names) while segmentation MATCHING stays raw — chosen over offset-mapping complexity because NFD then degrades to no-split, never to a wrong split. The read-only fold is the one deliberate exception to "nothing rewrites the text" (rules.md T Background), and it never enters a token. +- 2026-07-29 (ja amendment §1a) — script CLASSIFICATION NFC-normalizes its input (NFD katakana carries combining marks outside the block; NFD hangul decomposes to jamo, which would miss the order rule for macOS-origin names) while SEGMENTATION stays raw — chosen over offset-mapping complexity because NFD then degrades to no-split, never to a wrong split. The read-only folds are the deliberate exceptions to "nothing rewrites the text" (rules.md T Background), and neither ever enters a token. (Corrected 2026-09-10, #cjk-full-stops: this read "segmentation MATCHING stays raw" and "the one deliberate exception". Vocabulary matching composes NFC too since #322 — `_lexicon._normalize` folds every lookup and every stored entry the same way — so the folds are two, and what stays raw is segmentation proper: the surname site's membership test and the peel's tail slice, which index the token's own text. The no-split degradation is what still holds there.) Declined: @@ -525,7 +527,7 @@ Closes #346, #344 and #343 as one bundle: each CLASS is decided once, and each s - **Bengali বাবু is trailing where Devanagari बाबू is leading** ("অমল বাবু" against a leading बाबू). Different codepoints, so the two cannot interact; the divergence is stated in both wordlist comments so that a sweep "harmonizing" them has to argue with the data module rather than discover the split afterwards. - **The Latin md history, corrected.** md entered TITLES in 8cd3123 (2015-09-05, #32) as the medical degree written before the name, alongside do, which #296's audit later took out (#vocabulary-collisions). The Muhammad reading is a 2026 reinterpretation, recorded in the case row audit_md_leading_stays_a_title; both readings agree on the field, which is why the entry survived #296's audit as the one deviation from that audit's table. Recorded here so the record stops implying md was added for Muhammad. - **মোঃ/মো and মোসাঃ/মোছাঃ go into TITLES**, mirroring Latin md. Semantically they are religious name prefixes rather than titles. TITLES is the FUNCTIONAL home because `given` must stay the name the person is addressed by — মোঃ আবদুল করিম is called আবদুল করিম — and the parser has no name-prefix field. The bound-given-name set is the wrong home for the same kind of reason it is the right one for Arabic abdul: abdul is an inseparable half of one name, মোঃ is detachable. Latin Mst waits for the opt-in pack (#345); the most already in TITLES is the English "Most Rev." word and is unrelated. -- **Abbreviation marks and periods.** An entry carrying an abbreviation mark is spelled with it: the Bengali visarga ঃ (U+0983) is a spacing combining mark (Unicode category Mc, not a letter and not punctuation), and `_lexicon._normalize` lowercases and strips edge whitespace and ASCII periods only — no NFC, no NFKC, no casefold anywhere in that module — so the mark is simply never touched and মোঃ reaches the lexicon and matches intact. Recompute with `Lexicon.default().add(titles={"মোঃ"}).titles & {"মোঃ"}`, which is non-empty iff the mark survived, and `unicodedata.category("ঃ")`, which is `Mc`. Period spellings (মো., মোসা., ডা., प्रो., पं.) match the bare stem through the same edge-period strip, so every BENGALI abbreviation carrying the mark ships as two entries, the visarga form and the bare stem (মোঃ and মো, মোসাঃ and মোসা, মোছাঃ and মোছা, মিঃ and মি), while the Devanagari abbreviations carry no such mark and ship as the bare stem alone (प्रो, पं, प्रा, डा); the dotted form is never listed in either script. +- **Abbreviation marks and periods.** An entry carrying an abbreviation mark is spelled with it: the Bengali visarga ঃ (U+0983) is a spacing combining mark (Unicode category Mc, not a letter and not punctuation), and `_lexicon._normalize` lowercases, composes NFC for a non-ASCII word and strips edge whitespace and the four full stops of `FULL_STOPS` — no NFKC, no casefold anywhere in that module — so the mark is simply never touched and মোঃ reaches the lexicon and matches intact (corrected 2026-09-10, #cjk-full-stops: the fold was ASCII periods only and no NFC when this was written; the visarga argument is unchanged, ঃ having no canonical decomposition, so NFC leaves it exactly where it was). Recompute with `Lexicon.default().add(titles={"মোঃ"}).titles & {"মোঃ"}`, which is non-empty iff the mark survived, and `unicodedata.category("ঃ")`, which is `Mc`. Period spellings (মো., মোসা., ডা., प्रो., पं.) match the bare stem through the same edge-period strip, so every BENGALI abbreviation carrying the mark ships as two entries, the visarga form and the bare stem (মোঃ and মো, মোসাঃ and মোসা, মোছাঃ and মোছা, মিঃ and মি), while the Devanagari abbreviations carry no such mark and ship as the bare stem alone (प्रो, पं, प्रा, डा); the dotted form is never listed in either script. - **Vocabulary beats the initial reading.** A one-letter-plus-period Bengali abbreviation is is_initial-shaped and the title entry wins: "ড. মুহাম্মদ ইউনূস" reads title ড., given মুহাম্মদ, family ইউনূস, while "র. কে. নারায়ণ" — real initials with no entry behind them — is unchanged. #343 verified the same fork at 2.1.0; it is re-verified here on the shipped set. This is MEASURED behavior pinned by rows, not a normative rule — no rule in rules.md states that title vocabulary outranks the one-letter-plus-period initial shape, rules.md#H2 speaking only to the bare-initial and the shape-versus-suffix cases — so a future shape veto must re-check "ড. মুহাম্মদ ইউনূস" before it lands. - **rules.md#H2's Accepted clause keeps its claim and loses its example.** The clause says an unlisted abugida honorific stays a name word, because the shape rule cannot see an unbroken run of letters through combining vowel signs; that is still true. Its example used প্রফেসর, which this bundle lists, so the example is now "প্রকৌশলী. Sen" → given="প্রকৌশলী." — Engineer, a loanword deliberately not shipped. The example is a contract-tier name in corpus_rules.jsonl, so the swap landed in the Bengali commit rather than in the records commit; leaving it would have exited the gate 1 there. - **Not recorded as excluded:** the Tier 4 candidates — transliterated military ranks, Advocate, Engineer, Marhum, and the kinship forms — are skipped pending a corpus check, not declined, and मां/माता/মা and the hafiz/syed/shahid forms are left alone the same way. An Excluded entry is a standing prohibition, and none of these has been argued to that standard. One of them carries a duty anyway: প্রকৌশলী (Engineer) is rules.md#H2's executable example of an unlisted abugida honorific, so whoever ships it must move the example to another unlisted word first, or the doc test and the contract-tier gate fail together. @@ -562,7 +564,7 @@ Closes #469, and continues the corpus-tier arc below rather than standing apart - 2026-09-05 — W2's PUNCTUATION HALF MOVED to W3 with its example; ITS SECOND HALF IS NORMATIVE AND STAYED, and this bullet is the within-branch correction of a first draft that took both. The clause read "The split-off ignores surrounding punctuation, but never treats a part that is not name text as the name's end". The FIRST half had no pure witness and could not be given one — native CJK writing puts no punctuation around a name for a split-off to ignore, so every input it describes is a listing form, which is W3's ground — and it moved there as description, rewritten to what is measured rather than to what it asserted (the bullet below). The SECOND half is a different claim and the reason given for cutting it does not reach it: it is what the parser does with NO comma and no period anywhere in the input, and contract-tier pure names depend on it. Measured 2026-09-05 by forcing `_is_post_nominal` to False in memory — the predicate the site scan `_peel_site` reads in nameparser/_pipeline/_script_segment.py — and re-parsing (family, given, suffix, title): `田中さん 様` ships `('田中', '', 'さん, 様', '')` and gives `('田中さん', '', '様', '')` without the step; `김민준 박사님` ships `('김', '민준', '박사님', '')` and gives `('김', '민준', '박사, 님', '')`; `선생님` ships `('선생님', '', '', '')` and gives `('선', '생', '님', '')`. The three break DIFFERENT clauses of the half, and a first draft of this bullet said all three were dissection. The restored sentence makes two claims — the post-nominal word is never the name's END, and it is never DISSECTED — and `田中さん 様` fails only the first: `様` comes out WHOLE as the suffix, undissected, but it is what the split-off takes as its SITE, so the peel lands on the post-nominal word and `田中さん` behind it is left unpeeled, where shipped the scan steps past `様` and peels `さん` off `田中さん`. `김민준 박사님` and `선생님` are the dissection failures, `박사님` coming apart into `박사, 님` and `선생님` into `선, 생, 님`. Between them the three witness both clauses, and neither clause is witnessed twice over. Two of the three, `김민준 박사님` and `선생님`, are `corpus_cjk.jsonl` names on the CONTRACT tier, and `ko_honorific_glued_doctor_spaced` is the #308/#271 regression row for the first; that row pins `suffix="박사님"`, which is what `parse('김민준 박사님').suffix` returns under the default facade, measured 2026-09-05. Under W2's remaining FIRST statement alone — a listed honorific glued to the end of the name's last name word splits off once — `박사님` ends in the listed tail `님` and reads as peelable, giving `박사` + `님`. So a W2 cut back to that one sentence would have BEGUN contradicting the regression row its own example line pins, not stopped: the second half is what keeps rules.md agreeing with the shipped parse, and that is the argument for restoring it. Restored in W2's register as "A part that is not name text — a post-nominal word standing on its own — is never the name's end: the split-off steps past it to the name word behind, and never dissects it." — witnessed by `김민준 박사님` → `suffix="박사님"` and the boundary line `선생님` → `family="선생님"`, both texts `corpus_cjk.jsonl` already held, so the two example lines arrive in `corpus_rules.jsonl` without adding a name to the pool. `田中さん 様` would have been the third witness and is NOT added: it is not a case row, and a new contract name needs a 1.4.0 classification this bundle does not have. `_script_segment.py` carries the restored sentence as a second W2 citation, on `_peel_site` where the scan-back lives. RECOMPUTE: rebind `_is_post_nominal` to `lambda state, i: False` on the module and re-parse the three. -- 2026-09-05, FOUND while rewriting the clause — A PERIOD ON THE NAME'S OWN LAST WORD IS NOT IGNORED; IT IS DECISIVE, and the moved half asserted the opposite. The half said the split-off "ignores surrounding punctuation", and W3's first descriptive rewrite of it kept that shape, attributing the crossing to the punctuation. Five parses, 2026-09-05 under the default facade, say otherwise — counted by INPUT STRING, the convention the rest of this bullet uses when it says "the five inputs" and "parse the five strings". A first draft counted four by counting the SENTENCES below instead, the last of which carries two inputs (`田中さん.` and `김민준씨.`). Six, if the no-period twin `田中さん 太郎` quoted inside the third sentence as its control counts as a parse of its own rather than as part of that item. `田中さん 様` → family `田中`, suffix `さん, 様`: the crossing happens with no punctuation in the input at all, so the vocabulary licenses it and not the period. `田中さん 様.` → family `田中`, suffix `さん, 様.`: the period rides along and moves nothing, `様.` reading post-nominal-strict exactly as `様` does. `田中さん 太郎.` → family `太郎.`, given `田中さん`: a period on a NAME word, and no crossing — and not ignored either, since the same input without it is family `田中さん`, given `太郎`, the order reading flipped by the period alone. `田中さん.` → title `田中さん.`, and `김민준씨.` → title `김민준씨.`: a period glued to the honorific's own word stands between the honorific and that word's end, no listed tail matches, the peel declines and the whole text reads as a title — the parse a period changes most, not least. Of the five inputs, the ONLY one where a period changes nothing is the one where it sits on a word the split-off was going to step past regardless. So W3 now says the step past a post-nominal word is W2's and needs no punctuation, that a period on a separate post-nominal word rides into the suffix, and that a period glued to the honorific's own word decides against the split-off. RECOMPUTE: parse the five strings, each also without its period, and read `title` alongside the name fields — a rewrite that reads only `family` and `suffix` sees two empty results and can call them a null effect. +- 2026-09-05, FOUND while rewriting the clause — A PERIOD ON THE NAME'S OWN LAST WORD IS NOT IGNORED; IT IS DECISIVE, and the moved half asserted the opposite. The half said the split-off "ignores surrounding punctuation", and W3's first descriptive rewrite of it kept that shape, attributing the crossing to the punctuation. Five parses, 2026-09-05 under the default facade, say otherwise — counted by INPUT STRING, the convention the rest of this bullet uses when it says "the five inputs" and "parse the five strings". A first draft counted four by counting the SENTENCES below instead, the last of which carries two inputs (`田中さん.` and `김민준씨.`). Six, if the no-period twin `田中さん 太郎` quoted inside the third sentence as its control counts as a parse of its own rather than as part of that item. `田中さん 様` → family `田中`, suffix `さん, 様`: the crossing happens with no punctuation in the input at all, so the vocabulary licenses it and not the period. `田中さん 様.` → family `田中`, suffix `さん, 様.`: the period rides along and moves nothing, `様.` reading post-nominal-strict exactly as `様` does. `田中さん 太郎.` → family `太郎.`, given `田中さん`: a period on a NAME word, and no crossing — and not ignored either, since the same input without it is family `田中さん`, given `太郎`, the order reading flipped by the period alone. `田中さん.` → title `田中さん.`, and `김민준씨.` → title `김민준씨.`: a period glued to the honorific's own word stands between the honorific and that word's end, no listed tail matches, the peel declines and the whole text reads as a title — the parse a period changes most, not least. Of the five inputs, the ONLY one where a period changes nothing is the one where it sits on a word the split-off was going to step past regardless. So W3 now says the step past a post-nominal word is W2's and needs no punctuation, that a period on a separate post-nominal word rides into the suffix, and that a period glued to the honorific's own word decides against the split-off. AMENDED 2026-09-10 by #cjk-full-stops, and it is the summary that goes rather than the measurements: every parse above keeps its date and is what the parser did on 2026-09-05 (the two suffix values render space-joined since #436/#437, the next day, `さん 様` for `さん, 様`). Of the four period-bearing inputs, ALL FOUR now divide exactly as their no-period twins — `田中さん 太郎.` reads family `田中さん`, given `太郎.`, the classification fold making the period invisible to the order rule, and `田中さん.` and `김민준씨.` peel their honorific with the stop riding on it — so the period changes no division at all here, and the third clause of the sentence before this one is superseded: a period glued to the honorific's own word rides WITH the honorific now. What survives is the first two clauses and the finding this bullet is titled for, inverted in its consequence rather than in its content: the period was decisive because a site was reading it, and the answer was to stop three more sites reading it. RECOMPUTE: parse the five strings, each also without its period, and read `title` alongside the name fields — a rewrite that reads only `family` and `suffix` sees two empty results and can call them a null effect. - COUNTS, 2026-09-05, with the recompute beside each. `corpus_cjk.jsonl` went 73 → 70 and `corpus_cjk_tolerated.jsonl` 26 → 29 — the same identity as the first demotion, 99 names either side, so the split MOVED three texts and dropped none. `corpus_rules.jsonl` went 248 → 247 → 249: W2 gave up the `田中さん 様.` line, then took the two the review round restored the second half with (`김민준 박사님`, `선생님`). Both arrivals are texts `corpus_cjk.jsonl` already held, so the file grew and the compared POOL did not — the same shape as the shapes-corpus growth of 2026-09-01, and the reason a witness had to be picked from names already inside. 32 case rows carry `tolerated=True` over 29 distinct texts (recompute: read `CASES` and count `c.tolerated`, per row and per text). `_CORPUS_FLOORS["corpus_cjk.jsonl"]` is lowered 70 → 67 deliberately, and the tolerated file's floor of 22 still stands under 29. The stop condition held again: intentional diffs UNMOVED at 352 / 247 / 155 / 14 across 1.4.0 / 2.0.0 / 2.1.0 / 2.2.0, unexplained 0, radar unclassified 0 and exit 0 at each. RECOMPUTE THE SWEEP, which is the part worth having rather than the digits: it is the CJK-bearing (`_has_cjk`) rows that do NOT declare `tolerated`, carrying any ASCII character other than a space, grouped by the kind of character that trips them and joined to the corpus files holding each text. Over the tree before this bundle it returns eight distinct texts — the three period rows, the parenthesized nickname, and the four mixed-script rows — and over this one, five: the period class is gone and the letter class is untouched, which is the shape of the answer to look for rather than the count. @@ -1098,7 +1100,7 @@ Declined (2026-07-06, the post-1.3.0 modernization sweep; each on churn-outweigh ### initials-repertoire — is_initial across scripts (#320) -- The principle: alphabetic-vs-CJK, not Latin-vs-non-Latin — Han, kana and hangul characters are morphemes or syllables, so a single one cannot stand in for a name word, while any alphabet's single letter can. _NO_INITIALS is ENUMERATED per script rather than derived from the script ranges, so a new Script member forces a decision instead of silently deciding Thai has no initials. +- The principle: decided per script from CLDR's LOCALE-AUTHORED name patterns rather than from the script's typology — ja and ko author 29 and 32 patterns and use an initial in none of them, while ru uses one in 7 of 39 (`_policy._NO_INITIALS`' comment carries the counting recipe and the trap: `initialPattern` alone reports a period convention for locales that have none). Corrected 2026-09-10, #cjk-full-stops: this read "alphabetic-vs-CJK, not Latin-vs-non-Latin — Han, kana and hangul characters are morphemes or syllables, so a single one cannot stand in for a name word", which is the phonological restatement that comment forbids in as many words — Devanagari is an abugida and Arabic an abjad, neither has letters in that sense and both abbreviate, so a rule keyed on typology would put them here and their initials are real. _NO_INITIALS is ENUMERATED per script rather than derived from the script ranges, so a new Script member forces a decision instead of silently deciding Thai has no initials. Declined: @@ -1204,6 +1206,25 @@ Every number below is a py3.11 measurement of 2026-08-31, recomputable with `uv - bytes input: removed in 2.0 (#245; decode-first DeprecationWarnings shipped 1.3.0 per the bridge discipline); the shipped TypeError carries the decode hint. - no_vowels: removed in 2.0 (#268, filed 2026-07-07, closed 2026-07-28) — never consulted by any parser version, ASCII-only; the facade carries no replacement because there was nothing to replace. +### cjk-full-stops — which characters are a period on CJK text (2026-09-10, #322/#323) + +ONE GAP SEEN FROM TWO SIDES, which is why the two issues were answered together. #322 asked WHICH CHARACTERS count as a full stop: the parser knew the ASCII period and nothing else, so `김민준 씨．` — written with the fullwidth stop a Japanese or Chinese IME produces by default — missed the suffix entry that `김민준 씨.` reached, and so did the ideographic `씨。` and its halfwidth `씨｡`. #323 asked WHERE a full stop is read at all: the one site that knew the ASCII period was the lexicon's edge strip, so a stop glued to a name word was invisible to a vocabulary lookup and to nothing else. Either answer alone leaves the other half broken — a fullwidth stop the lookup folds would still have taken its token out of the script reading — so the bundle is one arc with four code commits and this record. + +- **FULL_STOPS names four characters; five sites read them.** `nameparser/_lexicon.py::FULL_STOPS` is `".．。｡"`: the ASCII period, FULLWIDTH FULL STOP U+FF0E, IDEOGRAPHIC FULL STOP U+3002 and HALFWIDTH IDEOGRAPHIC FULL STOP U+FF61. Four sites import the string, so "which characters are a period" is answered once: `_lexicon._normalize` (the lookup AND storage fold), `_vocab._normalized_for_script` (the script-classification fold), and in `_script_segment` the surname site's head match and the honorific peel's tail match. The FIFTH site reads no stop set at all. `_pieces.is_leading_title` — rules.md#H2's opening-abbreviation shape — refuses a period-marked word whose text carries a character of a script with no initials, through `_vocab.in_initialless_script` (renamed from `_in_initialless_script`; the same veto `is_initial` has carried since #320), so `_PERIOD_ABBREV` stays an ASCII-period pattern and the four-character set is not copied there. Not every period test reads this set, and the ones that do not are named so the claim is checkable: the interior-period shapes — the dotted acronym `M.A.` and the split `Ph. D.` (`_vocab`) — and FOUR ASCII-only edge tests on Latin shapes across THREE modules — the initial `J.` (`_vocab`), the bracketed credential `(Mgr.)` and the word-internal apostrophe rule (`_extract`), and the trailing middle-initial carve-out `V.` (`_assign`) — stay ASCII. What that does NOT mean is that the wider stops leave Latin text alone. The TESTS stay ASCII; the vocabulary FOLD does not, so a Latin word wearing a wide stop reaches vocabulary an ASCII-only carve-out would have kept it out of — `V。` folds to `v` and does not end in the ASCII period that `_reads_as_a_trailing_suffix`'s carve-out tests (`text.endswith(".")`, `_assign.py` ~line 479, whose docstring calls the period "the whole carve-out"), so the carve-out never fires for `V。` and it is roman five, suffix, where `V.` stays a middle initial through that same carve-out — not through `is_initial`, which is not the decider despite reading true on both spellings: `is_initial("V")` is True, yet `Smith, John V` (no period at all) still reads suffix `V`, so being initial-shaped never by itself keeps a word out of suffix. `Jr。` needs no carve-out argument and simply reaches its entry. An earlier wording of this passage called `Ｍ．Ｂ．Ａ．` and `V。` "a fullwidth-Latin question this set does not raise" (corrected 2026-09-10): that was right for `V。` alone. `V。` is an edge stop on a bare word, and the fold raises the question at every such edge, moving the parse as above. `Ｍ．Ｂ．Ａ．` does NOT move — every one of its periods is interior, none at an edge for `_lexicon._normalize`'s edge strip to reach, so the fold never touches it at all, and the dotted-acronym reading in `_vocab.suffix_as_written` (`n.replace(".", "")`) is ASCII-only regardless of the fold: `_normalize("Ｍ．Ｂ．Ａ．")` is `ｍ．ｂ．ａ`, which matches no acronym form the ASCII-only test recognizes. MEASURED 2026-09-10: `parse("John Smith Ｍ．Ｂ．Ａ．")` and `parse("John Smith, Ｍ．Ｂ．Ａ．")` are identical on `d37b8ec` and on this tree — `Ｍ．Ｂ．Ａ．` stays exactly as it was. The set raises the question at every EDGE fold, and the four ASCII-only tests are the places it is not raised rather than a boundary on the set. MEASURED 2026-09-10 on this tree: `Andrew Perkins (Mgr。)` reads nickname `Mgr。` where `Andrew Perkins (Mgr.)` reads title `Mgr.`; `Smith, John V。` reads suffix `V。` where `Smith, John V.` reads middle `V.`; and `John Smith, Jr。` reads suffix `Jr。` where `d37b8ec` read given `Jr。`, family `John Smith`. The last two are case rows now (`latin_roman_numeral_with_an_ideographic_full_stop`, `latin_suffix_with_an_ideographic_full_stop`), and `is_suffix_strict("Jr。", Lexicon.default())` and `_normalize("Dr。") == "dr"` pin the fold's Latin reach at unit level. Nothing rewrites token text at any of the five: the stop stays on the token it arrived with, and every parsed part is still an exact slice of the input (rules.md's T Background). +- **NFC composes at LOOKUP, and that is what makes the fold symmetric.** `_normalize` now composes NFC for a non-ASCII word, at match time and at storage both. NFD hangul decomposes to jamo and missed every vocabulary entry, so `NFD("김민준, 씨.")` read title `씨.`, family `김민준` and now reads family `김민준`, suffix `씨.`. MEASURED 2026-09-09, re-measured 2026-09-10: no shipped entry changes under NFC — 0 of the 1735 strings in the twelve set-valued fields of `Lexicon.default()` (the standing pin `tests/v2/test_lexicon.py::test_every_shipped_entry_is_already_nfc` walks those twelve plus both halves of the five `capitalization_exceptions` pairs, 1745 strings, and asserts the same). RECOMPUTE: fold every string member of those fields through `unicodedata.normalize("NFC", w)` and count `!= w`. +- **The reach is not one script, and the Latin part of it was nobody's ask.** 310 of those 1735 entries are non-ASCII and 158 DECOMPOSE under NFD — and they divide by script in a way worth having in front of you, measured 2026-09-10: 117 HANGUL (every hangul syllable has a canonical decomposition, which is #322's own half — `씨`, `님`, `박사님`, `선생님`), 18 LATIN (`señor`, `née`, `attaché`, `fräulein`, `chargé`, `rozená` …), 13 ARABIC (`أبو`, `الأستاذة` …), 6 BENGALI and 4 CYRILLIC (`урождённая`, `й`). So NFD Latin input matches now too, which no issue asked for — measured against `d37b8ec`: `NFD("Señor Juan Garcia")` read given `Señor`, middle `Juan` and now reads title `Señor`, given `Juan`; `NFD("Attaché John Smith")` moves the same way; `NFD("Maria Kowalska née Nowak")` went from middle `Kowalska née`, family `Nowak` to family `Kowalska`, maiden `Nowak`. RECOMPUTE: count the entries `w` of those twelve fields where `unicodedata.normalize("NFD", w) != w`, and key the count by the first letter's Unicode name. The argument outlives the digits and is the SHAPE of the division rather than its size: the fix was reached for on a hangul gap, and every other composable script in the vocabulary rode along with it, so a decomposed export from a macOS-origin system was missing entries in five scripts and not in the one corner the issue named. +- **The stop set is script-agnostic at the lookup fold too, and the same unasked-for Latin reach exists for the three wider stops.** The edge strip in `_lexicon._normalize` takes any of the four `FULL_STOPS` off any word before a vocabulary lookup or a particle match, Latin and Cyrillic word forms included, so a Latin or Cyrillic word wearing the fullwidth, ideographic or halfwidth stop now reaches vocabulary and particle membership that only the ASCII period reached before — the same unasked-for reach the bullet above records for NFD, on a different axis of the fold. MEASURED 2026-09-10, against `d37b8ec`: `Dr。 John Smith` reads title `Dr。` where it read given `Dr。`; `Проф。 Иванов` reads title `Проф。` where it read given `Проф。`; `Beethoven, Ludwig van．` reads family `van． Beethoven` where it read family `Beethoven` alone. RECOMPUTE: parse the three strings on this tree and on `d37b8ec` and read `title` alongside the name fields. +- **NFKC was DECLINED, not deferred, and the proposal it declines is #322's own.** The issue asked for "an NFKC fold for vocabulary lookup only" on the argument that it "would address all three" of its mechanisms — one change with one risk profile instead of three patches. Measured, it addresses two and a half: `unicodedata.normalize("NFKC", "．")` is `"."` and `NFKC("｡")` is `"。"`, but `NFKC("。")` is `"。"` — the ideographic stop is not a compatibility character and no normalization folds it to a period, so `씨。`, one of the three spellings the issue opens with, would still have missed the entry and the character would still have had to be listed. And NFKC at the storage fold would REWRITE stored vocabulary (compatibility folds change ligatures, widths and superscripts), which is the one thing `_normalize`'s `lower()`-not-`casefold()` decision already refused. Listing four characters answers the question NFKC only half answers, and answers it without touching what a caller authored. RECOMPUTE: `unicodedata.normalize("NFKC", c)` for each of the four. +- **#323 turned out broader than it was filed.** The issue named the honorific peel. Making the classification fold read through an edge stop moved three readers of the `None` it used to return, and only one of them was the peel's neighbour test: the SURNAME SITE stepped past the family name onto the given name (`양. 지훈` cut 지훈 in half — given `양.`, middle `지`, family `훈` — and now reads given `지훈`, family `양.`), the ORDER RULE fell back to positional (`양 지훈.` lost family-first and now keeps it; `田中 太郎.` likewise), and the segmenter's neighbour precondition missed a writer-drawn boundary (`山田太郎 田中.` consulted a pluggable segmenter on `山田太郎` as if it stood alone, and is blocked now). The issue weighed two candidate fixes and chose neither: teaching the surname site to consult `is_suffix_strict` beside `effective_script`, or making the classification tolerant of a trailing period, which it called the broader one and expected to interact with #322. The broader one shipped, and the narrow one would have reached the filed name alone — `김.` is no suffix vocabulary, so `김. 민준` sits outside it entirely. +- **The surname-site amendment was found by measurement during execution, not designed.** With the classification fold alone, `김. 민준` read given `김`, middle `.`, family `민준`: the token classified as hangul, the site matched `김` against its own head, and what was left over — the stop — BECAME the remainder. So the site matches on `text.rstrip(FULL_STOPS)` and cuts the core, a head being a prefix, which makes the offset that cuts the core cut the text; the stop rides with the remainder (`김민준.` divides as `김` + `민준.`, `김. 민준` as family `김.` plus given `민준`). `rstrip`, not `strip`: a LEADING stop would break the prefix argument. The reading is recorded in `ebf64db`'s comment at the site. +- **Two readings JOINED the bundle, approved in session on 2026-09-09.** Neither was filed. FIRST, the glued-period honorific (rules.md#W3): a stop on the honorific's own word stood between the listed tail and the token's end, so nothing peeled and `田中さん.` and `김민준씨.` read as titles. The peel now matches the tail on the core and cuts BEFORE it, so both divide where their stop-less spellings do and the stop rides with the honorific. The move was free because W3 had recorded that reading as measured on 2026-09-05 and pinned by nothing, its own words being that neither string is a case row or a corpus line, so no row pinned those two readings; both are case rows now, tolerated. SECOND, H2's opening-abbreviation shape on a CJK word: `田中.` read title and now reads family `田中.`, `田中. 太郎` reads family `田中.`, given `太郎`. Han is what WITNESSES this one, and hangul cannot: hangul segmentation runs first, so `김민준.` never reaches the shape test and divides as `김` + `민준.` by the surname site instead. Latin and Cyrillic are untouched (`Smith. John` still reads title `Smith.`, `Проф. Иванов` title `Проф.`), and the veto is contains-any, so `Kim김. Smith` is refused a title too — deliberately: a word carrying a script with no abbreviations is not wearing an abbreviation's period. +- **Recorded limits, all five measured 2026-09-10 on this tree — and the segmenter one is a limit the review round CLOSED, kept in this bullet because this is where the claim stood.** HALFWIDTH KANA sits outside the script table, so it reads by the LATIN shapes at both period sites and not just at the initial one: `_vocab.is_initial("ﾗ.")` is `True` where `is_initial("ラ.")` is `False`, and H2's opening-abbreviation shape MISROUTES a longer halfwidth opener — `parse("ﾀﾅｶ. John")` gives title `ﾀﾅｶ.` where `parse("タナカ. John")` gives given `タナカ.`. Both because `_policy._SCRIPT_RANGES` excludes the halfwidth kana block U+FF65–FF9F on purpose — legacy bank and CSV data writes it, and the table's comment calls that a separate normalization problem, #272's separator handling touching only the halfwidth DOT U+FF65 and none of the kana behind it — and `_NO_INITIALS`, being a tuple of Script members, silently inherits the exclusion. That is #322's second mechanism, restated here because it is the one the bundle did NOT close. An earlier wording of this bullet said "nothing misroutes today, every kana honorific being multi-character", which had the direction backwards: the misroute is at the SHAPE site rather than the peel, and being multi-character is what carries a halfwidth opener INTO H2's shape rather than what keeps it out. NOTHING PINS EITHER READING — no case row, no corpus line, for the halfwidth block or for `ﾀﾅｶ. John` — so both can move without the suite or the differential saying so. It stays a recorded limit rather than a fix: what to do with halfwidth kana is a normalization question (widen the script table, or fold widths at classification) and not a full-stop one, and the bundle answered the full-stop one. RECOMPUTE: parse `ﾀﾅｶ. John` and `タナカ. John` and read `title` beside the name fields; `is_initial` alone reports only the shorter half. A PLUGGABLE SEGMENTER receives the CORE, the token with its trailing stops removed — the same string the vocabulary match reads, the surname site handing it `core` rather than `text` (corrected 2026-09-10, in the review round after this entry was first written: the site handed over the raw token, which this bullet recorded as a limit, and the fix costs nothing because a head is a prefix — every offset reported against the core cuts `text` in the same place, so the stops ride with the last piece and no answer of the segmenter's can make a stop a piece of its own). The shipped `ja` pack's `_wholly_japanese` therefore ACCEPTS `山田太郎.` now, where the raw token failed its repertoire test: measured 2026-09-10 with `namedivider-python` installed (`uv run --extra ja`), `parser_for(locales.JA, segmenter=locales.ja_segmenter())` reads `山田太郎.` as family 山田, given `太郎.` with a SEGMENTATION report scoring 0.44, where `d37b8ec` read title `山田太郎.` and the un-corrected branch read family `山田太郎.`; the already-divided `山田 太郎.` reads family 山田, given `太郎.` on this tree and given 山, middle 田, family `太郎.` at `d37b8ec`. Pinned by `test_a_consulted_segmenter_receives_the_core_without_the_stop`. A LEADING stop keeps the token out of the SURNAME site only, because the classification fold that gates it rstrips as it does: `.김민준` classifies as no script, never becomes a surname site and stays one whole word, reading given `.김민준` exactly as `d37b8ec` reads it (corrected 2026-09-10 in the same round: the fold stripped both edges when this entry was written, which moved the ROLE to family and — with a segmenter configured — offered the token to it raw, an answer of offset 1 dividing it into the stop and the name; the fold now matches the sites, and hiding such a token is the no-split rules.md#W1 already accepts). It does NOT keep the token out of the HONORIFIC PEEL, which consults no script at all — the tail alone is its license (rules.md#W2's Background) — so a leading stop does not defeat it: `.김민준씨` peels to `.김민준` and `씨` exactly as `김민준씨` does, pinned by `test_the_peel_reads_the_trailing_stop_only`. A BRACKETED CJK CREDENTIAL degrades: rules.md#S1's escape in `_extract` calls a clause suffix-shaped when it ends in an ASCII period, so `(김민준.) John Smith` is unwrapped and the H2 veto then leaves `김민준.` as name text the surname site divides — given 김, middle `민준. John`, family Smith, where 2.2.0 and `d37b8ec` read title `김민준.` — pinned as the tolerated `ko_name_with_a_period_in_a_bracketed_credential`. And a caller-authored NFD surname entry is composed at ingest, so raw NFD input then finds no entry and the name goes unsplit; pinned by `test_an_nfd_authored_entry_is_stored_composed`. +- **Pickle compatibility, decided 2026-09-10 (Derek): a release-note line, not a forgiving load.** `Lexicon.__setstate__` re-runs `_normalize` over the loaded state and rejects any entry the fold changes, so a `Lexicon` pickled by 2.1.x or 2.2.x whose caller-added entries the widened fold now touches (a non-ASCII entry written with a wide stop such as `씨。`, or authored in NFD) raises `ValueError: incompatible Lexicon pickle: entries are not normalized (titles: 씨。); this state was not written by this version of nameparser` — a message that is wrong about provenance for exactly these pickles, which a released version did write. Measured 2026-09-10 (recompute: take `Lexicon.default().add(titles={"씨"}).__getstate__()`, add `씨。` to its `titles`, and load it into `Lexicon.__new__(Lexicon)`). The shipped vocabulary is unaffected (zero of 1735 strings change under NFC, and no shipped entry carries a wide stop). Two remedies were weighed: a release-note line with the fix a caller applies (rebuild the `Lexicon` from its source rather than unpickle it), or a load path that accepts state whose only drift is one fold pass away, re-folds it and warns naming the entries. The note was chosen: the load guard was designed to refuse rewritten data (the guarded-raise design #3-0-reevaluations records as right regardless of the in-a-minor friction) rather than become a fourth place caller data is corrected without a word, the population is caller-added CJK entries pickled across a minor, and a rebuild is the documented way to carry a `Lexicon` across versions. The 2.3.0 release note carries the line; `tests/v2/test_lexicon.py::test_unpickling_rejects_unnormalized_entries` is the pin of the refusal itself. +- **CORRECTIONS this entry makes to entries above it.** #indic-honorifics' "Abbreviation marks and periods" bullet said `_normalize` "lowercases and strips edge whitespace and ASCII periods only — no NFC, no NFKC, no casefold anywhere in that module"; it now composes NFC for a non-ASCII word and strips all four stops. The VISARGA ARGUMENT still holds and holds for the same reason: ঃ U+0983 has no canonical decomposition, so NFC leaves it exactly where it was — recompute with that bullet's own one-liner, `Lexicon.default().add(titles={"মোঃ"}).titles & {"মোঃ"}`, still non-empty. #W1's 2026-07-29 §1a bullet said "segmentation MATCHING stays raw" and called the classification fold "the one deliberate exception"; matching is now folded too, so what stays raw is SEGMENTATION — the surname site's membership test and the peel's tail slice, which index the token's own text — and the folds are two, not one. #cjk-comma-demotion's 2026-09-05 five-parse bullet is amended in place: `田中さん 太郎.` no longer flips the order (the period is invisible to the classification, so it reads family `田中さん`, given `太郎.` as its no-period twin does), and `田中さん.` and `김민준씨.` no longer read as titles, so that bullet's summary — that the ONLY one of the five where a period changes nothing is the one where it sits on a word the split-off steps past — is superseded: the period changes nothing in four of the five now, and `田中さん 様.` is joined rather than alone. #initials-repertoire's principle bullet restated the veto phonologically ("morphemes or syllables"), which `_policy._NO_INITIALS`' own comment forbids in as many words — Devanagari is an abugida and Arabic an abjad, neither has letters in that sense and both abbreviate — so the bullet now states the CLDR criterion the constant actually uses. +- **What moved, measured 2026-09-10 and re-measured the same day after the review round.** SEVENTEEN names in `tools/differential/corpus*.jsonl` read differently against `d37b8ec`, and all seventeen are the bundle's own new rows, every one of them on the RADAR tier in `corpus_cjk_tolerated.jsonl`. Twelve are the CJK movers this bundle's landing commits wrote; the thirteenth, the wholly-katakana `マイケル.`, joined in the whole-branch review that produced this entry, reading given `マイケル.` where `d37b8ec` reads title `マイケル.` — the katakana arm of the same H2 veto, pinned as `ja_katakana_lone_name_with_a_period_is_not_a_title` in `tests/v2/cases.py`. The last FOUR joined in the review round after that, as pins on readings nothing held: three stop-bearing FAMILY_COMMA spellings (`김민준씨., J.씨`, `田中さん., V.`, `이, J.씨.`) and the bracketed `(김민준.) John Smith`, the recorded degradation in the limits bullet above. A leading-stop shape moves the same way but sits in no corpus at all: `.김민준씨.` — a stop no script ever writes, glued before a honorific-bearing word — read one whole given `.김민준씨.` at `d37b8ec` and now reads given `.김민준`, suffix `씨.`, pinned only by the stage tests for its two edges taken separately (`test_the_peel_reads_the_trailing_stop_only` for the leading stop, `test_peels_a_listed_tail_through_a_trailing_full_stop` for the trailing one) rather than by any corpus row or a combined case of its own. The three comma rows agree with their stop-less twins in every field once the riding stop is removed from the suffix (`씨., J.씨` vs `씨, J.씨`; `さん.` vs `さん`; `씨.` vs `씨`) — each differs from its twin in exactly that one field, by exactly the stop that rides — which is the peel's own "one name, two spellings" argument holding as far as a raw field comparison can show it; the bracketed row does not agree with anything and is not meant to. `田中. 太郎` stays there with the rest, though it is what WITNESSES rules.md#H2's Accepted clause added here: `tools/differential/compare.py`'s tier comment records the 2026-09-05 precedent for exactly this collision — where a demoted file holds a text a rules.md example line names, the EXAMPLE LINE moves into the tolerated rule and the row is not promoted, because marking the row alone leaves the name enforced and documented as demoted. So the clause's CJK reading is carried in W3's tolerated example block, H2's own block keeps the Latin control `Smith. John`, and all four #323 shape rows stay `tolerated`. THE POPULATION THAT COULD HAVE MOVED AND DID NOT is two different sets depending on which criterion is read, and both are worth having in front of you. Under the SHAPE this bundle is about — an edge full stop glued to a word carrying a classified character — the corpora hold 22 names: the seventeen movers and FIVE non-movers (`田中さん 様.`, `田中さん, 様.`, `김민준 씨.`, `김민준 양.`, `김민준, 씨.`), all five already in `corpus_cjk_tolerated.jsonl` before this bundle. Two of the twenty-two need the detector to be written carefully, which is why the recipe is spelled out below: the stop on `김민준씨., J.씨` and `田中さん., V.` sits before a COMMA, so a detector splitting on whitespace alone sees `김민준씨.,` and finds no edge stop (an 18 recorded earlier in the day came from such a detector, over a corpus four names smaller), and the stop on `(김민준.)` is inside a bracketed clause until rules.md#S1's escape unwraps it. Under the looser reading of an ASCII PERIOD ANYWHERE in a CJK-bearing name, the corpora held 20 distinct names on 21 rows before this bundle (19 rows in the tolerated file and 2 in `corpus_issues.jsonl`, `Dr 김민준씨, Jr.` being the one name on two of them) — unmoved by the katakana row, which is a name this bundle's own work wrote, not a pre-bundle count. The looser set is wider because it sweeps in periods sitting on LATIN tokens inside a CJK name — `毛 泽东 Dr.` and `田中さん, V.` — which is precisely what the veto is scoped not to touch, it reading the word and not the name. NEITHER the five nor the twenty moves — the seventeen movers are the whole of what did, and every one of them is a row this bundle wrote. An earlier wording of this bullet said "the twenty-one CJK names carrying an ASCII period": that counted rows as names and quoted the looser criterion beside the shape's argument. RECOMPUTE: parse the union of the corpus files on both trees and diff the seven fields and the ambiguity kinds; for the SHAPE population, take each name's tokens (`_pipeline._tokenize`, not a whitespace split) and ask whether any token's `strip(FULL_STOPS)` is shorter and still carries a classified character, then add the bracketed clause by hand; for the looser one, count distinct names and rows separately. The gate exits 0 at all four baselines with `radar unclassified: 0`; one ledger rule carries them, `fix(#322/#323)`, ELEVEN members at 1.4.0 and 2.0.0 and SEVENTEEN at 2.1.0 and 2.2.0 — the three hangul names the older ledgers hand to their native-script CJK rule are one part of the difference, 2.1.0 being the release that shipped that behavior, and the three FAMILY_COMMA rows are the other: at 2.0.0 none of the three diffs at all, and at 1.4.0 the two that do (`田中さん., V.`, `이, J.씨.`) are already claimed by the broad `fix(cjk-comma-compound)` and `fix(cjk-comma-honorific-peel)` rules that ledger carries, so adding them to this rule would only take a name off a rule that describes it — the same narrow-first reasoning that keeps the three hangul names out. +- **The invariant the rules imply, run over two inputs (AGENTS.md axis 11), measured 2026-09-10.** If an edge full stop is punctuation on a CJK word, then a CJK name written with one must divide exactly as the same name written without one. Written as a check over the corpora — for every corpus name carrying a classified character and an edge stop on some word, parse it, parse the spelling with every word's edge stops removed, and compare the seven fields with the stops removed from the FIRST parse's values too — 30 of the 34 such names agree (re-measured 2026-09-10 after the review round's four new rows; it read 28 of 30 before them, the katakana row マイケル. having joined the population and agreed). FOUR do not, and only three of those are about the parser. Two are the standing boundary, and neither is a CJK word: `毛 泽东 Dr.` (the period makes `Dr.` title vocabulary the trailing walk claims, where bare `Dr` is a name word) and `田中さん V.` (`V.` is a dotted numeral where bare `V` is the generational suffix) — both LATIN words wearing a Latin period inside a CJK name, which is exactly what the veto is scoped not to touch, it reading the word and not the name. The third is new and is the bracketed degradation the limits bullet records: `(김민준.) John Smith` reads given 김 / middle `민준. John` / family Smith where `(김민준) John Smith` reads nickname 김민준 — the period is what makes rules.md#S1 call the clause a credential rather than a nickname, so the two spellings are genuinely different inputs to `_extract` and the invariant should not hold across them. The FOURTH is the comparison's own artifact and not a finding: `이, J.씨.` and `이, J.씨` divide identically (given `J.`, family 이, suffix `씨.` and `씨`), but stripping the first parse's VALUES takes the period off the initial `J.`, where in the bare input that period is interior and survives. A normalization that runs over field values cannot tell a word-final stop in a value from a word-interior one in the name; that is worth knowing before reading the next round's count. RECOMPUTE with that description; it consults no rule statement, which is the point of the axis. +- **Parse cost, measured 2026-09-10 on py3.11** (`tools/perf/call_count.py --against d37b8ec`, and a script counting frames the way its `calls_for` does for the other names). The reference name never reaches the veto — `Dr.` is listed, so the title piece is taken before the shape test — and `parse=409.00 facade=446.00` is unchanged at both trees. On an H2-SHAPED OPENER the veto costs three to five frames: facade 304 → 308 for `Xyz. John Smith`, 385 → 390 for `Xyz. van Berg MA`, 231 → 232 for `Smith. John` (parse-only: 267 → 271, 348 → 353, 194 → 195). The number to quote is these and not "zero"; decisions.md#parse-cost's budget is about the reference name and this is off it. The largest movement measured is not the veto at all: `김민준씨.` goes facade 209 → 397, the peel now firing where the parse used to bail to a title reading, which is a whole segmentation path running rather than an added test. + ### script-table-placement — a recorded reversal - 2026-07-29 (script-ranges relocation) — REVERSES #271's documented choice ("tables deliberately live in _vocab"): the tables moved to _policy because packs cannot import the pipeline, evidenced by two hand copies with sync tests and a blocked 16–30× compiled-regex optimization (a module-level re.Pattern in a pack would flip its classification, hence the closure-held-pattern convention). Rider: U+3006 〆 joined the HAN span on a justification deliberately beyond UAX #24 — it is Script=Common but appears solely in Japanese surnames (〆木, 〆谷, 〆野). diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 159b54f1..966e76a4 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -55,7 +55,7 @@ Problem shape. "Which stage does X?" — asked before attributing behavior in pr ## ONE-PREDICATE-PER-QUESTION — one predicate answers it, and every other site calls that -Problem shape. Two stages need the same answer about the same input, and the one that does not own the decision is about to test for it. Contract statement. Where two sites ask the same question, exactly one predicate answers it and every other site calls that one — never a condition written to match it. The predicate belongs to the QUESTION, not to whichever stage decides: it may sit in a leaf both stages import, and for the leading-title test it must, since the deciding stage is assign and group cannot import assign. How it works. A hand-written mirror agrees with its original only until one of them moves, and the drift is invisible in both directions: each site keeps passing its own tests while they disagree about an input neither covers. Five instances, every one found as a defect before it was found as a pattern — #319 lifted the wholly-suffix predicate into the vocabulary layer "so the comma decision and the honorific peel's segment test cannot drift apart"; #401/#421 lifted the trailing-numeral fork out of assign so the bound-given reserve stopped carrying a copy, its hand-written mirror having been falsified in review more than once — the lesson recorded there being that what must be mirrored is assign's WALK, not merely its condition; #425 replaced that reserve's hand re-derivation of the trailing peel with one function over the view the join would leave; #424 moved assign's leading-title test down because group's own `title()` does not see H2's unlisted abbreviations, so `Xyz. van Johnson` chained where `Dr. van Johnson` did not; #429 moved the no-name-segment test down because group asked by segment INDEX where assign asks by CONTENT. The destination follows the LAYER, not the topic: a predicate over token text goes to `_vocab`, one over pieces and tags to `_pieces`. Both are leaves the stages sit on. The piece layer got its own module only in #439 — until then those predicates collected in `_group`, not because grouping owned them but because `_assign` imports `_group` and cannot be imported back, so group was the one place both stages could reach; five had accumulated across four PRs before the module existed. Stage order is this mechanism's limit, and it forecloses the alternative: where the reader comes AFTER the decider, record the answer on the state instead — `ParseState.order` is that shape, "Recorded rather than recomputed downstream, because the two can differ" — which is unavailable whenever the EARLIER stage is the one asking. (The concrete assign→group import that forced the `_group` collection is gone since #439; what remains is the ordering it was a symptom of, and tests/v2/test_layering.py is where the leaf's contract is now written down.) The cost is a second evaluation of the same predicate, measured for #429 at 1.2–2.2% of a family-comma parse and 0% of every other; recording that number was the right answer there over plumbing a state field the two sites would not otherwise share. Lives in. nameparser/_pipeline/_vocab.py over text (is_wholly_suffix; is_trailing_numeral_suffix — the #401/#421 instance, whose only caller since #439 is the shared peel rather than a stage; and maiden_marker_run, the #434 instance and the clearest two-stage case, called by classify over token texts and by extract over a clause's whitespace words, with group reading the tags classify recorded because it runs later; and delimiter_cores, the #436/#437 instance, read by group where a tail segment DROPS a configured delimiter core and by post_rules where the suffix view's entry boundary asks whether a dropped token was one, with a third reader inside this same module, is_wholly_suffix, where a configured core counts as suffix-shaped) and nameparser/_pipeline/_pieces.py over pieces: is_suffix_piece, leading_titles and peel_walk are called by both stages, while is_leading_title, is_title_piece and trailing_start are called by group alone (measured 2026-09-06 by call site: `is_leading_title` has no caller in `_assign.py`, which reads `leading_titles` instead — a first draft of this clause listed it among the shared ones) — `trailing_start` being the one to know, since it answers where the trailing run begins and is what P2's chain and M2's walk stop at — and segment_suffix_reading by assign alone since #436/#437, that last one being #430's instance, where THREE readers shared one answer until the render join, group's third, was replaced by a rule over the commas the writer typed (decisions.md#C1, 2026-09-06); it stays where it is, one call site being no reason to move a predicate that two sites will contest again. `trailing_titles` was that last shape for one day (2026-09-08, the #316/#489 bundle, rules.md#H5), and since the /simplify round of 2026-09-09 the SHARED predicate is `tail_reading` instead — the peel-and-chain fixed point that answers where the name pieces end (decisions.md#H5). Assign calls it at its main walk and group's bound-given reserve calls it twice, once per view the join compares, because that reserve reads the name words assign will leave and this walk is half of what leaves them (rules.md#P5; counting a trailing title word among them joined 'Prof. abdul rahman Prof.' where 'Prof. abdul rahman' does not). `peel_trailing` and `trailing_titles` are what that fixed point is BUILT from, and neither is a two-stage question any longer: `peel_trailing` has no caller outside `_pieces.py` at all, `trailing_start` and `tail_reading` being the only two and both in the leaf, and `trailing_titles` has exactly one, assign's family-comma segment-1 walk, which reads the chain without the re-peel; `_group.py` imports neither. The tail reading is in the leaf rather than inline because each assign site had been given a cheap frame-free gate written to match the walk's own first condition, which is a second implementation of the question and was removed in review; what the leaf costs is one frame per entry point, measured, and the walk's own first test is a compiled regex rather than a call, so an ordinary name pays a match and stops. The reserve's two calls cost the reference name nothing — it never enters that branch, having no bound given word — and the parse and facade frame counts did not move (measured 2026-09-09). Re-measured 2026-09-09 by an AST call-site census over `_pipeline/*.py` — every call node whose callee is one of these names, keyed by module and enclosing function, which is what caught the census claiming a share for `peel_trailing` that the round had just taken away — the rest of it holds unchanged: is_suffix_piece, leading_titles, peel_walk and now tail_reading shared, is_leading_title, is_title_piece and trailing_start group-only — assign still reads `leading_titles` and never `is_leading_title`, which is what keeps H2's shape inference out of the trailing slot. And nameparser/_pipeline/_post_rules.py over a state: suffix_entries, the #511 instance, the R1 entry pass as a function, the one instance living in a stage rather than in a leaf — it is a pass over a whole ParseState and no leaf takes one, and AGENTS.md names it as the exception — run by post_rules last in the stage (through its in-place worker) and by Parser.revise over a sub-parse whose roles it has forced, so a suffix value handed to revise() derives its entries by the rule a whole name uses rather than by a second reading of the value's commas (decisions.md#C1, 2026-09-06 #511). tests/v2/test_layering.py holds each module's contract, and a piece predicate growing a dependency on a STAGE shows up there as a widened entry. Reach for it when. You are about to write a condition that mirrors, matches or "does what X does" — or you find a comment saying one does. Grep for the other site's predicate and call it instead. +Problem shape. Two stages need the same answer about the same input, and the one that does not own the decision is about to test for it. Contract statement. Where two sites ask the same question, exactly one predicate answers it and every other site calls that one — never a condition written to match it. The predicate belongs to the QUESTION, not to whichever stage decides: it may sit in a leaf both stages import, and for the leading-title test it must, since the deciding stage is assign and group cannot import assign. How it works. A hand-written mirror agrees with its original only until one of them moves, and the drift is invisible in both directions: each site keeps passing its own tests while they disagree about an input neither covers. Five instances, every one found as a defect before it was found as a pattern — #319 lifted the wholly-suffix predicate into the vocabulary layer "so the comma decision and the honorific peel's segment test cannot drift apart"; #401/#421 lifted the trailing-numeral fork out of assign so the bound-given reserve stopped carrying a copy, its hand-written mirror having been falsified in review more than once — the lesson recorded there being that what must be mirrored is assign's WALK, not merely its condition; #425 replaced that reserve's hand re-derivation of the trailing peel with one function over the view the join would leave; #424 moved assign's leading-title test down because group's own `title()` does not see H2's unlisted abbreviations, so `Xyz. van Johnson` chained where `Dr. van Johnson` did not; #429 moved the no-name-segment test down because group asked by segment INDEX where assign asks by CONTENT. The destination follows the LAYER, not the topic: a predicate over token text goes to `_vocab`, one over pieces and tags to `_pieces`. Both are leaves the stages sit on. The piece layer got its own module only in #439 — until then those predicates collected in `_group`, not because grouping owned them but because `_assign` imports `_group` and cannot be imported back, so group was the one place both stages could reach; five had accumulated across four PRs before the module existed. Stage order is this mechanism's limit, and it forecloses the alternative: where the reader comes AFTER the decider, record the answer on the state instead — `ParseState.order` is that shape, "Recorded rather than recomputed downstream, because the two can differ" — which is unavailable whenever the EARLIER stage is the one asking. (The concrete assign→group import that forced the `_group` collection is gone since #439; what remains is the ordering it was a symptom of, and tests/v2/test_layering.py is where the leaf's contract is now written down.) The cost is a second evaluation of the same predicate, measured for #429 at 1.2–2.2% of a family-comma parse and 0% of every other; recording that number was the right answer there over plumbing a state field the two sites would not otherwise share. Lives in. nameparser/_pipeline/_vocab.py over text (is_wholly_suffix; is_trailing_numeral_suffix — the #401/#421 instance, whose only caller since #439 is the shared peel rather than a stage; and maiden_marker_run, the #434 instance and the clearest two-stage case, called by classify over token texts and by extract over a clause's whitespace words, with group reading the tags classify recorded because it runs later; and delimiter_cores, the #436/#437 instance, read by group where a tail segment DROPS a configured delimiter core and by post_rules where the suffix view's entry boundary asks whether a dropped token was one, with a third reader inside this same module, is_wholly_suffix, where a configured core counts as suffix-shaped; and in_initialless_script, the #322/#323 instance and the only one here that is a REPERTOIRE test rather than a vocabulary one — the script half of the #320 initial veto, read by is_initial one function away and by _pieces.is_leading_title, so "a script with no initials has no period abbreviations either" is one predicate over _policy._NO_INITIALS rather than a second reading of that table; it lost its leading underscore when the second caller arrived) and nameparser/_pipeline/_pieces.py over pieces: is_suffix_piece, leading_titles and peel_walk are called by both stages, while is_leading_title, is_title_piece and trailing_start are called by group alone (measured 2026-09-06 by call site: `is_leading_title` has no caller in `_assign.py`, which reads `leading_titles` instead — a first draft of this clause listed it among the shared ones) — `trailing_start` being the one to know, since it answers where the trailing run begins and is what P2's chain and M2's walk stop at — and segment_suffix_reading by assign alone since #436/#437, that last one being #430's instance, where THREE readers shared one answer until the render join, group's third, was replaced by a rule over the commas the writer typed (decisions.md#C1, 2026-09-06); it stays where it is, one call site being no reason to move a predicate that two sites will contest again. `trailing_titles` was that last shape for one day (2026-09-08, the #316/#489 bundle, rules.md#H5), and since the /simplify round of 2026-09-09 the SHARED predicate is `tail_reading` instead — the peel-and-chain fixed point that answers where the name pieces end (decisions.md#H5). Assign calls it at its main walk and group's bound-given reserve calls it twice, once per view the join compares, because that reserve reads the name words assign will leave and this walk is half of what leaves them (rules.md#P5; counting a trailing title word among them joined 'Prof. abdul rahman Prof.' where 'Prof. abdul rahman' does not). `peel_trailing` and `trailing_titles` are what that fixed point is BUILT from, and neither is a two-stage question any longer: `peel_trailing` has no caller outside `_pieces.py` at all, `trailing_start` and `tail_reading` being the only two and both in the leaf, and `trailing_titles` has exactly one, assign's family-comma segment-1 walk, which reads the chain without the re-peel; `_group.py` imports neither. The tail reading is in the leaf rather than inline because each assign site had been given a cheap frame-free gate written to match the walk's own first condition, which is a second implementation of the question and was removed in review; what the leaf costs is one frame per entry point, measured, and the walk's own first test is a compiled regex rather than a call, so an ordinary name pays a match and stops. The reserve's two calls cost the reference name nothing — it never enters that branch, having no bound given word — and the parse and facade frame counts did not move (measured 2026-09-09). Re-measured 2026-09-09 by an AST call-site census over `_pipeline/*.py` — every call node whose callee is one of these names, keyed by module and enclosing function, which is what caught the census claiming a share for `peel_trailing` that the round had just taken away — the rest of it holds unchanged: is_suffix_piece, leading_titles, peel_walk and now tail_reading shared, is_leading_title, is_title_piece and trailing_start group-only — assign still reads `leading_titles` and never `is_leading_title`, which is what keeps H2's shape inference out of the trailing slot. And nameparser/_pipeline/_post_rules.py over a state: suffix_entries, the #511 instance, the R1 entry pass as a function, the one instance living in a stage rather than in a leaf — it is a pass over a whole ParseState and no leaf takes one, and AGENTS.md names it as the exception — run by post_rules last in the stage (through its in-place worker) and by Parser.revise over a sub-parse whose roles it has forced, so a suffix value handed to revise() derives its entries by the rule a whole name uses rather than by a second reading of the value's commas (decisions.md#C1, 2026-09-06 #511). tests/v2/test_layering.py holds each module's contract, and a piece predicate growing a dependency on a STAGE shows up there as a widened entry. Reach for it when. You are about to write a condition that mirrors, matches or "does what X does" — or you find a comment saying one does. Grep for the other site's predicate and call it instead. ## RENDER-HONORS-THE-PARSE — the parse decides it, the views honor it diff --git a/docs/design/rules.md b/docs/design/rules.md index ba455a15..ca8bc389 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -108,6 +108,17 @@ H2. Rationale: before a name, an abbreviation is almost always a word, and only vocabulary can recognize it — which is what #343 and #344 supply for Bengali and Devanagari. "প্রকৌশলী. Sen" → given="প্রকৌশলী." + Accepted: the shape reads a Latin convention, and a script with + no initials has no period abbreviations either, so a period- + marked opening word carrying a Han, kana or hangul character — + as the script table classifies them; halfwidth katakana sits + outside it and still reads by the Latin shape, the limit + decisions.md#cjk-full-stops records — is a name word and never a + title by shape (#323; decisions.md#cjk-full-stops) — the same + veto that keeps 씨. from reading as an initial. The Latin + reading is unchanged, and W3's example block carries the CJK + reading this clause vetoes into. + "Smith. John" → title="Smith." Accepted: before a family comma the pre-comma text is wholly the family name (C1), so no shape or vocabulary reading makes a title there. @@ -117,7 +128,7 @@ H2. Rationale: before a name, an abbreviation is almost always a not open: the vocabulary decides, and "Esq." is the postnominal it is. "Smith, Esq." → suffix="Esq." - history: decisions.md#H2 · interacts: C1, P4, H5 · implemented: nameparser/_pipeline/_assign.py, nameparser/_pipeline/_pieces.py + history: decisions.md#H2 · interacts: C1, P4, H5, W3, W4 · implemented: nameparser/_pipeline/_assign.py, nameparser/_pipeline/_pieces.py H3. Rationale: compound titles are written as a run of title words, connectives included; a title word standing inside the name is @@ -1338,7 +1349,7 @@ O5. Rationale: O4 reads a name by comparing where its words stand, ## Scripts & writing systems (W) -Background: script-conditional behavior is permitted exactly where the writing system itself — not statistics about it — settles the convention; a language can never be inferred from Latin-script text, because transliteration destroys the signal. The facts this section builds on: Chinese and Japanese both write the family name first in native script, so the script settles the order without knowing the language. Hangul is written by exactly one language and Korean family names are a small closed census set. Han text does not identify its language — a Chinese surname list would divide Japanese 高橋一郎 as 高 + 橋一郎 — which is why Han division is opt-in and there is no Korean pack to opt into. Hiragana never transcribes a foreign name (transcriptions are katakana alone), so kanji-plus-kana is a Japanese name in Japanese order, while wholly-katakana is predominantly a transcribed foreign name already in given-first order. Real Chinese text is unspaced (毛泽东); the spaced 毛 泽东 is an artifact. A fuller narrative lives in docs/usage.rst's East Asian section. One fact carries its own consequence: none of the three writing systems marks the family name with a comma — position in the written form is what identifies it, so a comma standing between the family name and the given name is a listing convention carried in from elsewhere rather than a form the script produces. That is why the rule reading one (W3) is tolerated rather than normative. CLDR's own locale data says the same where a contrary convention would have had to appear: across its ko, zh and ja personName patterns not one of the 126 pattern strings carries a comma of any width, the surname-first referring patterns separating surname from given by a single space, and the only comma in reach belongs to the locale-neutral root's sorting format — a list-ordering format, which ko and zh override comma-free and ja all but one inherited slot (decisions.md#cjk-comma-demotion carries the pull verbatim, with its URLs, its commit and its date). +Background: script-conditional behavior is permitted exactly where the writing system itself — not statistics about it — settles the convention; a language can never be inferred from Latin-script text, because transliteration destroys the signal. The facts this section builds on: Chinese and Japanese both write the family name first in native script, so the script settles the order without knowing the language. Hangul is written by exactly one language and Korean family names are a small closed census set. Han text does not identify its language — a Chinese surname list would divide Japanese 高橋一郎 as 高 + 橋一郎 — which is why Han division is opt-in and there is no Korean pack to opt into. Hiragana never transcribes a foreign name (transcriptions are katakana alone), so kanji-plus-kana is a Japanese name in Japanese order, while wholly-katakana is predominantly a transcribed foreign name already in given-first order. Real Chinese text is unspaced (毛泽东); the spaced 毛 泽东 is an artifact. A fuller narrative lives in docs/usage.rst's East Asian section. One fact carries its own consequence: none of the three writing systems marks the family name with a comma — position in the written form is what identifies it, so a comma standing between the family name and the given name is a listing convention carried in from elsewhere rather than a form the script produces. That is why the rule reading one (W3) is tolerated rather than normative. CLDR's own locale data says the same where a contrary convention would have had to appear: across its ko, zh and ja personName patterns not one of the 126 pattern strings carries a comma of any width, the surname-first referring patterns separating surname from given by a single space, and the only comma in reach belongs to the locale-neutral root's sorting format — a list-ordering format, which ko and zh override comma-free and ja all but one inherited slot (decisions.md#cjk-comma-demotion carries the pull verbatim, with its URLs, its commit and its date). A full stop of any width — the ASCII period, the fullwidth ．, the ideographic 。 and its halfwidth ｡ — glued after a script-written word is punctuation and not part of the word: it is invisible to the script reading and to the vocabulary, and it stays in the text on the word it arrived with, because no East Asian script writes an initial or an abbreviation with a period (#322, #323; decisions.md#cjk-full-stops). A stop glued BEFORE the word is punctuation to the vocabulary lookup, which folds both edges away, so .씨 is still the honorific. The classification fold that feeds the two division sites reads the trailing edge only, so a word wearing a leading stop is given no script at all and never becomes a surname site: .김민준 stays one whole word, given, no script rule reaching it. The honorific peel (W2) is not gated by that fold — the tail alone is its license — so it reads such a token regardless of a leading stop, and its own trailing-edge fold is what decides there: .김민준씨 peels to .김민준 and 씨, and .김민준씨. peels to .김민준 and 씨. (tests/v2/pipeline/test_script_segment.py's test_the_peel_reads_the_trailing_stop_only). W1. Rationale: hangul is monoglot Korean and its surnames are a closed census set, so an unspaced hangul name divides at a @@ -1416,14 +1427,18 @@ W3. Rationale: a family name declared by a comma is the writer's vocabulary alone and taken with no punctuation anywhere in the input; a period on a SEPARATE post-nominal word rides along into the suffix and moves no division (田中さん 様. divides where - 田中さん 様 does). A period glued to the honorific's OWN word is - read instead, and decides against the split-off: the listed tail - no longer ends that word, so nothing peels and the text goes on - whole — 田中さん. and 김민준씨. each read as a title, measured - 2026-09-05. Neither string is a case row or a corpus line, so no - row pins those two readings and they can move without the suite - or the differential saying so; the sentence reports them rather - than promising them, even by this rule's standard. + 田中さん 様 does). A period glued to the honorific's OWN word + rides with the honorific: the split-off matches the listed tail + through it and cuts before it, so 田中さん. and 김민준씨. divide + where 田中さん and 김민준씨 do (#323; until 2026-09-10 each read + as a title, measured 2026-09-05 and pinned by nothing, which is + why the move was free), and a period-marked Han opener divides + as its stop-less spelling does, 田中. 太郎 reading family-first + — the reading H2's Accepted clause vetoes a title into, carried + here because an edge full stop makes it tolerated input like the + rest of this block. All three are case rows now, tolerated, and + the sentence reports them rather than promising them, even by + this rule's standard. decisions.md#cjk-comma-demotion carries the parses. "남궁민수" → family="남궁" "지훈, 남궁민수" → given="남궁민수" @@ -1431,8 +1446,10 @@ W3. Rationale: a family name declared by a comma is the writer's "田中さん, Dr." → family="田中さん" "田中さん, PhD" → suffix="さん, PhD" "田中さん 様." → suffix="さん 様." - tolerated: native CJK writing has neither a family-comma convention nor a period standing after an honorific, so the four comma lines above and the period line under them illustrate current behavior — changeable without notice — rather than promise it; the line carrying neither, beside them, is W1's claim, which is normative. All five stay watched at every released baseline on the differential's radar tier (tools/differential/corpus_cjk_tolerated.jsonl, projected from the `tolerated` rows of tests/v2/cases.py) instead of its contract tier, and those rows pin them at HEAD. - history: decisions.md#W3 · interacts: W1, W2, C1 · implemented: nameparser/_pipeline/_script_segment.py + "김민준씨." → suffix="씨." + "田中. 太郎" → family="田中." + tolerated: native CJK writing has neither a family-comma convention nor an edge full stop of any width on a name word, so the four comma lines above and the three period lines under them illustrate current behavior — changeable without notice — rather than promise it; the line carrying neither, beside them, is W1's claim, which is normative. All seven stay watched at every released baseline on the differential's radar tier (tools/differential/corpus_cjk_tolerated.jsonl, projected from the `tolerated` rows of tests/v2/cases.py) instead of its contract tier, and those rows pin them at HEAD. + history: decisions.md#W3 · interacts: W1, W2, C1, H2 · implemented: nameparser/_pipeline/_script_segment.py W4. Rationale: Chinese, Japanese and Korean all write the family name first in native script — the script settles the order @@ -1548,14 +1565,17 @@ R1. Rationale: a field is a way of reading the parse, not a stored revises back to itself wherever the value's words read as the whole name read them — a glued CJK honorific peels off an initial in a bare value where the whole name kept it glued, one corpus - name of the 368 with a suffix (decisions.md#C1, 2026-09-06). A - delimiter the configuration names parts a value only where the - value's own words, read as a name, give it a tail segment for the - core to be dropped on; a run of post-nominals has none, with or - without a comma of its own, so there the delimiter stays a word of - the run — write a comma at the boundary instead. Stated without an - example line because every line here names an input string, and - this shape needs a field revised after the parse. + name of the 372 with a suffix (decisions.md#C1, 2026-09-06, which + counted 368 of 1117 then; re-measured 2026-09-10 with this + bundle's rows in the corpora, 372 of 1156 distinct names and the + same single failure). A delimiter the configuration names parts a + value only where the value's own words, read as a name, give it a + tail segment for the core to be dropped on; a run of post-nominals + has none, with or without a comma of its own, so there the + delimiter stays a word of the run — write a comma at the boundary + instead. Stated without an example line because every line here + names an input string, and this shape needs a field revised after + the parse. history: decisions.md#C1 · interacts: O3, P6, R3 · implemented: nameparser/_parser.py, nameparser/_pipeline/_post_rules.py, nameparser/_types.py R2. Rationale: callers need the surname with and without its diff --git a/docs/release_log.rst b/docs/release_log.rst index bff433aa..0cfe93ab 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -28,6 +28,8 @@ Release Log - **Fix the East Slavic and Turkic patronymic rotations overriding a declared family-first name order.** With ``patronymic_rules`` opted in and ``Policy(name_order=FAMILY_FIRST)``, ``Мицкевич Адам Юзеф`` gave last ``Адам`` through 2.2.0 and now gives last ``Мицкевич`` -- the reading the declaration asks for -- and ``oglu Ahmad Vali Ali`` with Turkic handling gave last ``Ahmad`` and now ``oglu``. The rotations exist to restore the given-first reading a family-first listing hides, so under a declared family-first order the declaration decides. No corpus name moves. See the ``O1`` entry of ``docs/design/decisions.md`` (closes #384) + - **Fix CJK honorifics and names written with a full stop of any width.** ``HumanName("김민준 씨．").suffix`` is ``씨．`` (the fullwidth stop a Japanese or Chinese IME produces by default), with last ``김`` and first ``민준``, where every release since 2.1.0 gave first ``김``, middle ``민준``, last ``씨．`` and no suffix; the ideographic ``씨。`` and halfwidth ``씨｡`` spellings move the same way, and so does a decomposed (NFD) ``김민준, 씨.`` from macOS-origin data -- the vocabulary lookup now composes NFC, so a decomposed ``Señor`` or ``née`` is recognized too. A period glued to a name word no longer breaks the name: ``양. 지훈`` gives last ``양.``, first ``지훈`` where 2.1.0 through 2.2.0 gave first ``양.``, middle ``지``, last ``훈``; ``김. 민준`` moves the same way; ``양 지훈.`` keeps the family-first order; ``김민준씨.`` and ``田中さん.`` peel their honorific as the stop-less spellings do; and a lone ``田中.`` is the family name where it read as a title. The period stays on the word it was written with -- nothing is rewritten. A Latin name written with ASCII periods is untouched: ``Smith. John`` still reads title ``Smith.``; a Latin or Cyrillic word wearing one of the three wider stops now reaches the vocabulary too, so ``Dr。 John Smith`` reads title ``Dr。`` where it read first ``Dr。``, and ``Smith, John V。`` reads suffix ``V。`` where it read middle ``V。`` -- the wide stop is not the shape the Latin initial veto reads, so the roman numeral wins where the ASCII ``V.`` stays a middle initial. This retires the 2.2.0 note below that read *period* strictly: the fullwidth ``김민준 씨．`` it named as unrecognized is recognized now. Seventeen names in the differential corpora read differently for this rule, every one of them a period-marked CJK form the corpora hold on the radar tier. One incompatibility, by decision: a ``Lexicon`` pickled by 2.1.x or 2.2.x that carries a caller-added entry the widened fold now changes -- a non-ASCII entry written with a fullwidth or ideographic stop, or in NFD -- no longer loads (``ValueError: incompatible Lexicon pickle: entries are not normalized``); the shipped vocabulary is unaffected, and the remedy is to rebuild the ``Lexicon`` from its source rather than unpickle it. See the ``cjk-full-stops`` entry of ``docs/design/decisions.md`` (closes #322, closes #323) + **Additions** - **Add the renunciate titles to the given-name title list, so a renunciate's one name is a given name.** ``HumanName("Swami Vivekananda")`` gives first ``Vivekananda`` with an empty last name, where every release since 1.4.0 gave last ``Vivekananda``; ``Guru Nanak``, ``Baba Ramdev`` and ``Lama Zopa`` move the same way, and so do the Devanagari and Bengali spellings added below. Two name words behind the title are unchanged -- ``Swami Vivekananda Saraswati`` keeps last ``Saraswati`` -- and a surname-retaining title is untouched: ``Rabbi Cohen`` still gives last ``Cohen``. ``venerable`` is deliberately not in the list, the traditions using it splitting on whether the family name survives. See the ``indic-honorifics`` entry of ``docs/design/decisions.md`` (closes #346) @@ -212,7 +214,7 @@ Release Log - **Fix glued CJK honorifics parsing as part of the name.** ``田中さん``, ``김민준씨`` and ``王小明先生`` now split the honorific off the end of the name into ``suffix``. Previously it stayed in the name: ``田中さん`` was entirely the family name, and ``김민준씨`` gave given 민준씨. The peel runs before the name is split or ordered, so ``김민준씨`` still divides into family 김, given 민준. Only entries that can never end a name peel; 양, 군, 氏, 博士 and 殿 are recognized in their spaced form only, since 김지양 is a given name and some ninety Japanese surnames end in 殿. **Default-on**, and a lone family name written with a glued honorific now divides where it did not. See :ref:`east-asian-names` for the full set and :doc:`customize` for the off-switch (closes #308) - **Fix a comma or a 间隔号 stopping the glued-honorific peel.** ``김, 민준씨`` now gives family 김, given 민준, suffix 씨, the same as the spaced ``김 민준씨``, and likewise ``田中, 太郎さん``. Previously each left the honorific inside the name. Both marks say where a name divides into surname and given, and an honorific is not part of the name in either reading. What a comma does instead is say which runs to look in: the two around a family comma. Anything past those is out of reach, so ``김, 민준 지훈씨`` peels while ``김, 민준, 지훈씨`` does not. **Default-on**. See :ref:`east-asian-names` (closes #312) - **Fix a glued honorific staying inside the name when the whole post-comma remainder is a credential.** ``田中さん, V.`` and ``田中さん, Ph. D.`` now give up さん to ``suffix`` the way ``田中さん, PhD`` already did. The peel had been taking the post-comma run for name text on the strength of the comma alone, walking into the credentials and abandoning the peel there. That run is now tested with the same rule that decides the comma structure, and declined where it is credentials, provided the part before the comma offers a peel site of its own. Where the credential itself lands is still the comma's business and still differs by spelling. **Default-on** (closes #319) - - **Fix an ASCII period after a CJK honorific stopping it being recognized.** ``씨.``, ``様.``, ``氏.``, ``님.``, ``군.``, ``양.`` and ``殿.`` now route to ``suffix`` like their periodless spellings, where the trailing period had left them inside the name. The cause was v1's initial regex, whose ``\w`` is Unicode-aware and matched a hangul syllable or Han ideograph as readily as a letter; a veto written for Latin was being asked of scripts it was never about. Alphabets keep their initials untouched, and ``"А. С. Пушкин"`` is unaffected. Read *period* strictly: only the ASCII full stop is covered, so ``"김민준 씨．"`` written with the fullwidth stop still reads the honorific as the family name. **Default-on** (#320) + - **Fix an ASCII period after a CJK honorific stopping it being recognized.** ``씨.``, ``様.``, ``氏.``, ``님.``, ``군.``, ``양.`` and ``殿.`` now route to ``suffix`` like their periodless spellings, where the trailing period had left them inside the name. The cause was v1's initial regex, whose ``\w`` is Unicode-aware and matched a hangul syllable or Han ideograph as readily as a letter; a veto written for Latin was being asked of scripts it was never about. Alphabets keep their initials untouched, and ``"А. С. Пушкин"`` is unaffected. Read *period* strictly: only the ASCII full stop is covered, so ``"김민준 씨．"`` written with the fullwidth stop still reads the honorific as the family name (superseded in 2.3.0, above). **Default-on** (#320) - **Fix NFD-decomposed input missing the East Asian defaults entirely.** Script classification now normalizes to NFC before deciding, so a Korean or Japanese name typed on macOS, where decomposed text is routine, gets the same order rule as its composed twin. Segmentation matching deliberately stays raw, so an unspaced NFD hangul name is ordered correctly but not split, rather than being split in the wrong place. One gotcha: parse output preserves the encoding it was given, so for NFD input ``name.family == "김"`` is ``False`` even though it is the same name. Compare NFC-normalized text when comparing across encodings. See :ref:`east-asian-names` (#272) - **Fix the Ukrainian conjunction й not joining the pieces around it.** It is the euphonic alternate of ``і``, chosen by the surrounding sounds rather than by meaning, so real Ukrainian data carries both spellings. ``"Олесь й Олена Коваленки"`` now gives given ``"Олесь й Олена"`` where the ``й`` previously landed in ``middle``. Same treatment as the ``и``/``і`` entries added in 2.0.0: the conjunction joins only once the name has enough pieces, and a punctuated initial still wins, so ``"Й. Сліпий"`` is unaffected. Raised in a comment on #267 - **Add the Japanese maiden-name marker 旧姓 to the default vocabulary.** ``"山田花子 旧姓 佐藤"`` now gives family ``山田花子`` and maiden ``佐藤``, where 1.4.0 left the marker in the name. It sits beside the Cyrillic ``урожд.`` and German ``geb.`` entries rather than in ``locales.JA``, on the rule that admitted those: a native-script marker cannot collide with a Latin-script name, so it is safe as a default. Matching is whole-token, so the marker has to *be* a token, which for Japanese means a space or a configured delimiter must divide it from the name. The fullwidth colon does not, so ``"山田（旧姓：佐藤）"`` still returns maiden ``"旧姓：佐藤"``; that one wants the head-peel #317 tracks. **Default-on**. See :doc:`customize` (#309) diff --git a/docs/usage.rst b/docs/usage.rst index 17faa2a4..55262267 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -722,10 +722,13 @@ rule applies there in exactly the same way: >>> parse("Morse, Det. Insp. Jane").title 'Det. Insp.' -The rule is bounded in three ways, so it doesn't swallow ordinary +The rule is bounded in four ways, so it doesn't swallow ordinary names. Single initials are left alone, so are abbreviations with -interior periods, and it applies only to that leading run — the same -word after the given name is a middle name: +interior periods, it applies only to that leading run (the same +word after the given name is a middle name), and a period-marked +opening word carrying a Han, kana or hangul character is a name word +rather than a title, since those scripts write no abbreviation with a +period (``田中.`` is the family name): .. doctest:: diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index ff654dd1..72be2519 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -10,6 +10,7 @@ import dataclasses import functools import sys +import unicodedata import warnings from collections.abc import Iterable, Mapping, Set from dataclasses import dataclass, field @@ -98,13 +99,44 @@ _PHRASE_FIELDS = ("given_name_titles", "maiden_markers") +#: Every character the parser reads as a full stop at a WORD'S EDGE +#: (#322/#323): the ASCII period, the fullwidth FULL STOP U+FF0E, the +#: IDEOGRAPHIC FULL STOP U+3002 and its halfwidth form U+FF61. One +#: string, four readers -- the lookup fold below, _vocab's script +#: classification fold, _script_segment's surname site and honorific +#: tail match -- so that "which characters are a period" is answered +#: once; the bundle's fifth site, _pieces' opening-abbreviation shape, +#: reads no stop set at all but the no-initials repertoire +#: (_policy._NO_INITIALS) that licenses every reader here. NFKC is +#: NOT a substitute for listing them: it folds U+FF0E to '.' and +#: U+FF61 to U+3002, and leaves U+3002 as it is. +#: Not every period test reads this set, and the ones that do not are +#: named so the claim is checkable: the interior-period shapes -- the +#: dotted acronym 'M.A.' and the split 'Ph. D.' (_vocab) -- and FOUR +#: ASCII-only edge tests on Latin shapes, across THREE modules -- the +#: initial 'J.' (_vocab), the bracketed credential '(Mgr.)' and the +#: word-internal apostrophe rule (_extract), and the trailing +#: middle-initial carve-out 'V.' (_assign) -- stay ASCII. That is a +#: fact about those TESTS and not about this set's reach: the fold +#: below is script-agnostic, so a LATIN word wearing a wide stop +#: reaches vocabulary the ASCII veto would have taken it out of. +#: 'V。' folds to 'v' and does not end in the ASCII period the +#: trailing-suffix carve-out tests (_assign, not is_initial), so +#: it is roman five -- suffix -- where 'V.' stays a middle +#: initial through that same carve-out; 'Jr。' simply reaches its +#: entry, and 'Dr。' its title. +#: decisions.md#cjk-full-stops carries the measurements. +FULL_STOPS = ".．。｡" + + def _normalize(word: str) -> str: - """Lowercase, strip whitespace and EDGE periods -- v1's lc() - semantics. Interior periods survive on purpose: 'J.R.' must not - collapse to 'jr' and hit the periodless vocabulary (v1 parity, - pinned live 2026-07-17). Suffix-ACRONYM membership alone uses the - period-free form (see _vocab.suffix_as_written), mirroring v1's - is_suffix, which removed periods only for the acronym test. + """Lowercase, NFC-compose, strip whitespace and EDGE full stops + (FULL_STOPS, not the ASCII period alone -- #322). Interior periods + survive on purpose: 'J.R.' must not collapse to 'jr' and hit the + periodless vocabulary (v1 parity, pinned live 2026-07-17). + Suffix-ACRONYM membership alone uses the period-free form (see + _vocab.suffix_as_written), mirroring v1's is_suffix, which removed + periods only for the acronym test. lower(), NOT casefold(): casefold's caseless-matching folds mutate the stored vocabulary itself -- 'κος' becomes the misspelling 'κοσ' @@ -114,14 +146,26 @@ def _normalize(word: str) -> str: match-time lookups, so matching stays symmetric either way; lower() is what v1's lc() used, preserving which cross-spellings match. - Strips to a FIXED POINT. A single strip().strip(".") leaves - periods-around-whitespace half done ('. a .' -> ' a '), so a value + NFC, since #322, and only for a non-ASCII word (ASCII is already + NFC, and this runs per token on the parse path -- + decisions.md#parse-cost). NFD hangul decomposes to jamo and missed + every vocabulary entry; composing at lookup AND at storage keeps + the two symmetric, and no shipped entry changes under it (measured + 2026-09-09: fold every string member of every Lexicon.default() + field through NFC and count the changes; zero of 1735). This is + the same NFC composition script classification already applies + (_vocab._normalized_for_script); token text is never rewritten. + + Strips to a FIXED POINT. A single strip().strip(FULL_STOPS) leaves + stops-around-whitespace half done ('. a .' -> ' a '), so a value that is re-normalized later -- on unpickle, or by a second add() -- would change under its owner. v1 never re-normalized, so this only matters now that storage and match-time share one fold.""" word = word.lower() + if not word.isascii(): + word = unicodedata.normalize("NFC", word) while True: - stripped = word.strip().strip(".") + stripped = word.strip().strip(FULL_STOPS) if stripped == word: return word word = stripped @@ -301,7 +345,7 @@ def _normset( if not n: raise ValueError( f"Lexicon.{field_name} entry {w!r} normalizes to empty " - f"(lowercase + strip periods/whitespace leaves nothing)" + f"(lowercase + strip full stops/whitespace leaves nothing)" ) # Every field outside _PHRASE_FIELDS is matched one word at a # time, so a multi-word entry can never match -- the library @@ -376,7 +420,7 @@ def _normpairs( if not normalized_key: raise ValueError( f"capitalization_exceptions key {k!r} normalizes to " - f"empty (lowercase + strip periods/whitespace leaves " + f"empty (lowercase + strip full stops/whitespace leaves " f"nothing)" ) # capitalized() looks words up one at a time (the _WORD regex @@ -398,15 +442,15 @@ class Lexicon: Start from :meth:`default` (the shipped vocabulary) or :meth:`empty`, derive variants with :meth:`add` / :meth:`remove` / ``|`` (union), and pass the result to ``Parser(lexicon=...)``. - Entries are normalized at construction -- lowercased, edge periods - stripped -- so matching is case-insensitive. Vocabulary entries are - single words -- a multi-word entry warns at construction and can - never match. Two fields are exempt, and they differ in HOW they - match: ``given_name_titles`` is looked up against the run of words - the parse has ALREADY read as titles -- the whole run space-joined, - or that run's last word -- while ``maiden_markers`` is matched by - lookahead, longest first, over words that need not be markers on - their own (``"z domu"``). + Entries are normalized at construction -- lowercased, NFC-composed, + edge full stops stripped -- so matching is case-insensitive. + Vocabulary entries are single words -- a multi-word entry warns at + construction and can never match. Two fields are exempt, and they + differ in HOW they match: ``given_name_titles`` is looked up + against the run of words the parse has ALREADY read as titles -- + the whole run space-joined, or that run's last word -- while + ``maiden_markers`` is matched by lookahead, longest first, over + words that need not be markers on their own (``"z domu"``). Field docs below show examples, not full contents; inspect any field's shipped vocabulary directly, e.g. ``Lexicon.default().conjunctions``.""" diff --git a/nameparser/_pipeline/_pieces.py b/nameparser/_pipeline/_pieces.py index 384a1632..9af23707 100644 --- a/nameparser/_pipeline/_pieces.py +++ b/nameparser/_pipeline/_pieces.py @@ -51,7 +51,9 @@ from typing import NamedTuple from nameparser._pipeline._state import WorkToken -from nameparser._pipeline._vocab import is_trailing_numeral_suffix +from nameparser._pipeline._vocab import ( + in_initialless_script, is_trailing_numeral_suffix, +) # rules.md#H3: "successive title words at the start of the part @@ -81,8 +83,22 @@ def is_leading_title(piece: Sequence[int], ptags: Set[str], tokens: Sequence[WorkToken]) -> bool: if is_title_piece(piece, ptags, tokens): return True - return (len(piece) == 1 - and bool(_PERIOD_ABBREV.match(tokens[piece[0]].text))) + if len(piece) != 1: + return False + text = tokens[piece[0]].text + # The shape reads a Latin convention: a period marks an + # abbreviation. Scripts with no initials have no period + # abbreviations either (_policy._NO_INITIALS, the #320 veto + # is_initial carries), so a CJK word wearing a period is a name + # word, not a title -- a lone '田中.' is the family name (#323). + # _PERIOD_ABBREV stays ASCII-period only: a word wearing '。' never + # matched it, and the veto is what makes the ASCII spelling agree. + # ASCII text can carry no _NO_INITIALS character (every range sits + # above U+3000), so the C-level test declines before the regex + # search runs -- four frames per unlisted-abbreviation opener per + # parse, is_leading_title running four times per piece. + return (bool(_PERIOD_ABBREV.match(text)) + and (text.isascii() or not in_initialless_script(text))) def leading_titles(pieces: Sequence[Sequence[int]], @@ -416,6 +432,10 @@ def trailing_titles(rest: Sequence[int], pieces: Sequence[Sequence[int]], while k > 1: idx = rest[k - 1] piece = pieces[idx] + # no #323 veto on the shape here, unlike is_leading_title's: + # the shape is ANDed with is_title_piece, so the word is listed + # vocabulary, and a listed CJK title wearing a stop should read + # as a title. if (len(piece) == 1 and _PERIOD_ABBREV.match(tokens[piece[0]].text) and is_title_piece(piece, ptags[idx], tokens)): diff --git a/nameparser/_pipeline/_script_segment.py b/nameparser/_pipeline/_script_segment.py index e17a7a01..8d3fd72b 100644 --- a/nameparser/_pipeline/_script_segment.py +++ b/nameparser/_pipeline/_script_segment.py @@ -39,11 +39,16 @@ glued-honorific peel) and W3 (the writer's divisions are respected) of docs/design/rules.md, cited at their code below; the decision chain (#308, #312, #319, the vetting bars, the measured -spaced-honorific trade) is decisions.md#W1, #W2 and #W3. Both splits make sub-slices of one token, -rewriting nothing -- spans still index the original exactly, so the -anti-#100 invariant holds by construction. The peel runs FIRST, so -suffix classification can claim the tail and the surname match or -segmenter consult sees the name rather than name-plus-honorific; its +spaced-honorific trade) is decisions.md#W1, #W2 and #W3. Both splits +make sub-slices of one token, rewriting nothing -- spans still index +the original exactly, so the anti-#100 invariant holds by +construction. Both also match on the token's CORE, its +text.rstrip(FULL_STOPS) (#323) -- rstrip and never strip, because +both offsets are measured from the text's START, so the core has to +stay a prefix of the text for the offset the match yields to land +where the match did. The peel runs FIRST, so suffix classification +can claim the tail and the surname match or segmenter consult sees +the name rather than name-plus-honorific; its ASCII bail sits above everything here, so a caller-added LATIN tail fires only on a name carrying at least one non-ASCII character (see the bail's own comment, and honorific_tails' field note). @@ -54,6 +59,7 @@ import functools from collections.abc import Sequence +from nameparser._lexicon import FULL_STOPS from nameparser._pipeline._state import ( ParseState, PendingAmbiguity, Structure, WorkToken, ) @@ -244,8 +250,9 @@ def _is_post_nominal(state: ParseState, i: int) -> bool: def _peel_site(state: ParseState, flat: Sequence[int], tails: frozenset[str]) -> tuple[int, int] | None: """Where a peel would land in the token run `flat`: the index of the - token to cut and the LENGTH of the listed tail to take off it, or - None where that run offers no peel. + token to cut and the OFFSET to cut it at (the listed tail, and any + full stops riding behind it, are what comes off the end), or None + where that run offers no peel. Two callers, one answer, which is the point of naming it. The peel itself asks it once, of the runs it decided to scan. The gate above @@ -274,13 +281,26 @@ def _peel_site(state: ParseState, flat: Sequence[int], # nothing but post-nominals, or no tokens at all return None text = state.tokens[i].text + # The tail is matched on the token's core (#323): a stop glued after + # the honorific ('김민준씨.', '田中さん.') stands between the listed + # tail and the token's end and used to defeat the match. What the + # text read instead depended on the stop: the ASCII spelling went to + # a title downstream (H2's shape is ASCII-period-only), while a + # fullwidth or ideographic stop left the whole text a lone name word + # ('田中さん。' read given). The cut lands BEFORE the tail, so the + # stops ride with the honorific piece and the token text is never + # rewritten. TRAILING only: a leading stop is not between the name + # and its honorific, and the offset returned below is the core's + # length less the tail's, one subtraction. + core = text.rstrip(FULL_STOPS) # range/cap construction identical to the surname match below, and # for the same two reasons: longest-first, and a len-1 cap that - # makes the offset interior by construction (_split's contract). - cap = min(_longest_entry(tails), len(text) - 1) + # makes the offset interior by construction (_split's contract). An + # empty or one-character core caps below 1 and the loop is empty. + cap = min(_longest_entry(tails), len(core) - 1) for length in range(cap, 0, -1): - if text[-length:] in tails: - return i, length + if core[-length:] in tails: + return i, len(core) - length return None @@ -290,19 +310,20 @@ def _peel_site(state: ParseState, flat: Sequence[int], # That the peel also reaches ACROSS a family comma is stated at # rules.md#W3 instead, which is a tolerated rule since the # 2026-09-01 comma demotion -- the crossing is what the parser does -# today, not something W2 promises. W3 also took, on 2026-09-05, the -# reading of a period a listing leaves behind -# (decisions.md#cjk-comma-demotion): a period on a SEPARATE +# today, not something W2 promises. W3 also carries the reading of a +# period a listing leaves behind (decisions.md#cjk-comma-demotion, +# amended by decisions.md#cjk-full-stops): a period on a SEPARATE # post-nominal word rides into the suffix and moves nothing ('様.' # is post-nominal-strict and the scan steps past it as it steps past -# '様'), while a period glued to the honorific's OWN token stands -# between the honorific and that token's end, so no listed tail -# matches, the peel declines, and the whole text reads as a title -# downstream ('田中さん.', '김민준씨.') -- measured 2026-09-05 and -# pinned by nothing, since neither string is a case row or a corpus -# line, so that reading can move with nothing reporting it. Neither -# is a promise; the step past the post-nominal word itself is W2's, -# above, and is. +# '様'), and since #323 a period glued to the honorific's OWN token +# rides with the honorific too -- _peel_site matches the tail through +# the trailing stop and cuts before it, so '田中さん.' divides where +# '田中さん' does, and '김민준씨.' where '김민준씨' does (both are +# case rows now; before 2026-09-10 they read as a title, measured +# 2026-09-05 and pinned by nothing, and for one commit of the #323 +# branch the surname site read '김민준씨.' as 김 + 민준씨.). Neither +# reading is a promise; the step past the post-nominal word itself +# is W2's, above, and is. def _peel_honorific_tail(state: ParseState) -> ParseState: """#308: split a listed honorific off the END of the name's last NON-POST-NOMINAL token -- 田中さん -> 田中 + さん -- and let @@ -466,20 +487,21 @@ def _peel_honorific_tail(state: ParseState) -> ParseState: runs = state.segments[:1] if state.structure is Structure.FAMILY_COMMA: second = [state.tokens[j].text for j in state.segments[1]] + # a site here is asked about, not used: the offset it carries is + # >= 1 by the cap, so a site is always truthy and None never is if not (is_wholly_suffix(second, state.lexicon, state.policy) and _peel_site(state, state.segments[0], tails)): runs = state.segments[:2] site = _peel_site(state, [j for seg in runs for j in seg], tails) if site is None: return state - i, length = site + i, offset = site # The tail carries a tag because this stage MANUFACTURED it. The # segmenter's neighbour test below needs to tell it from a token # somebody wrote, and no vocabulary question can: the two spellings # put the same word in the same place, and only the provenance # differs. - return _split(state, i, (len(state.tokens[i].text) - length,), None, - tail_tag=_PEELED_TAG) + return _split(state, i, (offset,), None, tail_tag=_PEELED_TAG) def _split_surname_site(state: ParseState) -> ParseState: @@ -525,26 +547,39 @@ def _split_surname_site(state: ParseState) -> ParseState: token = state.tokens[i] text = token.text surnames = state.lexicon.surnames + # The vocabulary match reads the token's CORE (#323). Without it + # the stop WAS the remainder: '김.' matched its own head and became + # 김 + '.'; the stop rides with the remainder instead, so '김민준.' + # divides as 김 + '민준.'. A leading stop never reaches THIS site -- + # the classification fold in front rstrips too + # (_vocab._normalized_for_script) and hides such a token before it + # arrives -- while the peel above has no such gate in front of it + # and its own rstrip is what does the work there. + core = text.rstrip(FULL_STOPS) # A token that IS a surname never splits: a bare "남궁" must not # become 남 + 궁 just because the single-syllable surname also # matches -- there is nothing to split off, and a lone token's - # role is the order resolution's call. - if text in surnames: + # role is the order resolution's call, stop or no stop. + if core in surnames: return state # Longest-first (compound-before-single falls out of it), capped # so the remainder is never empty. Direct membership, no - # _normalize: the script gate admits only CJK text, which the - # storage fold stores unchanged. An empty vocabulary skips the - # match rather than bailing the stage (_longest_entry's max() has - # nothing to take): a surname-less lexicon declines every token, - # which is exactly the condition the segmenter is consulted on, so - # an early bail would make a configured segmenter silently inert - # under Lexicon.empty() -- the JA pack's own shape. + # _normalize: the script gate admits only CJK text, and a CJK + # entry's stored form is its NFC composition with no case or + # edge-stop change (#322) -- so an entry AUTHORED in NFD is + # composed on the way in, raw NFD input matches nothing here, and + # the name goes unsplit, which is the no-split rules.md#W1 already + # accepts for NFD text. An empty vocabulary skips the match rather + # than bailing the stage (_longest_entry's max() has nothing to + # take): a surname-less lexicon declines every token, which is + # exactly the condition the segmenter is consulted on, so an early + # bail would make a configured segmenter silently inert under + # Lexicon.empty() -- the JA pack's own shape. matches: list[int] = [] if surnames: - cap = min(_longest_entry(surnames), len(text) - 1) + cap = min(_longest_entry(surnames), len(core) - 1) matches = [length for length in range(cap, 0, -1) - if text[:length] in surnames] + if core[:length] in surnames] if matches: take = matches[0] detail = None @@ -617,7 +652,12 @@ def _split_surname_site(state: ParseState) -> ParseState: # asked it, which is a fact about the CONTENT, not a broken # protocol. Bounded like every message here: the type's NAME, never # its contents. - answer = state.segmenter(text) + # The CORE, not the raw token (#323): handed the raw token, a + # segmenter answering len-1 would make the stop a piece of its own + # ('田中太郎' + '。'), where against the core the trailing stops + # ride with the last piece -- '山田太郎.' answered at 2 divides as + # 山田 + '太郎.'. + answer = state.segmenter(core) if answer is not None and not isinstance(answer, Segmentation): # a duck-typed answer carrying a .splits of its own would # otherwise wander into the split path and surface as a @@ -634,10 +674,14 @@ def _split_surname_site(state: ParseState) -> ParseState: # piece. Declining silently here (as this did before the review) # made an off-by-one segmenter undebuggable: every answer it gave # vanished, and the parse merely looked unsegmented. - if answer.splits[-1] >= len(text): + # The bound is the CORE's length, the string the segmenter was + # actually handed: an offset it could not have derived from its own + # input is the same author bug whether or not `text` is longer. + if answer.splits[-1] >= len(core): raise ValueError( f"segmenter returned splits beyond the token: last offset " - f"{answer.splits[-1]}, token length {len(text)}") + f"{answer.splits[-1]}, the segmenter was given " + f"{len(core)} characters") conf = answer.confidence detail = None if conf is not None and conf < _SEGMENTER_CONFIDENCE_FLOOR: diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index da342eff..2cb4cd1d 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -30,7 +30,7 @@ import unicodedata from collections.abc import Callable, Iterable, Sequence -from nameparser._lexicon import Lexicon, _normalize +from nameparser._lexicon import FULL_STOPS, Lexicon, _normalize from nameparser._policy import (Policy, Script, _JA_SCRIPTS, _NO_INITIALS, _SCRIPT_RANGES, _script_matcher) @@ -39,7 +39,7 @@ # sync by hand; layering forbids importing the config package here. # "Verbatim" is a promise about the PATTERN, not about the predicate: # since #320 is_initial is this SHAPE test ANDed with a repertoire test -# (_in_initialless_script, below), so _INITIAL.fullmatch(text) and +# (in_initialless_script, below), so _INITIAL.fullmatch(text) and # is_initial(text) are no longer the same question -- '씨.' answers yes # to the first and no to the second. Call is_initial; the bare pattern # is not the thing to ask. The narrowing lives in the predicate @@ -86,7 +86,12 @@ # contains-any, not whole=True: the shape half has already admitted the # trailing period, so the text reaching here is '씨.' rather than '씨' # and a wholly-of match would be False for every case this exists for. -_in_initialless_script = _script_matcher(*_NO_INITIALS, whole=False) +# The second caller, _pieces.is_leading_title, admits two or more +# characters, so contains-any there means one CJK character anywhere +# vetoes the whole word -- 'Kim김.' is refused as a title along with +# '田中.' -- and that is deliberate: a word carrying a script with no +# abbreviations is not wearing an abbreviation's period. +in_initialless_script = _script_matcher(*_NO_INITIALS, whole=False) def is_initial_shaped(text: str) -> bool: @@ -136,7 +141,7 @@ def is_initial(text: str) -> bool: 'vocab:suffix' either way, and is_suffix_lenient took it either way too. Downstream of that one strict-test No, the glued honorific in a name carrying such a token went unpeeled ('田中さん 様.').""" - return is_initial_shaped(text) and not _in_initialless_script(text) + return is_initial_shaped(text) and not in_initialless_script(text) _DOTTED = re.compile(r"(?:[^\W\d_]\.)+") @@ -398,13 +403,44 @@ def maiden_marker_run(words: Sequence[str], markers: frozenset[str]) -> int: def _normalized_for_script(text: str) -> str | None: - """The guard AND the NFC normalization single_script and + """The guard AND the two normalizations single_script and effective_script's license path both need, single-sourced so they - cannot drift: None for the two shapes neither ever classifies - (empty, and the common all-ASCII Latin token -- skipped before + cannot drift: trailing full stops are dropped (FULL_STOPS, #323), + then None for the two shapes neither ever classifies (nothing + left, and the common all-ASCII Latin token -- skipped before normalizing, since ASCII is already NFC and every _SCRIPT_RANGES entry is non-ASCII regardless), else an NFC-normalized copy. + Trailing stops, not raw: a period glued to a script-written token + ('양.', '太郎.') is not a character of any script, so classifying + raw text handed the token no script at all, and three readers + spent that None -- the surname site stepped past the family name + onto the given name ('양. 지훈' cut 지훈 in half), the order rule + fell back to positional ('양 지훈.' lost family-first), and the + segmenter's neighbour precondition missed a writer-drawn boundary + ('山田太郎 田中.' consulted the segmenter on 山田太郎 as if it stood + alone). The scripts this classifies -- every _SCRIPT_RANGES entry, + which today coincide with _policy._NO_INITIALS (#320), a + coincidence _policy says a new member must not inherit -- have no + initials and no period abbreviations, so a stop on such a token + carries no information about the word; ASCII text is stripped too, + but the guard below returns None for it regardless, so 'Smith.' + never classifies. TRAILING only, matching the surname site's own + rstrip in _script_segment (the same arithmetic, not a shared + gate -- this fold decides only whether the surname site, the order + rule and the segmenter ever see the token): a leading stop is not + a shape any script writes before a name word, and HIDING such a + token from those three readers -- no script, so no surname site, + which is what the tree before #323 did -- is safer than admitting + it. Admitted, '.김민준' classifies as hangul, becomes a surname + site, is declined by the head match (which rstrips) and falls + through to a configured segmenter, which answering offset 1 + divides it into the stop and the name. The peel is not gated by + this fold; its own rstrip carries a leading-stop token, see + _script_segment. The vocabulary fold alone reads BOTH edges + (_lexicon._normalize): '.씨' is still the honorific, and a lookup + divides nothing. + NFC, not raw: NFD input decomposes precomposed katakana onto a base character plus a COMBINING mark (U+3099/U+309A, which sit in the HIRAGANA block, not katakana's), so classifying raw NFD text @@ -412,16 +448,19 @@ def _normalized_for_script(text: str) -> str | None: also decomposes Hangul syllables onto bare jamo (U+1100-U+11FF), entirely outside the HANGUL range, so raw NFD Korean input misses the shipped family-first order rule rather than merely misfiring. - Normalizing first fixes both. This is classification-only and - read-only: the returned copy is never what gets tokenized, so - token text and spans stay exactly what the caller wrote. - - MATCHING (is_initial, suffix lookups, etc.) deliberately stays on - raw text elsewhere in this module -- unlike script classification, - NFD only ever costs a match there (a suffix word written NFD fails - to match its NFC vocabulary entry), never wrong-matches, so the - asymmetry is safe: one direction needs a fix, the other doesn't. + Normalizing first fixes both. Classification-only and read-only: + the returned copy is never what gets tokenized, so token text and + spans stay exactly what the caller wrote. + + Vocabulary MATCHING composes NFC too, since #322 + (_lexicon._normalize folds every lookup and every stored entry the + same way), so an NFD suffix word reaches its NFC entry. What stays + raw is SEGMENTATION -- the surname site's direct membership test + and the peel's tail slice index the token's own text -- where NFD + degrades to no-split, never to a wrong split (decisions.md#W1, + the 2026-07-29 ja amendment). """ + text = text.rstrip(FULL_STOPS) if not text or text.isascii(): return None return unicodedata.normalize("NFC", text) diff --git a/nameparser/config/_invariants.py b/nameparser/config/_invariants.py index 366981b6..ab08ae23 100644 --- a/nameparser/config/_invariants.py +++ b/nameparser/config/_invariants.py @@ -7,11 +7,12 @@ papers over the typo. Checking at import turns a silently-inert entry into an immediate failure. -Deliberately a weaker fold than ``_normalize``, which also strips edge -periods: entries like ``'esq.'`` are legitimate data here, and this -module cannot import ``_lexicon`` anyway (``_lexicon`` imports these -constants). The relationship checks between constants stay in the -modules that own them -- those encode facts about the data, not hygiene. +Deliberately a weaker fold than ``_normalize``, which also +NFC-composes and strips edge full stops: entries like ``'esq.'`` are +legitimate data here, and this module cannot import ``_lexicon`` +anyway (``_lexicon`` imports these constants). The relationship checks +between constants stay in the modules that own them -- those encode +facts about the data, not hygiene. Interior whitespace is checked but not forbidden. A PHRASE entry is legitimate in the two fields ``_lexicon._PHRASE_FIELDS`` names, and diff --git a/nameparser/util.py b/nameparser/util.py index 70705e40..312a9829 100644 --- a/nameparser/util.py +++ b/nameparser/util.py @@ -1,8 +1,9 @@ """v1 normalization, kept for the 1.x compatibility layer. Removed in 3.0 with the rest of that layer. The 2.0 core does NOT use -this: :func:`nameparser._lexicon._normalize` is its fold, which strips -edge periods to a fixed point where ``lc()`` strips once. +this: :func:`nameparser._lexicon._normalize` is its fold, which +NFC-composes and strips edge full stops to a fixed point where +``lc()`` strips ASCII periods once. """ diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 67bc1013..1a0a656e 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -32,6 +32,7 @@ """ from __future__ import annotations +import unicodedata from dataclasses import dataclass from nameparser import (FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, GIVEN_FIRST, @@ -126,8 +127,9 @@ class Case: #: tag -- mutually exclusive with `shape`, since a shape ADMITS a #: text to the contract and tolerated deliberately does not. Every #: composed/wrapped CJK form (a comma listing, a Latin title or - #: credential around a CJK name, and since 2026-09-05 a trailing - #: ASCII period on an honorific) is this table's ground for it, + #: credential around a CJK name, since 2026-09-05 a trailing ASCII + #: period on an honorific, and since #323 an edge full stop of any + #: width on any CJK name word) is this table's ground for it, #: not shapes 6/7's. Restricted to CJK-bearing text (`_has_cjk`): #: it exists to demote composed/wrapped CJK forms specifically, and #: a Latin row asking for it is a smell until some future arc @@ -4528,6 +4530,267 @@ def _check_cjk_shape_purity(self) -> None: "listing artifact no writing system produces. The row " "still pins #320's mechanism at HEAD", tolerated=True), + Case("ko_honorific_with_a_fullwidth_full_stop", "김민준 씨．", + {"family": "김", "given": "민준", "suffix": "씨．"}, + classification="fix(#322)", + notes="U+FF0E, the stop a Japanese or Chinese IME produces by " + "default. The lookup fold reads it as an edge period " + "(_lexicon.FULL_STOPS), so 씨． reaches the suffix entry " + "the ASCII spelling already reached; the token text " + "keeps its stop. Pins the fold on the one stop NFKC " + "would ALSO have folded -- the row beside it pins the " + "one it would not -- and earns its place as a member " + "of a set the library SHIPS, which " + "mechanisms.md#VOCABULARY-EXERCISES-FORKS says is " + "caller-visible behavior wanting a row per member, " + "unlike caller-configured vocabulary. 2.2.0 read given " + "김, middle 민준, family 씨．.", + tolerated=True), + Case("ko_honorific_with_an_ideographic_full_stop", "김민준 씨。", + {"family": "김", "given": "민준", "suffix": "씨。"}, + classification="fix(#322)", + notes="U+3002, which NFKC leaves alone -- the row that shows " + "the stop SET is what fixes #322, not a Unicode " + "normalization. 2.2.0 read given 김, middle 민준, " + "family 씨。.", + tolerated=True), + Case("ko_honorific_with_a_halfwidth_ideographic_full_stop", + "김민준 씨｡", + {"family": "김", "given": "민준", "suffix": "씨｡"}, + classification="fix(#322)", + notes="U+FF61, which NFKC folds to U+3002 and no further -- " + "so even under NFKC this spelling needs the set. 2.2.0 " + "read given 김, middle 민준, family 씨｡.", + tolerated=True), + Case("ko_honorific_written_nfd_after_a_family_comma", + unicodedata.normalize("NFD", "김민준, 씨."), + {"family": unicodedata.normalize("NFD", "김민준"), + "suffix": unicodedata.normalize("NFD", "씨.")}, + classification="fix(#322)", + notes="the lookup fold composes NFC, so decomposed 씨. reaches " + "the suffix entry (2.2.0 read title 씨.). The family " + "stays WHOLE and stays NFD: segmentation matches raw " + "text on purpose (decisions.md#W1, 2026-07-29 ja " + "amendment), so NFD degrades to no-split, never to a " + "wrong split, and no " + "token text is rewritten. Contrast 'NFD(田中さん, 様.)', " + "which matched before: Han does not decompose. NEITHER " + "tolerated NOR shape-tagged, deliberately: this table's " + "_has_cjk reads raw codepoints (jamo sit outside " + "_SCRIPT_RANGES), so a decomposed text is not CJK to the " + "purity gate and reaches no corpus; the row is a HEAD " + "pin only."), + Case("latin_title_written_nfd_is_still_a_title", + unicodedata.normalize("NFD", "Señor Juan Garcia"), + {"title": unicodedata.normalize("NFD", "Señor"), + "given": "Juan", "family": "Garcia"}, + classification="fix(#322)", + notes="the Latin side of the NFC fork: the lookup fold " + "composes every non-ASCII word, not hangul alone, so a " + "decomposed spelling of a shipped diacritic entry " + "(señor, née, attaché) reaches it. 2.2.0 read given " + "Señor, middle Juan, family Garcia. The token keeps its " + "NFD text; only the lookup composes."), + Case("ko_leading_honorific_surname_with_a_period_keeps_the_given_name", + "양. 지훈", + {"family": "양.", "given": "지훈"}, + classification="fix(#323)", + notes="the issue's own input. effective_script('양.') is " + "HANGUL once the classification fold drops the edge " + "stop, so the surname site lands on 양. -- which is " + "post-nominal vocabulary in the LEADING position, " + "which _is_post_nominal's docstring says the surname " + "site reads as an ANSWER rather than a token to step " + "past -- and declines, leaving 지훈 whole. The stop " + "stays on the token: nothing rewrites text (rules.md T " + "Background). 2.2.0 read given 양., middle 지, family " + "훈.", + tolerated=True), + Case("ko_leading_family_name_with_a_period_keeps_the_given_name", + "김. 민준", + {"family": "김.", "given": "민준"}, + classification="fix(#323)", + notes="the non-honorific twin of the row above, and the row " + "that decides between #323's two candidate fixes: " + "consulting is_suffix_strict at the surname site would " + "have rescued 양. and left this one cut (김 is no " + "suffix), so the fix is the classification fold. The " + "site lands on 김. and matches on its CORE: 김 is itself " + "a listed surname, so nothing splits and the stop stays " + "on the family name (matched on the raw text, 김 was a " + "listed HEAD and the stop was the remainder, cutting " + "김 + '.'). 2.2.0 read given 김., middle 민, " + "family 준.", + tolerated=True), + Case("ko_trailing_period_keeps_the_family_first_order", "양 지훈.", + {"family": "양", "given": "지훈."}, + classification="fix(#323)", + notes="the ORDER reader of the same None: assign's script " + "walk returns the declared order the moment one piece " + "has no script, so a stop on the LAST token flipped " + "the name to given-first. rules.md#W4's family-first " + "reading survives the stop now. 2.2.0 read given 양, " + "family 지훈..", + tolerated=True), + Case("ko_glued_honorific_with_a_period_peels", "김민준씨.", + {"family": "김", "given": "민준", "suffix": "씨."}, + classification="fix(#323)", + notes="the W3 reading that was pinned by nothing (measured " + "2026-09-05 as title 김민준씨.): the peel now matches " + "its tail through the trailing stop and cuts before it, " + "so this divides exactly as '김민준 씨.' does and the " + "stop stays on the honorific. Peel first, then the " + "surname site divides 김민준. 2.2.0 read title 김민준씨.", + tolerated=True), + Case("ja_glued_honorific_with_a_period_peels", "田中さん.", + {"family": "田中", "suffix": "さん."}, + classification="fix(#323)", + notes="the Japanese twin, and the one where nothing follows " + "the peel: 田中 is a lone Han token under the default " + "policy, so no division runs and the family stands " + "whole, as it does for '田中さん'. 2.2.0 read title " + "田中さん.", + tolerated=True), + Case("ko_lone_name_with_a_period_is_not_a_title", "김민준.", + {"family": "김", "given": "민준."}, + classification="fix(#323)", + notes="reads this way since the surname site learned the " + "core match, NOT through H2's veto: HANGUL segmentation " + "is on by default and script_segment runs before " + "assign, so 김민준. is divided into 김 + 민준. before " + "any period-marked opening word exists for H2 to see. " + "Kept as the hangul reading of the shape; the veto's " + "own witness is the Han row below, where no default " + "segmentation stands in front. The stop rides with the " + "given name -- nothing rewrites text. 2.2.0 read title " + "김민준.", + tolerated=True), + Case("ko_period_marked_first_word_then_a_name_word", "김민준. 지훈", + {"family": "김", "given": "민준.", "middle": "지훈"}, + classification="fix(#323)", + notes="the two-word hangul shape, divided by the surname " + "site before assign as the row above is: family 김, " + "given 민준., middle 지훈 -- the reading of '김민준 " + "지훈' with the stop kept. H2's veto is not what " + "decides it (see the row above); the Han two-word row " + "below is where the veto is the whole difference. " + "2.2.0 read title 김민준., family 지, given 훈 -- " + "hangul segmentation ran on the trailing word.", + tolerated=True), + Case("ja_lone_name_with_a_period_is_not_a_title", "田中.", + {"family": "田中."}, + classification="fix(#323)", + notes="the H2 veto's witness: Han has no default segmentation, " + "so the period-marked word reaches assign whole, and " + "rules.md#H2's shape -- a Latin convention, an " + "abbreviation's period -- declines it because Han has " + "no period abbreviations (the #320 veto, extended from " + "is_initial to H2). A lone name word is the family " + "name. Remove the veto and this reads title 田中. " + "again, which is what 2.2.0 read.", + tolerated=True), + Case("ja_period_marked_first_word_then_a_name_word", "田中. 太郎", + {"family": "田中.", "given": "太郎"}, + classification="fix(#323)", + notes="the two-word Han shape, where the veto is the whole " + "difference: without it H2 fires on 田中. and 太郎 " + "becomes the entire name. With it the pair reads as " + "'田中 太郎' does, family-first by script (rules.md#W4), " + "with the stop kept. The stop stays on 田中, the word " + "that carried it: family 田中., given 太郎. 2.2.0 read " + "title 田中., family 太郎. Stays tolerated like its " + "three #323 siblings, H2's Accepted clause being " + "illustrated in W3's tolerated example block rather " + "than by promoting this row, per the 2026-09-05 " + "precedent recorded in tools/differential/compare.py: " + "marking the row alone would leave the name enforced " + "and documented as demoted.", + tolerated=True), + Case("ja_katakana_lone_name_with_a_period_is_not_a_title", "マイケル.", + {"given": "マイケル."}, ambiguities=("given-or-family",), + classification="fix(#323)", + notes="the katakana arm of the H2 veto, and the one where it " + "cuts across rules.md#W4 -- a wholly-katakana name keeps " + "the declared order, so the lone word is the GIVEN name " + "-- the veto decides title-or-name, the order rule " + "decides which name. 2.2.0 read title マイケル.", + tolerated=True), + Case("latin_period_marked_opening_word_is_still_a_title", + "Smith. John", + {"title": "Smith.", "family": "John"}, + classification="parity", + notes="the Latin side of the #323 veto's fork, pinned so the " + "veto can never widen onto Latin: Latin has " + "period abbreviations, H2's shape fires, and an " + "unlisted period-marked opening word is a title even " + "when it is a surname."), + Case("cyrillic_period_marked_opening_word_is_still_a_title", + "Проф. Иванов", + {"title": "Проф.", "family": "Иванов"}, + classification="parity", + notes="the alphabet _policy._NO_INITIALS's own comment names " + "as the one it would be wrong to add: Cyrillic has " + "initials and abbreviations, so H2's shape fires and " + "the #323 veto stays out of its way. Pins the veto's " + "repertoire at the four CJK scripts from the other " + "side."), + Case("ko_name_with_a_period_in_a_bracketed_credential", + "(김민준.) John Smith", + {"given": "김", "middle": "민준. John", "family": "Smith"}, + classification="fix(#323)", + notes="a recorded DEGRADATION, pinned so it cannot move " + "silently. rules.md#S1's bracketed-credential escape in " + "_extract calls a clause suffix-shaped when it ends in " + "an ASCII period -- an unwidened test, deliberately -- " + "so the brackets are dropped and the content reads as " + "if written bare. Bare, '김민준. John Smith' used to " + "reach H2 and give title 김민준. (2.2.0's reading); " + "since #323 the veto declines a period-marked opening " + "word written in an initialless script, so 김민준. is " + "name text, the surname site divides it, and the pieces " + "spread across the Latin name -- given 김, middle " + "'민준. John', family Smith. The TRAILING spelling " + "degrades the same way ('John Smith (김민준.)' reads " + "given John, middle 'Smith 김', family 민준., where " + "2.2.0 read given John, middle Smith, family 김민준.); " + "one row is enough, the mechanism being the unwrap in " + "front rather than the position. Latin wrapped around a " + "CJK name is the 2026-09-01 demotion's own ground, so " + "the row is tolerated and the degradation is recorded " + "in decisions.md#cjk-full-stops rather than fixed here", + tolerated=True), + Case("latin_suffix_with_an_ideographic_full_stop", "John Smith, Jr。", + {"given": "John", "family": "Smith", "suffix": "Jr。"}, + classification="fix(#322)", + notes="the stop set is script-agnostic at the lookup fold, so " + "the reach it bought is not CJK-only: _normalize strips " + "any of the four FULL_STOPS off any word, and a LATIN " + "suffix wearing the ideographic stop now folds to its " + "entry. 2.2.0 read given 'Jr。' / family 'John Smith' -- " + "the stop defeated the lookup and the credential became " + "name text. Not tolerated and carrying no shape: the " + "text is Latin, which _has_cjk does not see, and the " + "row is a HEAD pin on the fold's Latin reach"), + Case("latin_roman_numeral_with_an_ideographic_full_stop", + "Smith, John V。", + {"given": "John", "family": "Smith", "suffix": "V。"}, + classification="fix(#322)", + notes="the same reach where it costs something, and the " + "asymmetry worth pinning: 'Smith, John V.' still reads " + "middle 'V.', because _reads_as_a_trailing_suffix's " + "carve-out (_assign) tests an ASCII period ending the " + "piece's own text -- text.endswith('.') -- and nothing " + "else; is_initial is not the decider: is_initial('V') " + "is True, yet 'Smith, John V', no period at all, still " + "reads suffix 'V'. The WIDE stop does not end the text " + "in an ASCII period, so the carve-out never fires, the " + "fold takes the stop off, and 'v' is roman five -- " + "suffix. 2.2.0 read middle 'V。', the whole word being " + "unfoldable then. Neither reading was designed; the " + "bundle widened the stop set and this is where the " + "widening lands on Latin text, which " + "decisions.md#cjk-full-stops records as an unasked-for " + "reach rather than a promise"), Case("ko_honorific_glued_teacher", "김선생님", {"family": "김", "suffix": "선생님"}, classification="fix(#307) + fix(#271)", @@ -5000,6 +5263,65 @@ def _check_cjk_shape_purity(self) -> None: "_peel_site's docstring derives the bound. " "1.4.0 gave first 'J.씨' / last 선생님", tolerated=True), + Case("ko_honorific_glued_family_comma_stop_on_the_first_run", + "김민준씨., J.씨", + {"family": "김민준", "suffix": "씨., J.씨"}, + classification="fix(#323)", + notes="the stop-bearing spelling of " + "ko_honorific_glued_family_comma_site_in_both_runs, " + "here because the FAMILY_COMMA decline is a gate of TWO " + "conjuncts and a trailing stop reaches exactly one of " + "them. Measured on this tree and on d37b8ec: the first " + "conjunct, is_wholly_suffix of the post-comma run, is " + "True on both -- the stop is not there. The SECOND, " + "segments[0] holding a peel site, is what the stop " + "flipped, False before #323 (the tail match ran on the " + "raw text and 김민준씨. ends in a stop, not in 씨) and " + "True now. So the decline stands, 씨. peels off 김민준씨. " + "and 'J.씨' is consumed whole. Agrees with the stop-less " + "twin field for field, the stop riding on the honorific " + "piece it arrived with (family 김민준, suffix '씨, J.씨' " + "there). 2.2.0 read given 'J.', family 김민준씨., suffix " + "씨 -- the junk 씨 peeled off 'J.씨' while the person's " + "own honorific stayed in the family name", + tolerated=True), + Case("ja_honorific_glued_family_comma_stop_on_the_first_run", + "田中さん., V.", + {"given": "V.", "family": "田中", "suffix": "さん."}, + classification="fix(#323)", + notes="the kana twin of the row above, and the same conjunct: " + "is_wholly_suffix(['V.']) is True on both trees under " + "the lenient default, and the segments[0] site is False " + "before #323 and True now. Before, the gate therefore " + "did not decline, the scan crossed to 'V.', which ends " + "in no tail, and the peel was abandoned with さん. left " + "in the family name -- 2.2.0 read given 'V.', family " + "田中さん., no suffix at all. Agrees with " + "ja_honorific_glued_family_comma_suffixy_second_run " + "field for field (given 'V.', family 田中, suffix さん " + "there), the stop riding on the honorific", + tolerated=True), + Case("ko_honorific_glued_family_comma_stop_beyond_the_comma", + "이, J.씨.", + {"given": "J.", "family": "이", "suffix": "씨."}, + classification="fix(#323)", + notes="the stop on the OTHER side of the comma, and the row " + "that keeps the two rows above from reading as 'a stop " + "moves the gate'. Measured on this tree and on d37b8ec, " + "NEITHER conjunct moves: the post-comma run is wholly " + "suffix-shaped on both, and segments[0] -- the lone 이 " + "-- holds no peel site on either, so the gate declines " + "nothing and the scan crosses the comma under #312 both " + "times. What the stop moved is the site scan itself. " + "'J.씨.' ended in no listed tail before #323, so the " + "peel found no site at all and the whole run went to " + "suffix (2.2.0 read family 이, suffix 'J.씨.'); now the " + "tail matches through the stop and the cut lands before " + "it. Agrees with " + "ko_honorific_glued_family_comma_site_only_beyond_the_" + "comma field for field (given 'J.', family 이, suffix " + "씨 there), the stop riding on 씨", + tolerated=True), Case("ko_honorific_glued_given_after_family_comma", "김, 민준씨", {"family": "김", "given": "민준", "suffix": "씨"}, classification="fix(#312)", diff --git a/tests/v2/pipeline/test_pieces.py b/tests/v2/pipeline/test_pieces.py index 462108c6..91153f53 100644 --- a/tests/v2/pipeline/test_pieces.py +++ b/tests/v2/pipeline/test_pieces.py @@ -5,13 +5,15 @@ These pin the two contracts that shape cannot reach: a defensive branch no parse can produce, and the stability its readers rest on. """ +import pytest + from nameparser._lexicon import Lexicon from nameparser._pipeline._assign import assign from nameparser._pipeline._classify import classify from nameparser._pipeline._group import group from nameparser._pipeline._pieces import ( - _numeral_behind_the_initial_veto, leading_titles, peel_trailing, - peel_walk, segment_suffix_reading, trailing_titles, + _numeral_behind_the_initial_veto, is_leading_title, leading_titles, + peel_trailing, peel_walk, segment_suffix_reading, trailing_titles, ) from nameparser._pipeline._segment import segment from nameparser._pipeline._state import ParseState @@ -231,3 +233,20 @@ def test_the_trailing_run_refuses_a_joined_piece() -> None: """ assert _trailing("John de la Prof.") == 2 assert _trailing("John Smith Prof. and Dr.") == 3 + + +@pytest.mark.parametrize("text, expected", [ + ("Xyz.", True), # H2 (rules.md): an unlisted Latin abbreviation + ("김민준.", False), # #323: hangul has no period abbreviations + ("田中.", False), # nor Han + ("たなか.", False), # nor kana + ("Kim김.", False), # contains-any, not wholly-of: one CJK char vetoes + ("J.", False), # a bare initial never was (H2 boundary) +]) +def test_leading_title_shape_refuses_an_initialless_script( + text: str, expected: bool) -> None: + # Xyz. rather than Rev. for the Latin side: Rev. is LISTED, so + # is_title_piece claims it before the shape is asked + state = _through_group(text + " Smith") + assert is_leading_title(state.pieces[0][0], state.piece_tags[0][0], + state.tokens) is expected diff --git a/tests/v2/pipeline/test_script_segment.py b/tests/v2/pipeline/test_script_segment.py index 2a0a5f84..d3ec60a3 100644 --- a/tests/v2/pipeline/test_script_segment.py +++ b/tests/v2/pipeline/test_script_segment.py @@ -349,12 +349,22 @@ def test_out_of_bounds_split_raises() -> None: # every answer vanished and the name merely looked undivided. with pytest.raises(ValueError, match=r"segmenter returned splits beyond the " - r"token: last offset 5, token length 2"): + r"token: last offset 5, the segmenter was " + r"given 2 characters"): _run("山田", policy=_JA, segmenter=_fake((5,))) # 2 on a two-character token is the boundary itself: the check is - # >=, not >, since a cut at len(text) would leave an empty piece - with pytest.raises(ValueError, match="last offset 2, token length 2"): + # >=, not >, since a cut at len(core) would leave an empty piece + with pytest.raises(ValueError, + match="last offset 2, the segmenter was given 2"): _run("山田", policy=_JA, segmenter=_fake((2,))) + # #323: the bound is the CORE's length, which is what the segmenter + # was handed -- 2 is past the end of '山田' whether or not the token + # wears a stop, and the message reports the core's length too. + # Against len(text) this answer would pass the check and make the + # stop a piece of its own ('山田' + '.'). + with pytest.raises(ValueError, + match="last offset 2, the segmenter was given 2"): + _run("山田.", policy=_JA, segmenter=_fake((2,))) def test_family_comma_gates_the_segmenter_too() -> None: @@ -408,6 +418,45 @@ def test_a_neighbour_in_an_UNACTIVATED_script_still_blocks_the_consult() -> None assert out.ambiguities == (), name +def test_a_period_marked_neighbour_still_blocks_the_consult() -> None: + # #323: the neighbour precondition reads effective_script, which + # now sees through an edge stop, so '田中.' beside 山田太郎 counts + # as the writer's own boundary and the segmenter is not consulted + # (before, the stop hid the neighbour and 山田太郎 was divided as + # if it stood alone) + asked: list[str] = [] + def seg(text: str) -> Segmentation | None: + asked.append(text) + return Segmentation((2,)) + out = _run("山田太郎 田中.", policy=_JA, lexicon=Lexicon.empty(), + segmenter=seg) + assert asked == [] + assert _texts(out) == ["山田太郎", "田中."] + + +def test_a_consulted_segmenter_receives_the_core_without_the_stop() -> None: + # #323: the segmenter is handed the same CORE the vocabulary match + # reads, not the raw token -- so a repertoire test like the ja + # pack's _wholly_japanese sees a string the stop cannot spoil, and + # no answer of the segmenter's can make the stop a piece of its + # own. A head is a prefix, so the offsets come back valid for the + # whole token and the stop rides with the last piece. + asked: list[str] = [] + def seg(text: str) -> Segmentation | None: + asked.append(text) + return None + _run("山田太郎.", policy=_JA, lexicon=Lexicon.empty(), segmenter=seg) + assert asked == ["山田太郎"] + + out = _run("山田太郎.", policy=_JA, lexicon=Lexicon.empty(), + segmenter=_fake((2,))) + assert _texts(out) == ["山田", "太郎."] + assert [(t.span.start, t.span.end) for t in out.tokens] == [ + (0, 2), (2, 5)] + assert all(out.original[t.span.start:t.span.end] == t.text + for t in out.tokens) + + def test_only_a_manufactured_tail_does_not_block_the_consult() -> None: # The exemption is PROVENANCE, not vocabulary. A tail the peel # manufactured is a boundary nobody drew -- glued 山田太郎様 was @@ -467,6 +516,70 @@ def test_peel_then_segmentation_compose() -> None: lexicon=_LEX_TAILS)) == ["김", "민준", "씨"] +@pytest.mark.parametrize("stop", [".", "．", "。", "｡", "。。"]) +def test_peels_a_listed_tail_through_a_trailing_full_stop(stop: str) -> None: + # #323 (the W3 reading that was pinned by nothing): the tail is + # matched on the token with its trailing stops removed, and the + # cut lands BEFORE the tail, so the stop rides with the honorific. + # Spans stay sub-slices of the original (anti-#100). + # a run of stops rides whole: the offset is len(core) - tail, and + # stops only lengthen the remainder. + out = _run("김민준씨" + stop, lexicon=_LEX_TAILS) + assert _texts(out) == ["김민준", "씨" + stop] + assert [(t.span.start, t.span.end) for t in out.tokens] == [ + (0, 3), (3, 4 + len(stop))] + assert all(out.original[t.span.start:t.span.end] == t.text + for t in out.tokens) + + +def test_the_peel_reads_the_trailing_stop_only() -> None: + # the rstrip in _peel_site is not strip, and this is what the + # difference costs: a LEADING stop is not between the name and its + # honorific, so folding it away would shorten the token the split + # arithmetic measures from and cut two characters early + # ('.김민' + '준씨'). The peel is licensed by the tail, not by the + # script, so it fires here whatever the classification fold said. + out = _run(".김민준씨", lexicon=_LEX_TAILS) + assert _texts(out) == [".김민준", "씨"] + assert [(t.span.start, t.span.end) for t in out.tokens] == [ + (0, 4), (4, 5)] + assert all(out.original[t.span.start:t.span.end] == t.text + for t in out.tokens) + + +def test_a_token_that_is_only_full_stops_offers_no_peel_site() -> None: + # pins the len(core) - 1 arithmetic staying inert, not a reading: + # the scan-back stops at the first non-post-nominal token, which is + # the lone stop; its core is empty, the cap goes negative and the + # match loop never runs. Passes on the pre-#323 tree too (cap 0 + # there); it is here so a cap that ever went positive on an empty + # core would be caught. + out = _run("김민준 。", lexicon=_LEX_TAILS) + assert _texts(out) == ["김민준", "。"] + + +def test_surname_site_matches_through_a_trailing_full_stop() -> None: + # #323: the head is matched on the token's core, so the stop rides + # with the remainder rather than BEING the remainder + assert _texts(_run("김민준.", policy=_HANGUL, + lexicon=_LEX)) == ["김", "민준."] + # a token whose core IS a surname never splits, stop or no stop: + # 남궁 is a compound surname and 남 a single one, so the head match + # would find 남 and leave 궁. -- the whole-token guard has to read + # the CORE, not the text, or the stop takes 남궁. out of it + assert _texts(_run("남궁.", policy=_HANGUL, lexicon=_LEX)) == ["남궁."] + # the one-character cap carries these two: len(core) - 1 is 0, so + # the match loop never runs whatever the guard above did + assert _texts(_run("김.", policy=_HANGUL, lexicon=_LEX)) == ["김."] + assert _texts(_run("김", policy=_HANGUL, lexicon=_LEX)) == ["김"] + # a LEADING stop is not a site at all: the classification fold in + # front rstrips too (_vocab._normalized_for_script), so '.김민준' + # classifies as no script and never reaches this stage. The rstrip + # here is belt and braces -- were the token to arrive, the text is + # its own core's prefix only when the stop trails. + assert _texts(_run(".김민준", policy=_HANGUL, lexicon=_LEX)) == [".김민준"] + + def test_a_token_that_is_a_tail_never_peels() -> None: # nothing to split off, and the guard is the peel site's scan past # post-nominals, not the length cap: 선생님 ENDS in the shorter @@ -476,6 +589,14 @@ def test_a_token_that_is_a_tail_never_peels() -> None: lexicon=_LEX_TAILS)) == ["さん"] assert _texts(_run("선생님", policy=_HANGUL, lexicon=_LEX_TAILS)) == ["선생님"] + # #323: since the tail is matched on the CORE, a trailing stop no + # longer stands between 님 and the end -- what keeps 선생님. whole + # is is_suffix_strict normalizing the edge stop, so the scan-back + # steps past it as a post-nominal. Without that the cap reaches + # 님 and dissects it (measured by removing the normalization). + assert _texts(_run("선생님.", policy=_HANGUL, + lexicon=_LEX_TAILS)) == ["선생님."] + assert _texts(_run("씨.", policy=_HANGUL, lexicon=_LEX_TAILS)) == ["씨."] def test_longest_tail_wins() -> None: diff --git a/tests/v2/pipeline/test_vocab.py b/tests/v2/pipeline/test_vocab.py index 2ffd6aaa..53db2029 100644 --- a/tests/v2/pipeline/test_vocab.py +++ b/tests/v2/pipeline/test_vocab.py @@ -50,6 +50,18 @@ def test_is_initial_script_repertoire() -> None: assert not is_initial("राम.") +@pytest.mark.parametrize("word", ["씨.", "씨．", "씨。", "씨｡", + unicodedata.normalize("NFD", "씨.")]) +def test_suffix_lookup_reads_every_full_stop(word: str) -> None: + # #322: the four stops and NFD all reach the one stored entry. + # is_initial stays False on every spelling: the initial veto is a + # repertoire test on the raw text (#320) and the stop set does not + # touch it. + lex = Lexicon(suffix_words=frozenset({"씨"})) + assert is_suffix_strict(word, lex) + assert not is_initial(word) + + def test_is_initial_shaped_keeps_the_shape_half_reachable() -> None: """The two halves are separately askable (#320): assign's roman-numeral fork asks the SHAPE question about the piece before a @@ -149,6 +161,20 @@ def test_strict_suffix_initial_veto() -> None: assert is_suffix_strict("Jr", _LEX) +def test_a_wide_stop_on_a_latin_word_reaches_the_vocabulary() -> None: + # #322's unasked-for Latin reach, pinned rather than argued: the + # fold strips all four FULL_STOPS off any word, while the initial + # veto reads an ASCII-period pattern alone. So the WIDE spelling of + # a Latin word gets past a veto its ASCII twin does not -- 'V。' + # folds to 'v', is not initial-shaped, and reads as roman five, + # where 'V.' stays a middle initial. 'Jr。' needs no veto argument + # and simply reaches its entry. + d = Lexicon.default() + assert is_suffix_strict("Jr。", d) + assert is_suffix_strict("V。", d) + assert not is_suffix_strict("V.", d) + + def test_ambiguous_acronym_needs_periods_and_beats_the_veto() -> None: assert is_suffix_strict("M.A.", _LEX) assert not is_suffix_strict("Ma", _LEX) @@ -380,6 +406,38 @@ def test_effective_script_kana_license() -> None: assert effective_script("") is None +def test_script_classification_ignores_edge_full_stops() -> None: + # #323: a period glued to a script-written token is not part of + # its script and must not remove the token from classification -- + # the surname site, the order rule and the segmenter's neighbour + # precondition all read this answer. Each of the four stops + # trailing; the remainder still has to be classifiable on its own. + assert effective_script("양.") is Script.HANGUL + assert effective_script("양．") is Script.HANGUL + assert effective_script("양。") is Script.HANGUL + assert effective_script("양｡") is Script.HANGUL + # TRAILING ONLY, and each of the four leading pins it. This fold + # feeds the two division sites, which index a word from its start, + # so a leading stop is the one edge classification must not hide: + # classified, '.김민준' becomes a surname site whose head match + # (rstrip, so it never matches through a leading stop) declines, + # and a configured segmenter is then handed the raw token and can + # answer offset 1 -- '.' as the given name, the name as the family. + # No script means no site, which is the pre-#323 reading and the + # no-split rules.md#W1 already accepts. + assert effective_script(".양") is None + assert effective_script("．양") is None + assert effective_script("。양") is None + assert effective_script("｡양") is None + assert single_script("太郎.") is Script.HAN + assert effective_script("高橋みなみ。") is Script.HIRAGANA # license survives + # nothing left, or ASCII left: no script, as before + assert effective_script(".") is None + assert effective_script("。") is None + assert effective_script("abc。") is None + assert single_script("Smith.") is None + + def test_resolve_script_set_generalizes_the_license_across_pieces() -> None: # a single script passes through as-is, including a script with no # order-default entry (KATAKANA): the caller decides what to do diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 3cf92310..ff84bcb6 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -43,7 +43,7 @@ class declares, which members an alternation offers. Those are exact # assert_normalized touches -- looser in the dangerous direction, and a # hand copy of a constant with a source of truth, inside the module # written to forbid exactly that. -from nameparser._lexicon import _PHRASE_FIELDS, _normalize +from nameparser._lexicon import FULL_STOPS, _PHRASE_FIELDS, _normalize from nameparser.config.bound_given_names import BOUND_GIVEN_NAMES from nameparser.config.conjunctions import CONJUNCTIONS from nameparser.config.maiden_markers import MAIDEN_MARKERS @@ -991,8 +991,9 @@ def test_case_shape_ids_exist_in_the_inventory() -> None: ("John Smith Xyz.", "John Smith Sir", "Mary Jane King", "John Smith Esq.", "Smith, Prof."), # The CJK member of the same argument is its own literal rule -- - # an alternation holding a script-classified member belongs to the - # honorific pin -- so a reach of 1 is one _CORPUS_CLAIMS cannot + # an UNDECLARED alternation holding a script-classified member + # belongs to the honorific pin, which until #322/#323 was every + # such alternation -- so a reach of 1 is one _CORPUS_CLAIMS cannot # police on its own, and these are the wall. All three are # tolerated CJK corpus names the rule must not take: a Latin # post-nominal behind a comma, the comma spelling of this very @@ -1005,6 +1006,16 @@ def test_case_shape_ids_exist_in_the_inventory() -> None: "change(suffix-acronym-collisions) esq leaves the acronym set": ("John Smith Esq", "John Smith Esq.", "Esq. Smith", "Smith, Esq.", "Esq. van Gogh"), + # The #322/#323 rule is a literal alternation of seventeen names at + # most, so _CORPUS_CLAIMS cannot see a widening into names the + # corpora lack and these probes are the wall. Each is a reading + # the bundle deliberately left where it was: the ASCII spelling + # that reached the suffix entry before #322 ever ran, a stop on a + # SEPARATE post-nominal word (rules.md#W3's unchanged case), the + # comma listing form routed by the segment gate, and the Latin H2 + # abbreviation the initialless-script veto never touches. + "fix(#322/#323)": ("김민준 씨.", "田中さん 様.", "김민준, 씨.", + "Smith. John"), } @@ -1358,6 +1369,38 @@ def _cjk_alternations(name_regex: str) -> list[set[str]]: if any(has_classified(m) for m in members)] +def _copies_honorific_vocabulary(members: set[str], + sources: frozenset[str]) -> bool: + """Whether a classified alternation is a hand copy of `sources`: + some member, read as the regex it IS, FOLDS into the vocabulary -- + it fullmatches a classified entry, or that entry wearing exactly + one full stop. + + The stop is half the criterion because #323 is what taught the + parser to read through one: a hand copy spelled '(?:씨\\.|님\\.)' + drifts from the config exactly as a bare one does, and only the + period distinguishes them. Classified entries only, the same + script derivation the pin's expected set makes -- over the whole + of SUFFIX_WORDS a Latin member would fold on 'i' or 'v' and every + list of names would read as a copy. + """ + has_classified = _policy._script_matcher(*_policy._SCRIPT_RANGES) + entries = [entry for entry in sources if has_classified(entry)] + for member in members: + try: + pattern = re.compile(member) + except re.error as exc: + raise AssertionError( + f"CJK alternation member {member!r} is not a valid regex " + f"({exc}). A mis-split alternation can produce this -- " + f"see _ALTERNATION's notes") from None + if any(pattern.fullmatch(entry) + or any(pattern.fullmatch(entry + stop) for stop in FULL_STOPS) + for entry in entries): + return True + return False + + def test_differential_honorific_rules_match_their_vocabulary() -> None: """The honorific rules' alternations are hand copies of the CJK entries of SUFFIX_WORDS (#307) and of GLUED_HONORIFICS (#308) -- @@ -1382,6 +1425,24 @@ def test_differential_honorific_rules_match_their_vocabulary() -> None: existing rather than by an author remembering to enroll it. A roster key matching no rule fails as STALE, catching an entry left behind after a rule was renamed or deleted. + + Since #322/#323 a classified member no longer implies a hand copy: + an alternation may be a list of CJK NAMES. Which alternations this + pin owns is therefore DERIVED, not declared -- a copy is one whose + members FOLD into the vocabulary (_copies_honorific_vocabulary), + and a rule folding nowhere is skipped. Skipping is not silence: an + alternation that folds and names no source still fails as + undeclared, and _NOT_A_VOCABULARY_COPY remains the escape hatch + for the residual case the fold cannot tell apart. + + The `assert found` below is non-vacuity over ALL the ledgers at + once, not per ledger: a baseline whose every CJK alternation folds + nowhere contributes nothing and the pin still passes on the + others. Today the count rests entirely on the 1.4.0 and 2.0.0 + copies -- two rules each -- the 2.1.0 and 2.2.0 ledgers carrying + only #322/#323's list of names. Delete those four and this pin + goes quiet without failing, which is what the global scope buys + and what it costs. """ has_classified = _policy._script_matcher(*_policy._SCRIPT_RANGES) used: set[str] = set() @@ -1392,12 +1453,26 @@ def test_differential_honorific_rules_match_their_vocabulary() -> None: if not isinstance(regex, str): continue for declared in _cjk_alternations(regex): + # The escape hatch the Latin twin has had since #350, + # shared with this pin: the alternation of NAMES whose + # members the fold cannot tell from a copy. Consulted + # before the fold, and empty of CJK sets today. + if frozenset(declared) in _NOT_A_VOCABULARY_COPY: + continue keys = [k for k in _HONORIFIC_SOURCES if k in rule["issue"]] + # A rule naming no key is asked about the union: an + # undeclared copy has to be CAUGHT here and fail the + # assertion below, not skipped for want of a source. + sources = frozenset().union(*( + _HONORIFIC_SOURCES[k] for k in keys or _HONORIFIC_SOURCES)) + if not _copies_honorific_vocabulary(declared, sources): + continue # a list of CJK names (#322/#323) assert len(keys) == 1, ( f"{ledger.name}: rule {rule['issue']!r} carries a CJK " - f"alternation matching {len(keys)} roster keys " - f"({keys}); every such hand copy must name exactly " - f"one source in _HONORIFIC_SOURCES") + f"alternation that folds into honorific vocabulary and " + f"matches {len(keys)} roster keys ({keys}); every such " + f"hand copy must name exactly one source in " + f"_HONORIFIC_SOURCES") used.add(keys[0]) found += 1 expected = {entry for entry in _HONORIFIC_SOURCES[keys[0]] @@ -1419,6 +1494,33 @@ def test_differential_honorific_rules_match_their_vocabulary() -> None: f"pin.") +def test_a_period_suffixed_hand_copy_still_reads_as_a_copy() -> None: + """The criterion above, driven from both sides: the adversary it + exists to catch and the rule it must not catch. + + The adversary is built FROM the config rather than spelled out, so + it cannot go stale the way a hand-written '(?:씨\\.|님\\.)' would + -- entries wearing a period are the shape a member-shape test + ("space or period means it is a name") would have waved through, + and the whole reason the criterion reads the fold instead.""" + glued = _HONORIFIC_SOURCES["cjk-glued-honorific-peel"] + has_classified = _policy._script_matcher(*_policy._SCRIPT_RANGES) + entries = sorted(entry for entry in glued if has_classified(entry))[:3] + assert len(entries) == 3 + assert _copies_honorific_vocabulary({rf"{e}\." for e in entries}, glued) + + # and the bundle's own rule, a list of names carrying periods and + # spaces of its own, folds nowhere + ledger, = (p for p in _LEDGERS if p.name == "expected_since_2.2.0.toml") + rules = [r for r in _rules(ledger) + if r["issue"].startswith("fix(#322/#323)")] + assert len(rules) == 1 + alternations = _cjk_alternations(rules[0]["name_regex"]) + assert len(alternations) == 1 + sources = frozenset().union(*_HONORIFIC_SOURCES.values()) + assert not _copies_honorific_vocabulary(alternations[0], sources) + + @pytest.mark.parametrize(("ledger_name", "issue"), [ ("expected_since_1.4.0.toml", "fix(cjk-glued-honorific-peel) glued honorific peels into suffix"), @@ -1811,6 +1913,40 @@ class _LatinCopy(NamedTuple): r"John Smith Prof\. Jr\.", r"John Smith Rev\.", r"Mary Jane King\.", r"Sir John Prof\.", r"Smith Prof\.", r"Smith Sir\.", r"Smith, John Prof\."}), + # NO CJK SET HERE TODAY, and the hatch is kept open for one + # residual: an alternation of NAMES in which some member happens + # to BE a shipped word -- a bare '양\.' would fold, 양 being + # SUFFIX_WORDS, where #322/#323's '양 지훈\.' and '양\. 지훈' do + # not. Nothing in the tree needs that today; which classified + # alternations the honorific pin owns is DERIVED instead, by + # _copies_honorific_vocabulary. + # + # WHY NOT A MEMBER-SHAPE TEST, which is what the derived criterion + # replaced an enumeration with. The obvious cheaper spelling is "a + # member carrying a space or a period is a literal name, so let it + # through" -- and it would equally have exempted + # '(?:씨\.|님\.|선생\.)', a period-suffixed hand copy of + # GLUED_HONORIFICS, from the pin that exists to keep such a copy + # in step with the config. That is the silent unpinning this + # module is written to prevent, and a shape test cannot tell the + # two apart, because the period is exactly what the bundle taught + # the parser to read THROUGH. The fold test can, which is what + # makes it stronger than either: the period-suffixed copy folds + # into the vocabulary and is caught (two of that adversary's three + # members fold, 선생 being no entry), while every member of + # #322/#323's own lists folds nowhere -- 0 of 11 and 0 of 17, + # measured 2026-09-10. + # + # WHAT A SKIP FORGOES, so nobody reads it as free, derived or + # declared. Both pins pass over the alternation, so its rule is + # checked by neither _unjustified_reach nor _top_level_alternation + # -- the #350 precedent, where the Latin twin's escape was + # introduced on the same terms. What stands in for them is + # per-rule and stronger for a list of names than a reach bound + # would be: _CORPUS_CLAIMS pins the exact set of corpus names the + # whole rule claims, with a digest, so a widened regex fails on + # the count or the digest, and _MUST_NOT_MATCH names the readings + # the bundle deliberately left alone. }) def _unjustified_reach(name_regex: str, members: set[str]) -> list[str]: @@ -1918,7 +2054,12 @@ def test_latin_alternations_mean_something_the_vocabulary_ships() -> None: continue for members in _alternations(regex): if any(has_classified(m) for m in members): - continue # the honorific pin owns these + # An UNDECLARED classified alternation is the + # honorific pin's; a DECLARED one is skipped by + # both pins, and the roster is where its answer is + # written. Either way it is not this pin's, so the + # order of these two tests does not matter. + continue if frozenset(members) in _NOT_A_VOCABULARY_COPY: continue found += 1 @@ -2258,7 +2399,7 @@ def _claim(rule: dict) -> _Claim: "fix(#432) a dotted numeral behind a name is a middle initial, not the generation": _Claim(1, ('middle', 'suffix'), "e9f282da0d0f", None), "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": - _Claim(109, ('family', 'given', 'middle'), "864f9cffa977", None), + _Claim(126, ('family', 'given', 'middle'), "a3053e6567fb", None), "fix(#274) maiden markers consumed": _Claim(33, ('family', 'maiden', 'middle'), "6f8bf7136b09", None), "fix(cjk-maiden-marker) maiden marker consumed, compounding with the CJK order flip": @@ -2341,9 +2482,9 @@ def _claim(rule: dict) -> _Claim: "fix(cjk-fullwidth-paren-nickname) fullwidth-parenthesis recognition compounds with the CJK order flip": _Claim(1, ('family', 'given', 'middle', 'nickname'), "cf370e856ae7", None), "fix(cjk-comma-honorific-peel) glued honorific peels off a post-comma given name": - _Claim(23, ('given', 'suffix'), "344de804e2c6", None), + _Claim(26, ('given', 'suffix'), "7145d3aa16ca", None), "fix(cjk-comma-compound) comma routing compounds with the CJK order flip": - _Claim(23, ('family', 'given', 'suffix', 'title'), "344de804e2c6", None), + _Claim(26, ('family', 'given', 'suffix', 'title'), "7145d3aa16ca", None), # 37 -> 35 with the 2026-09-05 narrowing, which is a rule # NARROWING and not corpus movement: the three negative # lookbehinds stop the regex matching a listed honorific @@ -2353,8 +2494,18 @@ def _claim(rule: dict) -> _Claim: # same digest. Only '김민준 박사님' changed hands -- it goes to # fix(cjk-honorific-suffix) here -- '선생님' having been the # order rule's all along. + # 40 since #323, which is 35 + 2 + 3: '김민준씨.' and '田中さん.' + # entered the corpus, and the review round after them added the + # three stop-bearing FAMILY_COMMA rows ('田中さん., V.', + # '김민준씨., J.씨', '이, J.씨.'). The regex already carried the + # optional stop, so all five move the REACH on membership, not + # on the fix. None of the five is a diff this rule explains: + # the first two carry a title role its fields do not admit, so + # the #322/#323 rule at the foot of the ledger explains them + # instead, and the three comma rows produce no diff at this + # baseline at all. `radar unclassified` stays 0. "fix(cjk-glued-honorific-peel) glued honorific peels into suffix": - _Claim(35, ('family', 'given', 'suffix'), "9a1b4c202a65", None), + _Claim(40, ('family', 'given', 'suffix'), "4d7bacfc28a4", None), "fix(cjk-honorific-suffix) postnominal honorifics recognized, compounding with the CJK order flip": _Claim(19, ('family', 'given', 'middle', 'suffix'), "aa475ddd4745", None), "feat(#269) non-Latin titles/conjunctions recognized": @@ -2503,13 +2654,33 @@ def _claim(rule: dict) -> _Claim: "562e0e82a22b", None), # The CJK member of the same argument, its own literal rule # (an alternation holding a script-classified member belongs - # to the honorific pin above). ONE corpus name, and the roles - # are the one thing that differs by ledger here: this baseline - # read the Latin word as a post-nominal and the Han words - # given-first, so all four move. + # to the honorific pin above -- until #322/#323 that was every + # such alternation, and what the pin owns is derived from the + # fold now, not declared). + # ONE corpus name, and the roles are the one thing that + # differs by ledger here: this baseline read the Latin word as + # a post-nominal and the Han words given-first, so all four + # move. "fix(#316) a trailing Latin title on a native-script name is a title": _Claim(1, ('family', 'given', 'suffix', 'title'), "567f09dc9b45", None), + # #322/#323, last in every ledger. ELEVEN corpus names here + # and at 2.0.0, SEVENTEEN at the two 2.x ledgers below (nine + # CJK movers, the katakana 'マイケル.', and the bracketed '(김민준.) + # John Smith', the last two added 2026-09-10 in the + # whole-branch and follow-up review rounds): '김. 민준', '양 지훈.' + # and '양. 지훈' move within {given, middle, family} at this + # baseline -- {family, given}, as it happens -- which + # fix(#271/#272/#298) above declares, and the three + # stop-bearing FAMILY_COMMA rows the same later review round + # added produce no diff at this baseline at all -- so none of + # the six are members of this rule's regex here, and the reach + # is smaller by six. Five roles, the union over the eleven: a + # widening that took a sixth would change the row here before + # it reached the gate. + "fix(#322/#323) a full stop on a CJK token is read as punctuation and stays on its token": + _Claim(11, ('family', 'given', 'middle', 'suffix', 'title'), + "671c6c89cf61", None), }, "expected_since_2.0.0.toml": { # #436/#437's Latin alternation, first in every ledger. @@ -2608,13 +2779,15 @@ def _claim(rule: dict) -> _Claim: "fix(#379) a tussenvoegsel after a family comma attaches to the family": _Claim(13, ('_ambiguities', 'family', 'middle'), "973617235cda", None), "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": - _Claim(109, ('_ambiguities', 'family', 'given', 'middle'), "864f9cffa977", None), + _Claim(126, ('_ambiguities', 'family', 'given', 'middle'), "a3053e6567fb", None), # 37 -> 35 with the same 2026-09-05 narrowing as the 1.4 twin, # whose entry carries the reason. Here the one name that # changed hands, '김민준 박사님', goes to the spaced rule # fix(#307/#308/#320) -- the label its title states. + # 35 -> 40 since #323 by the same five names, the 1.4 twin's + # entry carrying the arithmetic. "fix(#308/#312/#319/#320) glued CJK honorific peeled off the name into suffix": - _Claim(35, ('family', 'given', 'suffix'), "9a1b4c202a65", None), + _Claim(40, ('family', 'given', 'suffix'), "4d7bacfc28a4", None), "fix(#307/#308/#320) spaced CJK postnominal honorific routed to suffix": _Claim(16, ('family', 'given', 'middle', 'suffix'), "6d390e518bd2", None), "fix(#309) 旧姓 maiden marker consumed, compounding with the CJK order flip": @@ -2759,12 +2932,24 @@ def _claim(rule: dict) -> _Claim: _Claim(16, ('family', 'given', 'middle', 'suffix', 'title'), "562e0e82a22b", None), # The CJK member of the same argument, its own literal rule - # (an alternation holding a script-classified member belongs - # to the honorific pin above). ONE corpus name; this baseline - # reads as 1.4.0 does, so the roles are the same four. + # (an UNDECLARED alternation holding a script-classified + # member belongs to the honorific pin above -- see the 1.4.0 + # mapping's note on what #322/#323 changed there). ONE corpus + # name; this baseline reads as 1.4.0 does, so the roles are + # the same four. "fix(#316) a trailing Latin title on a native-script name is a title": _Claim(1, ('family', 'given', 'suffix', 'title'), "567f09dc9b45", None), + # #322/#323. Reach and digest as in the 1.4.0 mapping, the + # same eleven-member regex over the same corpus; that entry + # carries why six of the bundle's seventeen movers are absent + # here (all seventeen are members at 2.1.0/2.2.0). + # `_ambiguities` joins the roles here and not at 1.4.0: the + # katakana mover carries a GIVEN_OR_FAMILY ambiguity that only + # a v2-surface comparison sees. + "fix(#322/#323) a full stop on a CJK token is read as punctuation and stays on its token": + _Claim(11, ('_ambiguities', 'family', 'given', 'middle', + 'suffix', 'title'), "671c6c89cf61", None), }, # The 2.3 cycle's first rule, and a facade-only render fix: every # role is identical, so `_initials` alone. Reach and digest as in @@ -2868,14 +3053,27 @@ def _claim(rule: dict) -> _Claim: _Claim(16, ('family', 'given', 'middle', 'suffix', 'title'), "562e0e82a22b", None), # The CJK member of the same argument, its own literal rule - # (an alternation holding a script-classified member belongs - # to the honorific pin above). ONE corpus name; 2.2.0 took - # `dr` out of the post-nominal vocabulary, so the Latin word - # was a name word there and `suffix` is empty on both sides - # while `middle` moves instead. + # (an UNDECLARED alternation holding a script-classified + # member belongs to the honorific pin above -- see the 1.4.0 + # mapping's note on what #322/#323 changed there). ONE corpus + # name; 2.2.0 took `dr` out of the post-nominal vocabulary, so + # the Latin word was a name word there and `suffix` is empty + # on both sides while `middle` moves instead. "fix(#316) a trailing Latin title on a native-script name is a title": _Claim(1, ('family', 'given', 'middle', 'title'), "567f09dc9b45", None), + # #322/#323, and SEVENTEEN names rather than the eleven the + # 1.4.0 and 2.0.0 mappings record: 2.1.0 shipped the hangul + # segmentation and the script order rule, so from that + # baseline on '김. 민준', '양 지훈.' and '양. 지훈' are this bundle's + # diffs and no older rule declares them, and the three + # stop-bearing FAMILY_COMMA rows the review round added diff + # from 2.1.0 on too -- six more members, not three. + # `_ambiguities` joins the roles here as it does at 2.0.0, for + # the same katakana mover. + "fix(#322/#323) a full stop on a CJK token is read as punctuation and stays on its token": + _Claim(17, ('_ambiguities', 'family', 'given', 'middle', + 'suffix', 'title'), "ef4a7afe791a", None), }, "expected_since_2.1.0.toml": { # #436/#437's Latin alternation, first in every ledger. @@ -3107,14 +3305,21 @@ def _claim(rule: dict) -> _Claim: _Claim(16, ('family', 'given', 'middle', 'suffix', 'title'), "562e0e82a22b", None), # The CJK member of the same argument, its own literal rule - # (an alternation holding a script-classified member belongs - # to the honorific pin above). ONE corpus name, and the FEWEST - # roles of any ledger: 2.1.0 shipped the Han family-first - # order, so only the Latin word moves, out of `suffix` and - # into `title`. + # (an UNDECLARED alternation holding a script-classified + # member belongs to the honorific pin above -- see the 1.4.0 + # mapping's note on what #322/#323 changed there). ONE corpus + # name, and the FEWEST roles of any ledger: 2.1.0 shipped the + # Han family-first order, so only the Latin word moves, out of + # `suffix` and into `title`. "fix(#316) a trailing Latin title on a native-script name is a title": _Claim(1, ('suffix', 'title'), "567f09dc9b45", None), + # #322/#323. Reach, roles and digest as in the 2.2.0 mapping + # above -- the same seventeen-member regex, nothing between + # 2.1.0 and 2.2.0 having touched any of these readings. + "fix(#322/#323) a full stop on a CJK token is read as punctuation and stays on its token": + _Claim(17, ('_ambiguities', 'family', 'given', 'middle', + 'suffix', 'title'), "ef4a7afe791a", None), }, } @@ -4462,7 +4667,14 @@ class _Excluded(NamedTuple): # shape-2 slot the same matrix opened, 'Kennedy, John (Jack)', # which is the paren spelling of that clause after a family # comma and costs the entry nothing for the same reason. - _Excluded(57, "35ac9a8c4195", ()), + # 57 -> 58 on 2026-09-10 for the tolerated + # '(김민준.) John Smith', whose bracketed clause this shape + # matches -- and which costs the entry nothing again, the + # clause reading as a CREDENTIAL rather than a nickname at + # every baseline (rules.md#S1's escape unwraps it), so the + # #322/#323 rule carries the name's diff and this exclusion + # silences none of it. + _Excluded(58, "57618fbebc6b", ()), } @@ -4699,7 +4911,7 @@ def test_a_rule_reaching_no_corpus_name_says_why_it_is_kept() -> None: ("fix(nickname-typographic-pairs) two typographic quote spans read as one nickname set", "feat(#273) typographic nickname delimiters recognized by default", 1), ("fix(cjk-comma-compound) comma routing compounds with the CJK order flip", - "fix(cjk-glued-honorific-peel) glued honorific peels into suffix", 17), + "fix(cjk-glued-honorific-peel) glued honorific peels into suffix", 20), ("fix(cjk-glued-honorific-peel) glued honorific peels into suffix", "fix(suffix-routing) a two-token name ending in a roman numeral keeps it in `suffix`", 1), ("fix(cjk-glued-honorific-peel) glued honorific peels into suffix", diff --git a/tests/v2/test_lexicon.py b/tests/v2/test_lexicon.py index acfa8c64..241d0708 100644 --- a/tests/v2/test_lexicon.py +++ b/tests/v2/test_lexicon.py @@ -1,5 +1,6 @@ import dataclasses import pickle +import unicodedata import warnings from collections.abc import Callable @@ -422,6 +423,15 @@ def test_removing_a_title_leaves_its_given_name_marker_alone() -> None: @pytest.mark.parametrize("word", [ "dr.", " Dr. ", ". a .", ". .", "..x..", " .b. ", + # the three non-ASCII full stops (#322), alone and mixed with + # the ASCII one, and around whitespace like the rows above + "씨．", "씨。", "씨｡", "．씨。", " 。씨 ", "。 씨 .", + # NFD input: the fold composes, and composing twice is a no-op + unicodedata.normalize("NFD", "씨."), + unicodedata.normalize("NFD", "Müller."), + # a case-mapping that produces a combining mark: lower() then NFC + # has to land somewhere a second pass leaves alone + "İ.", ]) def test_normalize_reaches_a_fixed_point(word: str) -> None: # strip() then strip(".") leaves periods-around-whitespace half @@ -432,6 +442,77 @@ def test_normalize_reaches_a_fixed_point(word: str) -> None: assert _normalize(once) == once +def test_normalize_folds_every_full_stop_and_composes_nfd() -> None: + # #322: the vocabulary is stored NFC with edge stops removed, so a + # lookup written with a fullwidth or ideographic stop, or in NFD, + # has to fold to the same key. Edge stops only: 'J.R.' keeps its + # interior period exactly as before. + assert _normalize("씨．") == "씨" + assert _normalize("씨。") == "씨" + assert _normalize("씨｡") == "씨" + # the fold is script-agnostic, so the reach is not CJK-only: a + # LATIN word wearing an ideographic stop folds to its entry too, + # and 'Dr。' consequently reads as a title where before it read as + # a name word. Pinned here because it is the shape the tests + # elsewhere in this bundle deliberately leave in ASCII. + assert _normalize("Dr。") == "dr" + assert _normalize(unicodedata.normalize("NFD", "씨.")) == "씨" + assert _normalize("J.R.") == "j.r" + assert _normalize("김.민준") == "김.민준" + # an INTERIOR ideographic stop survives too: the fold composes NFC, + # which leaves U+3002 exactly where it is, so this pins the + # interior survival the ASCII row above pins for '.' + assert _normalize("김。민준") == "김。민준" + # NFC, NOT NFKC, and this is the row that separates them. NFKC + # folds every fullwidth character to its ASCII compatibility form, + # so it would give 'm.b.a' here -- whose period-free form is 'mba', + # a shipped suffix acronym (_vocab.suffix_as_written strips + # periods for the acronym test alone). Folding widths at the + # lookup would therefore make a fullwidth-written credential match + # the ASCII vocabulary, which is a decision this bundle did not + # take: the stop set was widened, the letters were not. + assert _normalize("Ｍ．Ｂ．Ａ．") == "ｍ．ｂ．ａ" + + +def test_an_entry_that_is_only_full_stops_is_rejected() -> None: + # the "normalizes to empty" guard reads the same stop set as the + # fold, so an ideographic stop on its own is the data bug an ASCII + # one already is + with pytest.raises(ValueError, match="normalizes to empty"): + Lexicon(titles=frozenset({"。"})) + + +def test_an_nfd_authored_entry_is_stored_composed() -> None: + # the storage fold composes NFC (#322), so a caller who authors a + # surname in NFD gets the NFC entry -- and raw NFD input, which + # segmentation matches without folding, then finds no entry and + # goes unsplit (rules.md#W1's accepted degradation). Pinned + # because the surname site's comment relies on it. + nfd = unicodedata.normalize("NFD", "김") + lex = Lexicon(surnames=frozenset({nfd})) + assert lex.surnames == frozenset({"김"}) + assert nfd not in lex.surnames + + +def test_every_shipped_entry_is_already_nfc() -> None: + # the measurement behind _normalize's docstring, as a standing + # assertion: NFC at storage changes no shipped entry, so the + # vocabulary a caller reads back is the vocabulary that was + # authored. A data module written in NFD would fail here. + # + # The roster is _VOCAB_FIELDS, which is the roster __post_init__ + # and __setstate__ normalize -- so a thirteenth vocabulary field + # has to join it for add/remove to work at all, and joining it + # gets the field checked here for free (#322/#323 review). + lex = Lexicon.default() + for field in _VOCAB_FIELDS: + for word in getattr(lex, field): + assert unicodedata.normalize("NFC", word) == word, (field, word) + for pair in lex.capitalization_exceptions: + for word in pair: + assert unicodedata.normalize("NFC", word) == word, pair + + def test_unpickling_revalidates_invariants() -> None: # the field-layout guard catches SHAPE skew, but the likeliest skew # is same-name/flipped-meaning: particles_ambiguous inverted sense diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 9d3ab647..6835775c 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -1062,10 +1062,15 @@ def test_revise_reads_a_glued_honorific_on_its_own() -> None: assert p.revise(n, suffix=n.suffix).suffix == "씨, J. 씨" -#: The one suffix-bearing corpus name whose suffix does not revise back -#: to itself: the honorific peel pinned just above. Named here so the -#: guard below fails on a NEW exception and not on the known one. -_HONORIFIC_PEEL = frozenset({"김민준씨, J.씨"}) +#: The suffix-bearing corpus names whose suffix does not revise back +#: to itself: the honorific peel pinned just above, and since +#: 2026-09-10 its stop-bearing spelling, which parses to the same +#: suffix with a stop on the first word ('씨., J.씨' revising to +#: '씨., J. 씨'). One limit, two writings of one name -- the stop +#: rides on 씨 and reaches neither the sub-parse's peel nor the entry +#: join. Named here so the guard below fails on a NEW exception and +#: not on the known one. +_HONORIFIC_PEEL = frozenset({"김민준씨, J.씨", "김민준씨., J.씨"}) def _suffix_bearing_corpus_names() -> list[str]: @@ -1075,6 +1080,29 @@ def _suffix_bearing_corpus_names() -> list[str]: if p.parse(n).suffix and n not in _HONORIFIC_PEEL] +def test_the_known_round_trip_exceptions_are_one_limit() -> None: + # What makes the two spellings ONE limit rather than two names + # somebody enrolled: each one's parsed suffix carries a word that + # ENDS in a listed honorific_tails entry without BEING one ('J.씨' + # ends in 씨), which is exactly the shape the sub-parse peels and + # the whole-name parse leaves glued. A genuinely different + # round-trip failure added to the frozenset by the same gesture + # fails here rather than riding in on the exemption. The + # characterization selects the two members and nothing else in the + # suffix-bearing corpus, measured 2026-09-10. + p = Parser() + tails = Lexicon.default().honorific_tails + + def ends_in_a_tail_without_being_one(name: str) -> bool: + return any(word not in tails + and any(word.endswith(tail) for tail in tails) + for word in p.parse(name).suffix.split()) + + assert all(ends_in_a_tail_without_being_one(n) for n in _HONORIFIC_PEEL) + assert not [n for n in _suffix_bearing_corpus_names() + if ends_in_a_tail_without_being_one(n)] + + def test_the_suffix_bearing_corpus_is_not_empty() -> None: # 367 on 2026-09-06; a floor, so a filter that empties the # parametrization below fails here instead of skipping there diff --git a/tools/differential/compare.py b/tools/differential/compare.py index 2abc58be..549a344d 100644 --- a/tools/differential/compare.py +++ b/tools/differential/compare.py @@ -572,7 +572,8 @@ def _legal_orders() -> frozenset[str]: #: decision when a corpus is added, the way the Script tables do. _CORPUS_FLOORS = { "corpus.jsonl": 480, # 486 today, from v1's banks at a pinned ref - "corpus_cjk.jsonl": 67, # 70 today, generated from the case table. + "corpus_cjk.jsonl": 67, # 70 today (2026-09-10, `wc -l`), + # generated from the case table. # LOWERED 95 -> 70 on 2026-09-01, # deliberately: the CJK comma demotion # moved 25 tolerated texts out of this @@ -585,54 +586,45 @@ def _legal_orders() -> frozenset[str]: # the trailing-period honorifics # ('田中さん 様.' and its two twins) are # the listing artifact the first - # sweep's criterion could not see - "corpus_cjk_tolerated.jsonl": 22, # 29 today, the tolerated half of - # the same generator: composed and - # wrapped CJK forms (comma listings, - # Latin titles and credentials, - # trailing ASCII periods) whose - # handling the contract stopped - # promising on 2026-09-01. 25 on the - # day it was created; the 26th is - # '지훈, 남궁민수', which had no case - # row until rules.md#W3 was demoted - # and the rules corpus stopped - # carrying it -- the row was written - # so the text moved tiers instead of - # leaving the harness. 29 since - # 2026-09-05, the three period rows. - # Floor left at 22: it guards against - # the file emptying, and this half only - # grows as the contract narrows + # sweep's criterion could not see. + # UNMOVED by the #322/#323 full-stop + # bundle (2026-09-10): every row it + # added is tolerated, so all seventeen + # of its names landed in the file below + "corpus_cjk_tolerated.jsonl": 22, # 47 today (2026-09-10, `wc -l`), + # the tolerated half of the same + # generator: composed and wrapped CJK + # forms (comma listings, Latin titles + # and credentials, trailing ASCII + # periods) whose handling the contract + # stopped promising on 2026-09-01. 47 + # since 2026-09-10, the #322/#323 + # bundle's seventeen names, every one + # tolerated and so every one on the + # radar tier; floor left at 22, which + # guards against the file emptying, + # this half only growing as the + # contract narrows "corpus_issues.jsonl": 370, # 381 today, harvested and append-only - "corpus_rules.jsonl": 150, # 249 today, generated from rules.md. - # 248 until 2026-09-05, when W2's - # trailing-period example moved into - # the tolerated W3 and the builder - # stopped harvesting it -- the seventh - # text a CJK demotion has taken out of - # this file, and (measured 2026-09-05) - # the last CJK example anywhere outside - # W3 that carried a non-space ASCII - # character. 247 -> 249 later the same - # day, when the review round restored - # W2's second half and witnessed it - # with '김민준 박사님' and '선생님' -- - # both already in corpus_cjk.jsonl, so - # the file grew and the deduped pool - # did not. - # 252 until 2026-09-01, when W3 took - # rules.md's `tolerated:` marker and - # build_rules_corpus.py stopped - # harvesting a marked rule: six comma - # texts left (W3's two, W2's two, C1's - # two) and two pure ones arrived with - # the W2 swap. Every one of the six is - # still compared and classified, from - # corpus_cjk_tolerated.jsonl above. - # Floor left at 150: it guards against - # the file emptying, and a demotion - # this size is nowhere near it + "corpus_rules.jsonl": 150, # 287 today (2026-09-10, `wc -l`), + # generated from rules.md -- the "249" + # this line read was measured on + # 2026-09-05 and never re-measured as + # the document grew (286 the day + # before). The one row this bundle + # adds is 'Smith. John', H2's Latin + # witness for the #323 veto's fork; + # the CJK witness is W3's, and a + # tolerated rule's examples are not + # harvested here. The file tracks the + # document, so a demotion shrinks it -- + # a marked rule stops being harvested + # and its texts move to + # corpus_cjk_tolerated.jsonl above, + # still compared and classified; floor + # left at 150, which guards against the + # file emptying and is nowhere near any + # demotion this document has seen "corpus_shapes.jsonl": 35, # 37 today, generated from shape-tagged # case rows. Ratcheted 27 -> 35 on # 2026-09-01 with the shape 6/7 diff --git a/tools/differential/corpus_cjk_tolerated.jsonl b/tools/differential/corpus_cjk_tolerated.jsonl index bfa80e04..8feebee3 100644 --- a/tools/differential/corpus_cjk_tolerated.jsonl +++ b/tools/differential/corpus_cjk_tolerated.jsonl @@ -1,10 +1,14 @@ +"(김민준.) John Smith" "Dr 김민준, Jr." "Dr 김민준씨, Jr." "Dr 김민준씨, V." +"マイケル." "威廉·莎士比亚, PhD" "毛 泽东 Dr." "王先生, V." "田中, 太郎さん" +"田中." +"田中. 太郎" "田中さん II" "田中さん V." "田中さん 様." @@ -14,17 +18,30 @@ "田中さん, V." "田中さん, 太郎" "田中さん, 様." +"田中さん." +"田中さん., V." "김, 민준씨" "김, 민준씨 (Jimmy)" +"김. 민준" "김민준 씨." +"김민준 씨。" +"김민준 씨．" +"김민준 씨｡" "김민준 양." "김민준, 씨" "김민준, 씨." +"김민준." +"김민준. 지훈" "김민준씨 (Jimmy)" "김민준씨 Jr." "김민준씨, J.씨" "김민준씨, V." +"김민준씨." +"김민준씨., J.씨" "남궁민수, 지훈" "선생님, J.씨" +"양 지훈." +"양. 지훈" "이, J.씨" +"이, J.씨." "지훈, 남궁민수" diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index b30b894f..623332f9 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -214,6 +214,7 @@ "Smith, PhD" "Smith, Sr." "Smith, de Mesnil Jean" +"Smith. John" "The Right Hon. the President of the Queen's Bench Division" "Van Johnson" "Vega, Juan de la" diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 02d425b0..715776f1 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -3130,13 +3130,16 @@ issue = "fix(#316) a trailing Latin title on a native-script name is a title" # buys. # # ITS OWN rule rather than a seventeenth alternative above, and not by -# preference: an alternation holding a script-classified member is -# claimed by the honorific pin in tests/v2/test_ledger_guards.py, -# which requires such an alternation's members to BE the config's CJK -# honorific entries. So this stands alone, literal-anchored, reaching -# exactly one corpus name (_CORPUS_CLAIMS carries the 1) with -# _MUST_NOT_MATCH probes for the boundary -- the same shape the four -# #436/#437 CJK rules above take, and for the same reason. +# preference: an UNDECLARED alternation holding a script-classified +# member is claimed by the honorific pin in +# tests/v2/test_ledger_guards.py, which requires such an +# alternation's members to BE the config's CJK honorific entries -- +# and until #322/#323 taught that pin to ask whether the members FOLD +# into that vocabulary, that was every such alternation. So this +# stands alone, literal-anchored, +# reaching exactly one corpus name (_CORPUS_CLAIMS carries the 1) +# with _MUST_NOT_MATCH probes for the boundary -- the same shape the +# four #436/#437 CJK rules above take, and for the same reason. # # Classified 2026-09-09, in the review round that fixed which title # run H1 addresses by; the reading itself is untouched by that fix. @@ -3154,7 +3157,8 @@ issue = "fix(#316) a trailing Latin title on a native-script name is a title" # given-first -- so `title`, `given`, `family` and `suffix` all move. # # Literal and anchored, so the reach cannot widen into the other -# twenty-nine tolerated CJK names. _MUST_NOT_MATCH carries three of +# forty-six tolerated CJK names (twenty-nine until this bundle's +# seventeen rows joined the file). _MUST_NOT_MATCH carries three of # them as probes: '王先生, V.' (a Latin post-nominal behind a # comma, which the trailing walk never sees), '田中さん, Dr.' (the # comma spelling of this very shape, routed by the segment gate) and @@ -3162,3 +3166,82 @@ issue = "fix(#316) a trailing Latin title on a native-script name is a title" # vocabulary). name_regex = "^毛 泽东 Dr\\.$" fields = ["family", "given", "suffix", "title"] + +# #322/#323, LAST in the file. The parser knew ONE full stop -- +# the ASCII period -- and knew it at ONE site, the lexicon's edge +# strip. It now reads four: the ASCII period, the fullwidth '．', +# the ideographic '。' and its halfwidth '｡'. A full stop glued +# AFTER a script-written word is punctuation and not part of the +# word -- invisible to the script reading and to the vocabulary, +# and left in the text on the word it arrived with -- because no +# East Asian script writes an initial or an abbreviation with a +# period. A leading stop folds away for the vocabulary lookup, +# which reads both edges; the classification fold that feeds the +# surname site reads the TRAILING edge only, so a word wearing a +# leading stop is given no script and never becomes a surname +# site: '.김민준' stays one whole word, given. The honorific peel +# is not gated by that fold -- the tail alone licenses it -- so a +# leading stop does not stop it: '.김민준씨' peels to '.김민준' +# and '씨' (rules.md's W Background, decisions.md#cjk-full-stops). +# +# The eleven names here are the ones whose diff carries a `title` +# or a `suffix` move -- nine CJK, the katakana 'マイケル.' (a #323 +# mover added 2026-09-10 in the whole-branch review, its diff being +# title-or-given), and the bracketed '(김민준.) John Smith', added +# in the review round after that, whose diff carries the `title` +# the unwrapped clause used to take. SIX MORE of the bundle's +# seventeen movers are absent on purpose, and for one reason in two +# shapes. '김. 민준', '양 지훈.' and '양. 지훈' move within {given, +# middle, family} at this baseline -- {family, given}, as it +# happens -- which the native-script CJK rule above already +# declares. The three stop-bearing FAMILY_COMMA rows +# the same review round added are the other shape: '田中さん., V.' and '이, J.씨.' +# are claimed here by the broad 'fix(cjk-comma-compound)' and +# 'fix(cjk-comma-honorific-peel)' rules above, and '김민준씨., J.씨' +# produces no diff at this baseline at all. Adding +# any of the six would only take a name off a rule that describes +# it, or claim a diff that does not exist. +# +# LAST is narrow-first (#382), not taste. Two earlier rules reach +# these names by regex: 'fix(#271/#272/#298) native-script CJK' +# reaches all eleven (its script-range scope catches the katakana +# mover, and the bracketed row's CJK clause) and 'fix(cjk-glued-honorific-peel)' reaches the two +# glued spellings ('田中さん.', '김민준씨.'). This rule's five +# roles are a STRICT SUPERSET of the three each of those declares, +# so both are the narrower rule and both belong ahead of it; +# order_contests sees no pair at all, a contest needing the LATER +# rule's fields to nest inside the earlier one's. +# +# An anchored alternation of the NAMES rather than a shape, and the +# two criteria a shape could be written on both reach names this +# rule must not claim. "An edge full stop on a word carrying a +# classified character" reaches 22 corpus names, measured +# 2026-09-10: the bundle's seventeen movers and FIVE that do not +# move ('田中さん 様.', '田中さん, 様.', '김민준 씨.', '김민준 양.', +# '김민준, 씨.'). Measured with the TOKENIZER and not a whitespace +# split -- two of the movers wear their stop before a comma +# ('김민준씨.,', '田中さん.,'), which a whitespace split does not +# see -- plus the bracketed '(김민준.) John Smith' by hand, its stop +# sitting inside the clause until rules.md#S1 unwraps it; an 18 +# recorded earlier the same day came from the split, over a corpus +# four names smaller. A character class of the four stops would be a hand +# copy of nameparser._lexicon.FULL_STOPS besides, and would reach +# the looser population of "an ASCII period anywhere in a CJK- +# bearing name" -- 20 distinct names on 21 corpus rows before this +# bundle ('Dr 김민준씨, Jr.' being the name on two of them), wider +# because it sweeps in periods sitting on LATIN tokens inside a CJK +# name ('毛 泽东 Dr.', '田中さん, V.'). None of the five or the +# twenty is moved BY THIS BUNDLE, so a shape spelling would +# reach names whose diffs belong to other rules (most of them +# do diff against the older baselines -- 19 of the twenty at +# 1.4.0) or to no rule at all; decisions.md#cjk-full-stops +# carries both counts and the recompute. The members copy no wordlist, +# which the honorific pin in tests/v2/test_ledger_guards.py DERIVES +# rather than taking on trust -- no member of this alternation folds +# into the CJK honorific vocabulary, bare or wearing a full stop; +# _CORPUS_CLAIMS pins the reach at 11 with its digest, and +# _MUST_NOT_MATCH pins the four readings the bundle left alone. +[[change]] +issue = "fix(#322/#323) a full stop on a CJK token is read as punctuation and stays on its token" +name_regex = "^(?:\\x28김민준\\.\\x29 John Smith|マイケル\\.|田中\\.|田中\\. 太郎|田中さん\\.|김민준 씨。|김민준 씨．|김민준 씨｡|김민준\\.|김민준\\. 지훈|김민준씨\\.)$" +fields = ["family", "given", "middle", "suffix", "title"] diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index 754da1a6..4c6e0d50 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -77,7 +77,9 @@ issue = "feat(#449) a lone name word reports given-or-family" # The five CJK-bearing names in the same population ride five rules of # their own below, because a CJK member in an alternation is claimed # by the honorific pin, which would demand this be a copy of -# GLUED_HONORIFICS. +# GLUED_HONORIFICS. #322/#323 opened the one escape from that pin -- +# a member set declared in _NOT_A_VOCABULARY_COPY -- and nothing here +# declares one, so the five rules stand. name_regex = "(?i)^(?:'Smitty' Jones Jr\\.|Andrew|Carod i|Dean of Chemistry|Donald mc|Duke of Edinburgh|Duke of Wellington|Garcia|Jack M\\.A\\.|John & Jane|John V|John of the Doe|Juan & Garcia|Juan and Garcia|Mohamad X|Smith|Smith Jr\\.|e and e|part1 of The part2 of the part3 and part4|part1 of and The part2 of the part3 And part4|test|سلمان،)$" fields = ["_ambiguities"] @@ -1995,13 +1997,16 @@ issue = "fix(#316) a trailing Latin title on a native-script name is a title" # buys. # # ITS OWN rule rather than a seventeenth alternative above, and not by -# preference: an alternation holding a script-classified member is -# claimed by the honorific pin in tests/v2/test_ledger_guards.py, -# which requires such an alternation's members to BE the config's CJK -# honorific entries. So this stands alone, literal-anchored, reaching -# exactly one corpus name (_CORPUS_CLAIMS carries the 1) with -# _MUST_NOT_MATCH probes for the boundary -- the same shape the four -# #436/#437 CJK rules above take, and for the same reason. +# preference: an UNDECLARED alternation holding a script-classified +# member is claimed by the honorific pin in +# tests/v2/test_ledger_guards.py, which requires such an +# alternation's members to BE the config's CJK honorific entries -- +# and until #322/#323 taught that pin to ask whether the members FOLD +# into that vocabulary, that was every such alternation. So this +# stands alone, literal-anchored, +# reaching exactly one corpus name (_CORPUS_CLAIMS carries the 1) +# with _MUST_NOT_MATCH probes for the boundary -- the same shape the +# four #436/#437 CJK rules above take, and for the same reason. # # Classified 2026-09-09, in the review round that fixed which title # run H1 addresses by; the reading itself is untouched by that fix. @@ -2018,7 +2023,8 @@ issue = "fix(#316) a trailing Latin title on a native-script name is a title" # suffix 'Dr.' -- so `title`, `given`, `family` and `suffix` all move. # # Literal and anchored, so the reach cannot widen into the other -# twenty-nine tolerated CJK names. _MUST_NOT_MATCH carries three of +# forty-six tolerated CJK names (twenty-nine until this bundle's +# seventeen rows joined the file). _MUST_NOT_MATCH carries three of # them as probes: '王先生, V.' (a Latin post-nominal behind a # comma, which the trailing walk never sees), '田中さん, Dr.' (the # comma spelling of this very shape, routed by the segment gate) and @@ -2026,3 +2032,91 @@ issue = "fix(#316) a trailing Latin title on a native-script name is a title" # vocabulary). name_regex = "^毛 泽东 Dr\\.$" fields = ["family", "given", "suffix", "title"] + +# #322/#323, LAST in the file. The parser knew ONE full stop -- +# the ASCII period -- and knew it at ONE site, the lexicon's edge +# strip. It now reads four: the ASCII period, the fullwidth '．', +# the ideographic '。' and its halfwidth '｡'. A full stop glued +# AFTER a script-written word is punctuation and not part of the +# word -- invisible to the script reading and to the vocabulary, +# and left in the text on the word it arrived with -- because no +# East Asian script writes an initial or an abbreviation with a +# period. A leading stop folds away for the vocabulary lookup, +# which reads both edges; the classification fold that feeds the +# surname site reads the TRAILING edge only, so a word wearing a +# leading stop is given no script and never becomes a surname +# site: '.김민준' stays one whole word, given. The honorific peel +# is not gated by that fold -- the tail alone licenses it -- so a +# leading stop does not stop it: '.김민준씨' peels to '.김민준' +# and '씨' (rules.md's W Background, decisions.md#cjk-full-stops). +# +# The eleven names here are the ones whose diff carries a `title` +# or a `suffix` move -- nine CJK, the katakana 'マイケル.' (a #323 +# mover added 2026-09-10 in the whole-branch review, its diff being +# title-or-given), and the bracketed '(김민준.) John Smith', added +# in the review round after that, whose diff carries the `title` +# the unwrapped clause used to take. SIX MORE of the bundle's +# seventeen movers are absent on purpose, and for one reason in two +# shapes. '김. 민준', '양 지훈.' and '양. 지훈' move within {given, +# middle, family} at this baseline -- {family, given}, as it +# happens -- which the native-script CJK rule above already +# declares. The three stop-bearing FAMILY_COMMA rows +# the same review round added are the other shape: not one of the three +# produces a diff at this baseline at all, so there is nothing here +# for any rule to explain. Adding +# any of the six would only take a name off a rule that describes +# it, or claim a diff that does not exist. +# +# LAST, and at THIS baseline the two earlier rules that reach these +# names sit differently from their 1.4.0 twins. 'fix(#271/#272/#298) +# native-script CJK' reaches all eleven (its script-range scope +# catches the katakana mover, and the bracketed row's CJK clause). +# Its fields -- {_ambiguities, family, given, middle} -- are a +# STRICT SUBSET of the six declared here -- {_ambiguities, family, +# given, middle, suffix, title} -- so it is the narrower rule and +# belongs ahead, which is where it already sits (#382). +# 'fix(#308/#312/#319/#320) glued CJK honorific' reaches the two +# glued spellings ('田中さん.', '김민준씨.') with {family, given, +# suffix}, a STRICT SUBSET of the six here -- so that pair is +# nested narrow-first (#382) exactly as it stands, and reversing the +# two would owe a `precedes_narrower` block. +# +# An anchored alternation of the NAMES rather than a shape, and the +# two criteria a shape could be written on both reach names this +# rule must not claim. "An edge full stop on a word carrying a +# classified character" reaches 22 corpus names, measured +# 2026-09-10: the bundle's seventeen movers and FIVE that do not +# move ('田中さん 様.', '田中さん, 様.', '김민준 씨.', '김민준 양.', +# '김민준, 씨.'). Measured with the TOKENIZER and not a whitespace +# split -- two of the movers wear their stop before a comma +# ('김민준씨.,', '田中さん.,'), which a whitespace split does not +# see -- plus the bracketed '(김민준.) John Smith' by hand, its stop +# sitting inside the clause until rules.md#S1 unwraps it; an 18 +# recorded earlier the same day came from the split, over a corpus +# four names smaller. A character class of the four stops would be a hand +# copy of nameparser._lexicon.FULL_STOPS besides, and would reach +# the looser population of "an ASCII period anywhere in a CJK- +# bearing name" -- 20 distinct names on 21 corpus rows before this +# bundle ('Dr 김민준씨, Jr.' being the name on two of them), wider +# because it sweeps in periods sitting on LATIN tokens inside a CJK +# name ('毛 泽东 Dr.', '田中さん, V.'). None of the five or the +# twenty is moved BY THIS BUNDLE, so a shape spelling would +# reach names whose diffs belong to other rules (most of them +# do diff against the older baselines -- 19 of the twenty at +# 1.4.0) or to no rule at all; decisions.md#cjk-full-stops +# carries both counts and the recompute. The members copy no wordlist, +# which the honorific pin in tests/v2/test_ledger_guards.py DERIVES +# rather than taking on trust -- no member of this alternation folds +# into the CJK honorific vocabulary, bare or wearing a full stop; +# _CORPUS_CLAIMS pins the reach at 11 with its digest, and +# _MUST_NOT_MATCH pins the four readings the bundle left alone. One +# regex, identical at 1.4.0 and here; expected_since_2.1.0.toml and +# expected_since_2.2.0.toml carry the seventeen-member spelling +# instead, 2.1.0 being the release that shipped the hangul +# segmentation three of those names move through -- and the three +# FAMILY_COMMA rows are the rest of the difference, diffing from +# 2.1.0 on and not here. +[[change]] +issue = "fix(#322/#323) a full stop on a CJK token is read as punctuation and stays on its token" +name_regex = "^(?:\\x28김민준\\.\\x29 John Smith|マイケル\\.|田中\\.|田中\\. 太郎|田中さん\\.|김민준 씨。|김민준 씨．|김민준 씨｡|김민준\\.|김민준\\. 지훈|김민준씨\\.)$" +fields = ["_ambiguities", "family", "given", "middle", "suffix", "title"] diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index 93198433..6b4231a5 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -101,7 +101,9 @@ issue = "feat(#449) a lone name word reports given-or-family" # The five CJK-bearing names in the same population ride five rules of # their own below, because a CJK member in an alternation is claimed # by the honorific pin, which would demand this be a copy of -# GLUED_HONORIFICS. +# GLUED_HONORIFICS. #322/#323 opened the one escape from that pin -- +# a member set declared in _NOT_A_VOCABULARY_COPY -- and nothing here +# declares one, so the five rules stand. name_regex = "(?i)^(?:'Smitty' Jones Jr\\.|Andrew|Carod i|Dean of Chemistry|Donald mc|Duke of Edinburgh|Duke of Wellington|Garcia|Jack M\\.A\\.|John & Jane|John V|John of the Doe|Juan & Garcia|Juan and Garcia|Mohamad X|Smith|Smith Jr\\.|e and e|part1 of The part2 of the part3 and part4|part1 of and The part2 of the part3 And part4|test|سلمان،)$" fields = ["_ambiguities"] @@ -205,10 +207,12 @@ name_regex = "(?i)^(?:Attorney General of Minnesota|Deputy Secretary of State)$" fields = ["_ambiguities"] # The four CJK names of #436/#437's class, one rule each. Not one -# alternation: an alternation holding a script-classified member is -# claimed by the honorific pin in tests/v2/test_ledger_guards.py, -# which requires its members to BE the config's CJK honorific -# entries. These are names, so each stands alone, literal-anchored, +# alternation: an UNDECLARED alternation holding a script-classified +# member is claimed by the honorific pin in +# tests/v2/test_ledger_guards.py, which requires its members to BE +# the config's CJK honorific entries -- and until #322/#323 opened +# the _NOT_A_VOCABULARY_COPY escape that was every such alternation. +# These are names, so each stands alone, literal-anchored, # reaching exactly one corpus name (_CORPUS_CLAIMS carries the 1) # with _MUST_NOT_MATCH probes for the boundary each one has. # rules.md#R1: "a run of post-nominals written with spaces renders @@ -1914,13 +1918,16 @@ issue = "fix(#316) a trailing Latin title on a native-script name is a title" # buys. # # ITS OWN rule rather than a seventeenth alternative above, and not by -# preference: an alternation holding a script-classified member is -# claimed by the honorific pin in tests/v2/test_ledger_guards.py, -# which requires such an alternation's members to BE the config's CJK -# honorific entries. So this stands alone, literal-anchored, reaching -# exactly one corpus name (_CORPUS_CLAIMS carries the 1) with -# _MUST_NOT_MATCH probes for the boundary -- the same shape the four -# #436/#437 CJK rules above take, and for the same reason. +# preference: an UNDECLARED alternation holding a script-classified +# member is claimed by the honorific pin in +# tests/v2/test_ledger_guards.py, which requires such an +# alternation's members to BE the config's CJK honorific entries -- +# and until #322/#323 taught that pin to ask whether the members FOLD +# into that vocabulary, that was every such alternation. So this +# stands alone, literal-anchored, +# reaching exactly one corpus name (_CORPUS_CLAIMS carries the 1) +# with _MUST_NOT_MATCH probes for the boundary -- the same shape the +# four #436/#437 CJK rules above take, and for the same reason. # # Classified 2026-09-09, in the review round that fixed which title # run H1 addresses by; the reading itself is untouched by that fix. @@ -1938,7 +1945,8 @@ issue = "fix(#316) a trailing Latin title on a native-script name is a title" # `suffix` and into `title`. # # Literal and anchored, so the reach cannot widen into the other -# twenty-nine tolerated CJK names. _MUST_NOT_MATCH carries three of +# forty-six tolerated CJK names (twenty-nine until this bundle's +# seventeen rows joined the file). _MUST_NOT_MATCH carries three of # them as probes: '王先生, V.' (a Latin post-nominal behind a # comma, which the trailing walk never sees), '田中さん, Dr.' (the # comma spelling of this very shape, routed by the segment gate) and @@ -1946,3 +1954,74 @@ issue = "fix(#316) a trailing Latin title on a native-script name is a title" # vocabulary). name_regex = "^毛 泽东 Dr\\.$" fields = ["suffix", "title"] + +# #322/#323, LAST in the file. The parser knew ONE full stop -- +# the ASCII period -- and knew it at ONE site, the lexicon's edge +# strip. It now reads four: the ASCII period, the fullwidth '．', +# the ideographic '。' and its halfwidth '｡'. A full stop glued +# AFTER a script-written word is punctuation and not part of the +# word -- invisible to the script reading and to the vocabulary, +# and left in the text on the word it arrived with -- because no +# East Asian script writes an initial or an abbreviation with a +# period. A leading stop folds away for the vocabulary lookup, +# which reads both edges; the classification fold that feeds the +# surname site reads the TRAILING edge only, so a word wearing a +# leading stop is given no script and never becomes a surname +# site: '.김민준' stays one whole word, given. The honorific peel +# is not gated by that fold -- the tail alone licenses it -- so a +# leading stop does not stop it: '.김민준씨' peels to '.김민준' +# and '씨' (rules.md's W Background, decisions.md#cjk-full-stops). +# +# SEVENTEEN members here where the 1.4.0 and 2.0.0 ledgers carry +# eleven -- twelve CJK, the katakana 'マイケル.' (a #323 mover added +# 2026-09-10 in the whole-branch review), and four the review round +# after that added: the three stop-bearing FAMILY_COMMA rows +# ('田中さん., V.', '김민준씨., J.씨', '이, J.씨.') and the bracketed +# '(김민준.) John Smith', which is the one of the four the older +# ledgers carry too. 2.1.0 is the release that shipped the hangul +# segmentation and the script order rule, so from here '김. 민준', +# '양 지훈.' and '양. 지훈' are diffs of this bundle rather than of +# that one: the older ledgers hand them to their native-script CJK +# rule, which does not exist at this baseline. The three comma rows +# are absent there for a different reason -- at 2.0.0 none of them +# diffs at all, and at 1.4.0 two are claimed by that ledger's broad +# comma rules. +# +# No earlier rule in this file reaches any of the seventeen, so LAST +# is uniformity with the other three ledgers rather than a +# narrow-first ordering (#382) against anything: order_contests +# reports no pair here in either arrangement. +# +# An anchored alternation of the NAMES rather than a shape, and the +# two criteria a shape could be written on both reach names this +# rule must not claim. "An edge full stop on a word carrying a +# classified character" reaches 22 corpus names, measured +# 2026-09-10: the bundle's seventeen movers and FIVE that do not +# move ('田中さん 様.', '田中さん, 様.', '김민준 씨.', '김민준 양.', +# '김민준, 씨.'). Measured with the TOKENIZER and not a whitespace +# split -- two of the movers wear their stop before a comma +# ('김민준씨.,', '田中さん.,'), which a whitespace split does not +# see -- plus the bracketed '(김민준.) John Smith' by hand, its stop +# sitting inside the clause until rules.md#S1 unwraps it; an 18 +# recorded earlier the same day came from the split, over a corpus +# four names smaller. A character class of the four stops would be a hand +# copy of nameparser._lexicon.FULL_STOPS besides, and would reach +# the looser population of "an ASCII period anywhere in a CJK- +# bearing name" -- 20 distinct names on 21 corpus rows before this +# bundle ('Dr 김민준씨, Jr.' being the name on two of them), wider +# because it sweeps in periods sitting on LATIN tokens inside a CJK +# name ('毛 泽东 Dr.', '田中さん, V.'). None of the five or the +# twenty is moved BY THIS BUNDLE, so a shape spelling would +# reach names whose diffs belong to other rules (most of them +# do diff against the older baselines -- 19 of the twenty at +# 1.4.0) or to no rule at all; decisions.md#cjk-full-stops +# carries both counts and the recompute. The members copy no wordlist, +# which the honorific pin in tests/v2/test_ledger_guards.py DERIVES +# rather than taking on trust -- no member of this alternation folds +# into the CJK honorific vocabulary, bare or wearing a full stop; +# _CORPUS_CLAIMS pins the reach at 17 with its digest, and +# _MUST_NOT_MATCH pins the four readings the bundle left alone. +[[change]] +issue = "fix(#322/#323) a full stop on a CJK token is read as punctuation and stays on its token" +name_regex = "^(?:\\x28김민준\\.\\x29 John Smith|マイケル\\.|田中\\.|田中\\. 太郎|田中さん\\.|田中さん\\., V\\.|김\\. 민준|김민준 씨。|김민준 씨．|김민준 씨｡|김민준\\.|김민준\\. 지훈|김민준씨\\.|김민준씨\\., J\\.씨|양 지훈\\.|양\\. 지훈|이, J\\.씨\\.)$" +fields = ["_ambiguities", "family", "given", "middle", "suffix", "title"] diff --git a/tools/differential/expected_since_2.2.0.toml b/tools/differential/expected_since_2.2.0.toml index 1a499ba8..96073432 100644 --- a/tools/differential/expected_since_2.2.0.toml +++ b/tools/differential/expected_since_2.2.0.toml @@ -90,7 +90,9 @@ issue = "feat(#449) a lone name word reports given-or-family" # The five CJK-bearing names in the same population ride five rules of # their own below, because a CJK member in an alternation is claimed # by the honorific pin, which would demand this be a copy of -# GLUED_HONORIFICS. +# GLUED_HONORIFICS. #322/#323 opened the one escape from that pin -- +# a member set declared in _NOT_A_VOCABULARY_COPY -- and nothing here +# declares one, so the five rules stand. name_regex = "(?i)^(?:'Smitty' Jones Jr\\.|Andrew|Carod i|Dean of Chemistry|Donald mc|Duke of Edinburgh|Duke of Wellington|Garcia|Jack M\\.A\\.|John & Jane|John V|John of the Doe|Juan & Garcia|Juan and Garcia|Mohamad X|Smith|Smith Jr\\.|e and e|part1 of The part2 of the part3 and part4|part1 of and The part2 of the part3 And part4|test|سلمان،)$" fields = ["_ambiguities"] @@ -194,10 +196,12 @@ name_regex = "(?i)^(?:Attorney General of Minnesota|Deputy Secretary of State)$" fields = ["_ambiguities"] # The four CJK names of #436/#437's class, one rule each. Not one -# alternation: an alternation holding a script-classified member is -# claimed by the honorific pin in tests/v2/test_ledger_guards.py, -# which requires its members to BE the config's CJK honorific -# entries. These are names, so each stands alone, literal-anchored, +# alternation: an UNDECLARED alternation holding a script-classified +# member is claimed by the honorific pin in +# tests/v2/test_ledger_guards.py, which requires its members to BE +# the config's CJK honorific entries -- and until #322/#323 opened +# the _NOT_A_VOCABULARY_COPY escape that was every such alternation. +# These are names, so each stands alone, literal-anchored, # reaching exactly one corpus name (_CORPUS_CLAIMS carries the 1) # with _MUST_NOT_MATCH probes for the boundary each one has. # rules.md#R1: "a run of post-nominals written with spaces renders @@ -555,13 +559,16 @@ issue = "fix(#316) a trailing Latin title on a native-script name is a title" # buys. # # ITS OWN rule rather than a seventeenth alternative above, and not by -# preference: an alternation holding a script-classified member is -# claimed by the honorific pin in tests/v2/test_ledger_guards.py, -# which requires such an alternation's members to BE the config's CJK -# honorific entries. So this stands alone, literal-anchored, reaching -# exactly one corpus name (_CORPUS_CLAIMS carries the 1) with -# _MUST_NOT_MATCH probes for the boundary -- the same shape the four -# #436/#437 CJK rules above take, and for the same reason. +# preference: an UNDECLARED alternation holding a script-classified +# member is claimed by the honorific pin in +# tests/v2/test_ledger_guards.py, which requires such an +# alternation's members to BE the config's CJK honorific entries -- +# and until #322/#323 taught that pin to ask whether the members FOLD +# into that vocabulary, that was every such alternation. So this +# stands alone, literal-anchored, +# reaching exactly one corpus name (_CORPUS_CLAIMS carries the 1) +# with _MUST_NOT_MATCH probes for the boundary -- the same shape the +# four #436/#437 CJK rules above take, and for the same reason. # # Classified 2026-09-09, in the review round that fixed which title # run H1 addresses by; the reading itself is untouched by that fix. @@ -580,7 +587,8 @@ issue = "fix(#316) a trailing Latin title on a native-script name is a title" # `given`, `middle` and `family` move; `suffix` is empty on both sides. # # Literal and anchored, so the reach cannot widen into the other -# twenty-nine tolerated CJK names. _MUST_NOT_MATCH carries three of +# forty-six tolerated CJK names (twenty-nine until this bundle's +# seventeen rows joined the file). _MUST_NOT_MATCH carries three of # them as probes: '王先生, V.' (a Latin post-nominal behind a # comma, which the trailing walk never sees), '田中さん, Dr.' (the # comma spelling of this very shape, routed by the segment gate) and @@ -588,3 +596,83 @@ issue = "fix(#316) a trailing Latin title on a native-script name is a title" # vocabulary). name_regex = "^毛 泽东 Dr\\.$" fields = ["family", "given", "middle", "title"] + +# #322/#323, LAST in the file. The parser knew ONE full stop -- +# the ASCII period -- and knew it at ONE site, the lexicon's edge +# strip. It now reads four: the ASCII period, the fullwidth '．', +# the ideographic '。' and its halfwidth '｡'. A full stop glued +# AFTER a script-written word is punctuation and not part of the +# word -- invisible to the script reading and to the vocabulary, +# and left in the text on the word it arrived with -- because no +# East Asian script writes an initial or an abbreviation with a +# period. A leading stop folds away for the vocabulary lookup, +# which reads both edges; the classification fold that feeds the +# surname site reads the TRAILING edge only, so a word wearing a +# leading stop is given no script and never becomes a surname +# site: '.김민준' stays one whole word, given. The honorific peel +# is not gated by that fold -- the tail alone licenses it -- so a +# leading stop does not stop it: '.김민준씨' peels to '.김민준' +# and '씨' (rules.md's W Background, decisions.md#cjk-full-stops). +# +# SEVENTEEN members, the same set and the same regex as +# expected_since_2.1.0.toml: nothing between 2.1.0 and 2.2.0 touched +# any of these readings, so the two ledgers report the same diffs -- +# twelve CJK, the katakana 'マイケル.' (a #323 mover added 2026-09-10 +# in the whole-branch review), the three stop-bearing FAMILY_COMMA +# rows and the bracketed '(김민준.) John Smith', all four from the +# review round after it. The 1.4.0 and 2.0.0 ledgers carry eleven, +# those baselines predating the hangul segmentation that puts +# '김. 민준', '양 지훈.' and '양. 지훈' in this bundle's hands rather +# than the native-script rule's, and answering the three comma rows +# with no diff (2.0.0) or with their own broad comma rules +# (1.4.0). +# +# No earlier rule in this file reaches any of the seventeen, so LAST +# is uniformity with the other three ledgers rather than a +# narrow-first ordering (#382) against anything: order_contests +# reports no pair here in either arrangement. +# +# An anchored alternation of the NAMES rather than a shape, and the +# two criteria a shape could be written on both reach names this +# rule must not claim. "An edge full stop on a word carrying a +# classified character" reaches 22 corpus names, measured +# 2026-09-10: the bundle's seventeen movers and FIVE that do not +# move ('田中さん 様.', '田中さん, 様.', '김민준 씨.', '김민준 양.', +# '김민준, 씨.'). Measured with the TOKENIZER and not a whitespace +# split -- two of the movers wear their stop before a comma +# ('김민준씨.,', '田中さん.,'), which a whitespace split does not +# see -- plus the bracketed '(김민준.) John Smith' by hand, its stop +# sitting inside the clause until rules.md#S1 unwraps it; an 18 +# recorded earlier the same day came from the split, over a corpus +# four names smaller. A character class of the four stops would be a hand +# copy of nameparser._lexicon.FULL_STOPS besides, and would reach +# the looser population of "an ASCII period anywhere in a CJK- +# bearing name" -- 20 distinct names on 21 corpus rows before this +# bundle ('Dr 김민준씨, Jr.' being the name on two of them), wider +# because it sweeps in periods sitting on LATIN tokens inside a CJK +# name ('毛 泽东 Dr.', '田中さん, V.'). None of the five or the +# twenty is moved BY THIS BUNDLE, so a shape spelling would +# reach names whose diffs belong to other rules (most of them +# do diff against the older baselines -- 19 of the twenty at +# 1.4.0) or to no rule at all; decisions.md#cjk-full-stops +# carries both counts and the recompute. The members copy no wordlist, +# which the honorific pin in tests/v2/test_ledger_guards.py DERIVES +# rather than taking on trust -- no member of this alternation folds +# into the CJK honorific vocabulary, bare or wearing a full stop; +# _CORPUS_CLAIMS pins the reach at 17 with its digest, and +# _MUST_NOT_MATCH pins the four readings the bundle left alone. +# +# Script-classified members inside an ALTERNATION are new here -- +# every other CJK rule in the four ledgers is a lone literal, and +# the only classified alternations are the honorific rules' hand +# copies of GLUED_HONORIFICS and SUFFIX_WORDS, which the honorific +# pin owns. That pin passes over this one on a MEASUREMENT rather +# than a declaration: no member of it folds into the honorific +# vocabulary, bare or wearing a full stop. +# The _SPAN_BEARING_RULES note in this file's header still holds +# either way: the members are literal names, not a hand copy of +# _SCRIPT_RANGES, so the empty set stands. +[[change]] +issue = "fix(#322/#323) a full stop on a CJK token is read as punctuation and stays on its token" +name_regex = "^(?:\\x28김민준\\.\\x29 John Smith|マイケル\\.|田中\\.|田中\\. 太郎|田中さん\\.|田中さん\\., V\\.|김\\. 민준|김민준 씨。|김민준 씨．|김민준 씨｡|김민준\\.|김민준\\. 지훈|김민준씨\\.|김민준씨\\., J\\.씨|양 지훈\\.|양\\. 지훈|이, J\\.씨\\.)$" +fields = ["_ambiguities", "family", "given", "middle", "suffix", "title"]