CJK full stops: honorifics and names written with a period of any width (#322, #323)

[derek73]

Summary

The parser knew one full stop (ASCII) at one site (the lexicon's edge strip). It now knows four -- ., ． U+FF0E, 。 U+3002, ｡ U+FF61 -- through one constant, _lexicon.FULL_STOPS, read at four sites plus a fifth that reads the same no-initials repertoire. The period never leaves the text: it stays on the token it arrived with.

• Honorifics written with a fullwidth or ideographic full stop aren't recognized (김민준 씨．) #322 -- 김민준 씨． / 씨。 / 씨｡ and NFD 김민준, 씨. reach the suffix vocabulary. The lookup fold strips the stop set at word edges and NFC-composes non-ASCII words. NFKC was declined: it does not fold U+3002 at all (NFKC("。") == "。"), and it would rewrite stored vocabulary. No shipped entry changes under NFC (1735 strings, 0 changed; pinned by a test). The NFC half reaches Latin too: a decomposed Señor is a title now.
• 양. 지훈 splits the given name in half #323 -- a period glued to a CJK token no longer removes it from script classification, so 양. 지훈 keeps its given name, 김. 민준 too (the issue's first candidate fix would have missed it), and 양 지훈. keeps family-first. The surname site and the honorific peel match on the stop-stripped core so the stop rides with the remainder: 김민준씨. and 田中さん. peel as their stop-less spellings do. rules.md#H2's abbreviation shape refuses a word carrying a no-initials script (the veto is_initial has carried since A trailing period stops a CJK honorific being recognized ("김민준, 씨." → given "씨.") #320), so a lone 田中. is the family name; hangul never reaches that test because hangul segmentation runs first, so its witness is Han. Latin and Cyrillic openers are untouched.

Commits

Five, bisectable, each green on its own: lookup fold; classification fold plus the surname site; the honorific peel; the H2 veto; docs, ledgers and release note.

Measurements (py3.11, 2026-09-10)

• Corpus: 17 names move against master (after the review round's four new pins), all of them the bundle's own rows; the 20 pre-existing CJK names carrying an ASCII period (21 rows) are not moved by this bundle. One fix(#322/#323) rule per ledger, 11 names at 1.4.0/2.0.0 and 17 at 2.1.0/2.2.0; every baseline exits 0 with nothing unexplained and nothing unclassified.
• Parse cost: the reference name is unchanged at parse=409 facade=446 calls (tools/perf/call_count.py --against master); an H2-shaped opener pays the veto, Xyz. John Smith 304 to 308.
• Two-input invariant over the corpora: 28 of 30 CJK names with an edge stop divide exactly as their stop-less spelling; the two that do not are Latin words wearing a Latin period (毛 泽东 Dr., 田中さん V.), which is the boundary.

Decisions worth a glance

• 田中. 太郎 stays a tolerated row rather than becoming H2's contract-tier example line, following the 2026-09-05 precedent recorded in compare.py (the example moved into tolerated W3 instead).
• The ledger guard's honorific-alternation pin gained a _NOT_A_VOCABULARY_COPY roster so a literal-name alternation can be declared not-a-vocabulary-copy; the declaration forgoes two reach checks (the The #274 maiden rule matches "born", which is not a maiden marker, so a regression on "Max Born" reads as intended #350 precedent) and _CORPUS_CLAIMS plus _MUST_NOT_MATCH stand in for them.
• Recorded limits, not fixed: halfwidth kana is outside the script table, so ﾀﾅｶ. John still reads a title by shape; a bracketed ASCII-period CJK credential (김민준.) John Smith is unwrapped and divided (was a title); the stop set reaches Latin and Cyrillic too (Dr。 John Smith reads title, Smith, John V。 reads suffix where V. stays a middle initial).
• Review round (two commits on top): the classification fold reads TRAILING stops only, so a leading stop hides the token from the surname site as on master (a caller-configured segmenter could otherwise have divided .김민준 into . and the name); the segmenter receives the stop-stripped core, so the ja pack now divides 山田太郎.; the honorific peel is not script-gated and still reads a leading-stop token (.김민준씨 peels). Decided (Derek, 2026-09-10): a Lexicon pickled by 2.1.x/2.2.x with a caller-added wide-stop or NFD entry no longer loads (the load guard re-runs the widened fold; shipped vocabulary is unaffected) — carried as a release-note line with the remedy (rebuild the Lexicon from its source), not a forgiving re-fold on load; the decisions entry records both remedies weighed.
• /simplify round (one commit): the peel site returns the cut offset; the core/rstrip invariant is stated once; an isascii() short-circuit before the H2 veto; the ledger guard's enumerated exemption sets replaced by a derived vocabulary-fold criterion; call counts unchanged.
• decisions.md#cjk-full-stops corrects three earlier entries it supersedes (indic-honorifics' "no NFC in that module", W1's 2026-07-29 "matching stays raw", cjk-comma-demotion's five-parse summary).

Closes #322, closes #323.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.67%. Comparing base (d37b8ec) to head (d3f5597).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #522   +/-   ##
=======================================
  Coverage   98.67%   98.67%           
=======================================
  Files          45       45           
  Lines        3309     3320   +11     
=======================================
+ Hits         3265     3276   +11     
  Misses         44       44           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.