Install dehyph-exptl for pinned German patterns and follow the tinytex hyphenation map #15035

Follow-up to #15027 / #15029. The table of hyphenation names in parse-error.ts is a hand-written copy of the one in tinytex and it is already behind.

tinytex now generates this table from the execute AddHyphen lines of texlive.tlpdb (rstudio/tinytex#517, follow-up to rstudio/tinytex#516). The generated table has 41 entries, the 37 we have plus 4 that map to dehyph-exptl: german-x-latest, ngerman-x-latest, german-x-2024-02-28 and ngerman-x-2024-02-28.

What is missing here

1. dehyph-exptl can't be expressed. The values are suffixes and the package is always built as hyphen-${name}, then checked against ^hyphen-[a-z0-9-]+$.

So a log with You haven't defined the language 'ngerman-x-latest' yet gives the non-existent hyphen-ngerman-x-latest.

2. findMissingHyphenationFiles only returns one package. The warning regex has no g flag, so with several Package babel Warning: No hyphenation patterns were preloaded messages only the first language is handled.

3. A language name with a space, like the language `German (new orthography)' in a (babel) line, doesn't match language [`'](\S+)[`']. tinytex keeps the first word (german) for those.

4. polyglossia is not handled in src/command/render/latexmk/ (tinytex handles Package polyglossia Warning: No hyphenation patterns were loaded for ...).

The chinese skip list and the \l@nil fallback exist here only, and tinytex doesn't have them.

Plan

The tinytex daily build now exports the map as hyphen.json inside the regex.tar.gz of the daily release of rstudio/tinytex-releases, next to regex.json (once rstudio/tinytex#517 is merged):

{
  "ngerman": "hyphen-german",
  "ngerman-x-latest": "dehyph-exptl"
}

Values are full package names, only exceptions to hyphen-<name> are listed.

Other ways to share the data that I looked at:

• Our own generator in Deno against texlive.tlpdb: no dependency on tinytex, but two generators and a new workflow to maintain for the same data.
• Fetching R/hyphen.R from the tinytex repository at build time: it needs parsing R syntax from TypeScript and adds a network dependency to the build.
• Keeping two copies by hand: this is what we have now, and the 4 dehyph-exptl entries show that it drifts.

Reusing regex.tar.gz fits the existing setup: verify-tinytex-patterns.yml already downloads it daily from the daily release of rstudio/tinytex-releases. That workflow only opens or comments on an issue (label tinytex-patterns), it never opens a PR, so the table would still be updated by hand after the report. We could also extend it to open a PR with the new table, but the table in parse-error.ts is TypeScript with comments and the test file has its own copy, so I would start with the report only.

Two things to keep in mind: hyphen.json only exists in the release once rstudio/tinytex#517 is merged and the daily build ran, and a change of its format would break the comparison step (which today only reads regex.json with jq).

We could:

• Switch kHyphenationPackageNames to full package names, add the 4 dehyph-exptl entries, and validate the language name (^[a-z0-9-]+$) instead of the resulting package.
• Extend verify-tinytex-patterns.yml to also compare hyphen.json, so the existing tinytex-patterns issue tells us when the map changes. Its jq steps only read regex.json today. This keeps the table bundled, with no network access at build or run time.
• Update the copy of the table in tests/unit/latexmk/parse-error.test.ts and dev-docs/tinytex-pattern-maintenance.md.
• Look at points 2 to 4 above separately, if they turn out to be real.

Still to check

Everything above comes from reading the code, nothing was run yet.

• Reproduce points 2 to 4 with small log samples in a Deno unit test (tests/unit/latexmk/parse-error.test.ts), and confirm which ones are real with logs produced by babel/polyglossia.
• Point 1: confirm that findMissingHyphenationPackages returns hyphen-ngerman-x-latest for You haven't defined the language 'ngerman-x-latest' yet.
• Check that hyphen.json is really in the daily release asset once Generate the hyphenation language map from texlive.tlpdb rstudio/tinytex#517 is merged and the daily build ran, and that the existing regex.tar.gz download step still works with the extra file.
• Check if the chinese / chinese-hans skip list is still needed (Quarto v1.5 installs a nonexistent package hyphen-chinese-hans when compiling PDF #10291), and if pinyin -> hyphen-chinese in the generated map interacts with it.
• Decide if polyglossia support belongs here or if it is not needed for Quarto's PDF output.

Related: rstudio/tinytex#515, #10291.