ADFA-3305: Re-mint a 768 KiB dictionary on a larger sample

[davidschachterADFA]

ADFA-3305

remint_dictionary.py now re-mints a 768 KiB dictionary trained on up to 3,600 rows within 128 MiB (--sample-size 1200, --training-bytes), instead of the first mint's 256 KiB on up to 900 rows within 32 MiB. On the ship-time database that is 1.66 MB less file and 1.92 MB less .br. Stacked on #48.

Measured

The 2026-10-19 ship-time database, 4 seeds each, every row and template verified byte-identical, packed with rebuild_database.py --pack Content,TooltipTexts --page-size 4096, .br with brotli -q 11 -w 24:

Dictionary, sample	Stored by seed (MB)	File	.br
256 KiB, 900 rows (before)	47.92 / 47.99 / 47.82 / 47.89	90,726,400	72,436,600
768 KiB, 900 rows	53.04 / 51.75 / 46.19 / 46.30	89,604,096	70,924,003
256 KiB, 3,600 rows	47.54 / 47.50 / 47.53 / 47.53	90,419,200	72,191,833
512 KiB, 3,600 rows	46.47 / 46.63 / 46.46 / 46.37	89,526,272	71,148,049
768 KiB, 3,600 rows (after)	45.65 / 45.98 / 45.69 / 45.71	89,063,424	70,517,587
1 MiB, 3,600 rows	63.20 / 45.53 / 45.24 / 45.22	88,875,008	70,131,363

• The sample is what limits a large dictionary. With 900 rows, two of four 768 KiB seeds stored more than 256 KiB, and zstd could not fill 1 MiB at all: it returned 965,307 bytes, which stored 7.7% more than 256 KiB on a 1,500-row probe.
• Not the window: an explicit -w 24 gave byte-identical output to the CLI's default for every dictionary.
• Why 768 KiB: with the larger sample its seeds spread 0.33 MB (256 KiB's: 0.16 MB on the small sample, 0.03 MB on the large), but its worst seed still stores less than 512 KiB's best, and its best beats 512 KiB's by 0.71 MB. 1 MiB stored 0.43 MB less at best, but one seed in four stored 17.55 MB more than 768 KiB's best.
• With ADFA-6627: Minify the documentation's text without changing what pages show #44's keep-current rule: a 1-seed re-mint of the 3.0.0 ship-time database still replaced its 256 KiB dictionary with a 768 KiB one: 47,874,174 -> 45,653,276 stored bytes with templates (-4.6%), every item and template verified byte-identical.
• Phone: on a Pixel 6 Pro (the ADFA-3305 debug build), a 256 KiB database and the "768 KiB, 900 rows" one (same dictionary size as this default, trained on the small sample) gave the same page load times within run-to-run spread (TextView 580 vs 550 ms median, Kotlin Int 110 vs 136 ms, javadoc String 246 vs 258 ms). The app's memory showed no increase.

Change

• REMINT_DICT_SIZE, REMINT_SAMPLE_SIZE, REMINT_TRAINING_BYTES, used by remint() and the command line.
• populate_db.py's first mint keeps 256 KiB: the ship-time copy is always re-minted last.
• Docstring "Dictionary size" section and CLAUDE.md updated.

Who else gets the new size

Nothing that ships re-mints yet. CodeOnTheGo's compress_docdb.yml only Brotli-compresses the master from Drive and release.yml bundles that file, so today's release keeps the master's 256 KiB dictionary. The new default applies to a re-mint run by hand and to the ship-time pipeline once one exists. Of this repo's workflows only the Android docs pair calls the re-mint, and #49 removes it; the Kotlin and Java workflows don't. The app on stage reads a dictionary of any size: DocumentationCompression.kt checks only a minimum of 8 bytes (from reading the code; the device check above used the ADFA-3305 build, not a stage build). Training reads 86-93 MiB of text instead of about 22 MiB: in the review, sampling took 3.1 s and zstd training 15.7 s and 121 MB of memory (256 KiB: 3.8 s, 62 MB), and a default 4-seed run on the ship-time build 16 min 11 s with 8 workers.

Tests

Two new tests pin what remint() hands the sampler and trainer, and what the command line passes. Both fail on #48's head (15db30a), on the old defaults. Rebased onto #48's review fixes; the command-line test's stand-in for remint() now accepts its new before_write keyword. A third, from the review, pins the first mint's defaults (256 KiB, 300 rows, 32 MiB); it fails with populate_db.DEFAULT_DICT_SIZE at 768 KiB. 39 tests pass under pytest and as a plain script.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

[davidschachterADFA]

Rebased onto #48's review fixes (15db30a), so this branch was force-pushed: 622534d -> 0f4ce35. remint()'s signature and CLAUDE.md conflicted; both sides are kept. The command-line test's stand-in for remint() now accepts #48's new before_write keyword. The two tests this PR adds still fail on #48's head, on the old defaults. 38 tests pass; the CI step (Python 3.11) passes 286. The body's Tests paragraph is updated.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

[davidschachterADFA]

Independent Claude review (head 0f4ce35)

The change does what its body says. A re-mint with no size flags now trains a 768 KiB dictionary on up to 3,600 rows within 128 MiB. On a copy of the 3.0.0 ship-time build, a default --seeds 4 run reproduced the body's table row for that setting. The four seeds' stored totals, the kept seed and the packed file size match to the byte. The .br is within 0.09%, because the measurement packed a different layout of the same content. Every item and template verified byte-identical. The app on stage takes a dictionary of any size, and per row it decodes no slower with a 768 KiB one. The findings are about what the docs say happens to the new size, one docstring comparison, which database the phone check used, and an unpinned first-mint default:

ID	Sev	file:line	Finding
R53-01	low	body; CLAUDE.md:41-43, :116-120	Says the release gets 768 KiB; nothing that ships re-mints
R53-02	nit	remint:84-86	"Spread 0.33 MB, as 256 KiB's do": 256 KiB's is 0.16/0.03 MB
R53-03	nit	body ("Phone"); remint:86-88	Phone check used the 900-row 768 KiB file, not this default's
R53-04	nit	tests	No test pins the first mint's 256 KiB (a 768 KiB mutant passes)

remint is ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/remint_dictionary.py.

R53-01. "Who else gets the new size" says the Android-docs workflows' re-mint of the master gets 768 KiB, "and so does the database today's release ships". CLAUDE.md:116-120 says "The ship-time copy is always re-minted last, so what ships has the re-mint's size". CLAUDE.md:41-43, in the paragraph about ~/documentation.db as of 2026-08-20, now reads "256 KiB as first minted, 768 KiB after remint_dictionary.py". In fact:

• Releases: CodeOnTheGo's compress_docdb.yml (origin/stage) downloads the master from Drive and runs brotli -f -Z on it. Nothing in it re-mints. release.yml downloads that pre-compressed documentation.db.br from Drive into the APK's assets.
• Docs workflows: build-kotlin-docs.yaml and build-java-docs.yaml don't call remint_dictionary.py. Only build-android-docs.yaml and build-android-docs-local.yaml do (--seeds 4 --no-backup, no size flags, so they would get 768 KiB). Neither has ever run, and ADFA-3305: Store the a/ reference pages as JSON plus shared templates #49's 0765f83 removes both.
• The master today: the 2026-10-18 database's dictionary is 262,144 bytes.
• The ship-time copy: it is not built by any workflow yet.

So what ships today keeps 256 KiB. After #49 merges, nothing in CI re-mints at all, and the new default reaches a shipped file only through the ship-time pipeline, once that exists. Nothing breaks, but the text tells a release reader the opposite. Fix: say the new default applies to a re-mint run by hand (minify_text.py's "Next:" line) and to the ship-time pipeline once it exists. Say that today's release ships the master's 256 KiB dictionary. Drop the Android workflows from the list once #49 lands. In CLAUDE.md:41-43, keep the 2026-08-20 paragraph about the master, and leave the re-mint size to the new paragraph.

R53-02. The docstring says that with the larger sample, 768 KiB's "seeds spread 0.33 MB, as 256 KiB's do". From the per-seed logs of the measurement runs (Sample seed N: X stored bytes):

Setting	Stored, min - max	Spread
256 KiB, 900 rows	47,822,590 - 47,987,183	164,593
256 KiB, 3,600 rows	47,504,066 - 47,537,773	33,707
512 KiB, 3,600 rows	46,367,909 - 46,632,839	264,930
768 KiB, 3,600 rows	45,653,276 - 45,980,429	327,153
1 MiB, 3,600 rows	45,221,819 - 63,202,482	17,980,663

768 KiB's spread is twice 256 KiB's at the small sample and ten times it at the large one. The conclusion still holds: 768 KiB's worst seed (45.98 MB) stores less than 512 KiB's best (46.37 MB). Separately, "one seed in four stored 17.6 MB more" (body and docstring) doesn't name what it is more than. 63,202,482 is 17.55 MB above 768 KiB's best and 17.98 MB above 1 MiB's own best. Fix: drop "as 256 KiB's do", or give the 256 KiB spreads, and name the baseline of the 17.6 MB.

R53-03. The body's "Phone" bullet and the docstring's "no measurable difference in page load time or memory between a 256 KiB and a 768 KiB dictionary" come from two databases. The 768 KiB one was trained on the 900-row sample, the "768 KiB, 900 rows" row: r768-packed.db, 89,604,096 bytes, sha256 32880836…. This PR's default produces a different database: the "768 KiB, 3,600 rows" row, 89,063,424 bytes. Both dictionaries are 786,432 bytes, so the check does cover the size. That is the property that matters for decode cost, and the desktop check below agrees. The recorded runs back the stated medians: two rounds of three loads per page per database, after a warm-up. TextView median 580.5 against 549.5 ms, Kotlin Int 109.5 against 135.5 ms, String 246.5 against 258.0 ms. The app's total PSS was 558-567 MB with 256 KiB and 527-539 MB with 768 KiB. Fix: say which database the phone ran.

R53-04. The body's Change list says "populate_db.py's first mint keeps 256 KiB". That contrast is what the new docstring paragraph and CLAUDE.md:116 rest on, but nothing pins it. With populate_db.DEFAULT_DICT_SIZE = 768 * 1024, all 55 tests in test_remint_dictionary.py, test_populate_db_dictionary.py and test_migrate_content_to_dictionary_brotli.py pass. Fix: one assertion that the first mint's defaults stay 256 KiB, 300 rows and 32 MiB. Alternatively, state the first mint's size in one place only.

Checked and sound

• Defaults: REMINT_DICT_SIZE 786,432, REMINT_SAMPLE_SIZE 1,200 and REMINT_TRAINING_BYTES 134,217,728 feed remint()'s signature and the three CLI defaults. The sampler stops at 3 x sample_size rows or the byte budget, so "up to 3,600 rows within 128 MiB" is right. migrate_content_to_dictionary_brotli.py still imports DEFAULT_DICT_SIZE (256 KiB), 300 rows and 32 MiB for the first mint.
• Real data: a default run (--seeds 4 --no-backup --max-workers 8, no size flags) on a copy of final.db.
  • Seeds: 3,526 training rows each. Stored totals 45,653,276 / 45,980,429 / 45,685,121 / 45,707,309, the body's row to the byte. Seed 20819 was kept: 47,874,174 -> 45,653,276 bytes, against a 786,432-byte dictionary byte-identical to one trained separately from seed 20819's sample.
  • Verify: 22,745 of 22,745 items and 1,126 of 1,126 templates byte-identical.
  • brotli4j 1.18.0, as the app decodes: 1,126 of 1,126 templates decode to the original's text, and none decodes without the dictionary.
  • Cost: 16 min 11 s wall at 591% CPU, 507 MB peak RSS.
  • Packed with ADFA-3305: Convert animated GIFs to VP9 WebM and pack documentation.db #38's rebuild_database.py (c947a46): 89,063,424 bytes, as the body says. A second pack is byte-identical.
  • .br: 70,578,735, against the body's 70,517,587 (+61,148, 0.09%). The measurement packed an unpacked copy (its pack log: 94,711,808 -> 89,063,424, TooltipTexts 2,733 -> 2,596 leaf pages). This run packed the already-packed final.db (91,463,680 -> 89,063,424, TooltipTexts unchanged). Packing renumbers rows from the input's order, so the same content packs to the same size with different bytes. The table's rows all came from the measurement's one input, so they compare like with like.
• Measurements table: every figure matches the per-seed logs and the pack and .br results of the runs it came from (stored-by-seed values, file and .br for all six rows). The savings are 90,726,400 - 89,063,424 = 1,662,976 file and 72,436,600 - 70,517,587 = 1,919,013 .br, both against packed baselines, so the .br comparison is like for like. "768 KiB beats 512 KiB by 0.71 MB" is 46,367,909 - 45,653,276 = 714,633.
• Training cost (seed 20819, sampled and trained separately; measured while the 4-seed run was using 8 cores):
  • Sampling: 3,526 rows, 92,422,105 bytes (88.1 MiB) in 3.1 s, against 895 rows and 22.4 MiB in 0.7 s. Across the four seeds the large sample is 86.4-93.1 MiB, against the body's "about 86 MiB".
  • zstd --train-fastcover --maxdict=786432 on it: 15.7 s wall, 57.6 s CPU, 121 MB peak RSS, against 3.8 s and 62 MB for 256 KiB on the small sample.
  • Each candidate also writes its sample to a tempfile directory (/tmp by default), up to about 93 MiB at once, and removes it afterwards.
  • That is small next to a recompression pass and fits a 16 GB runner many times over. After ADFA-3305: Store the a/ reference pages as JSON plus shared templates #49 no workflow runs it anyway (R53-01).
• The app (CodeOnTheGo origin/stage, read-only):
  • loadCompressionDictionary checks only MIN_DICTIONARY_BYTES = 8; there is no upper limit.
  • DocumentationContentSource.codec() builds one BrotliDictionaryCodec per database and keeps it until a swap. Each decode attaches that same exact-size direct buffer to a new BrotliInputStream, with no copy per row.
  • brotli4j 1.18.0 on the desktop, used as the codec uses it, on 60 rows (1.53 MB): decode took 5.3 ms with 768 KiB against 7.5-8.1 ms with 256 KiB, and q11/w24 compress 1,385-1,455 against 1,427-1,473 ms, all round-tripping. These are training rows, so the 768 KiB compression ratio is flattering; only the timings are meant.
  • Outside this PR: plugin installs (PluginDocumentationManager, via warmUp()) build the encoder's prepared dictionary. On the desktop it is 3,077,520 bytes for 768 KiB, against 1,315,580 for 256 KiB. The KDoc on BrotliDictionaryCodec.warmUp says "~780 KB", which is stale for either size.
• Tests: 38 pass under pytest and as a plain script. Mutants: 13 run, 12 killed:
  • each of the three constants set back to the first mint's value;
  • each of remint()'s three defaults set back separately;
  • each of the three CLI defaults set back separately;
  • the CLI's sample and dict arguments swapped;
  • the trainer ignoring dict_size;
  • the sampler ignoring training_bytes.
    The survivor is the first mint's own default (R53-04).
• Not checked: the "explicit -w 24 gave byte-identical output for every dictionary" probe (ADFA-3305: Compress every template in the 3.0.0 migration #50 pins -w 24 as the stdin default); the 1,500-row probe behind "7.7% more than 256 KiB" (the 900-row 1 MiB re-mint in the ADFA-3305 analysis stored 7.1% more); the phone (not used).

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

[davidschachterADFA]

Response to the review

All four findings are addressed (0f4ce35..fc9b5da).

ID	Fixed in	Change
R53-01	0ee5863, body	CLAUDE.md and the body say nothing that ships re-mints yet; the master keeps 256 KiB
R53-02	c91f840, body	The 256 KiB spreads are given; the 1 MiB outlier is measured against 768 KiB's best
R53-03	c91f840, body	The phone check names its database: the 900-row 768 KiB one, same dictionary size
R53-04	fc9b5da	A test pins the first mint's 256 KiB, 300 rows and 32 MiB

• R53-01: "Who else gets the new size" now says today's release keeps the master's 256 KiB dictionary, because compress_docdb.yml only Brotli-compresses the master and release.yml bundles that file. The new default applies to a re-mint run by hand and to the ship-time pipeline once one exists. On this branch only build-android-docs.yaml and build-android-docs-local.yaml call remint_dictionary.py (checked by grep), and ADFA-3305: Store the a/ reference pages as JSON plus shared templates #49 removes both. CLAUDE.md's 2026-08-20 paragraph about the master is back to its 256 KiB, and the new paragraph says the rest. The body also gives your training figures in place of "not timed separately".
• R53-02: the docstring and body give 256 KiB's spreads (0.16 MB on the small sample, 0.03 MB on the large). The comparison that holds is kept: 768 KiB's worst seed stores less than 512 KiB's best. The outlier is 17.55 MB above 768 KiB's best.
• R53-03: the phone database was r768-packed.db, the table's second row. Its dictionary is this default's size (786,432 bytes), not its bytes, and the docstring and body now say so.
• R53-04: with populate_db.DEFAULT_DICT_SIZE at 768 KiB, the new test fails and the rest pass, as you found; the code is restored. 39 tests pass under pytest and as a plain script.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3