[davidschachterADFA]

ADFA-3305

android_pages_to_templates.py now stores once every text that 5 or more a/ pages repeat: in the 2026-10-19 ship-time build, 6,441 texts in 524 bundle templates. The texts are grouped by the pages that use them, so a page compiles little it doesn't use. The download is 928,748 bytes smaller and the file 905,216. On a phone, first page loads are 55-323 ms slower than without it. That is on a build with CodeOnTheGo #2168's lexer change, which makes first loads 21-41% faster with or without this PR. Against today's lexer without shared texts, three of the four timed pages still load faster (cross-session: EmojiButton 1,442 to 1,033 ms, TextView 1,671 to 1,537, VerticalGridView 1,800 to 1,407), and Pair loads slower (266 to 311). Stacked on #49 (fix/ADFA-3305-android-templates).

The change

• What is shared: a text of 32 or more characters that 5 or more pages hold, in a raw segment or a hole other than a link hole. Mostly repeated description paragraphs and inherited docs, then annotations, parameter lists and types.
• Where it goes: into android-text-<n>.peb, one branch per text of {% if j == 0 %}...{% elseif j == 1 %}...{% endif %}, at most 8,000 bytes of source per bundle. A page stores [<n>, <j>] under "<hole>@" ("r@" for a raw segment).
• Grouped by page: a page compiles every bundle it uses. So texts that the same pages use start in one group. Groups then merge, best first, while the merge lowers the sum over pages of compiled bundle source plus 8,000 bytes for each bundle. The pages that use both groups save a bundle each; every other page using one of them compiles the other for nothing. A merge must still fit one bundle, and may fill it exactly.
• The main template: only the holes that hold a shared text get the include alternative. It grows from 31,690 to 39,012 characters.
• Link holes stay in the pages. A same-page link is stored as its fragment, which often equals a member's id. A prototype that interned it lost the page's URL, and the render check kept 1,405 to 3,797 pages as HTML. A shared text may still hold a link in its markup; every such link in the a/ pages is rooted.
• Edge cases:
  • a text too long for a bundle stays in its pages;
  • a bundle that no converted page reads is not written;
  • a database holding bundles refuses a second run, as one holding the step's other templates does.
• Unchanged from ADFA-3305: Store the a/ reference pages as JSON plus shared templates #49:
  • every converted page renders back byte for byte before anything is written;
  • version 3 or later refuses;
  • dry run, one transaction, backup, VACUUM.
• App: no change needed.
  • include ... with is what ADFA-3305: Store the a/ reference pages as JSON plus shared templates #49's inherited-member includes already use.
  • Indexing a list (s["d@"][0]) is new to this step's templates. The app's own layout.pebble already does it, and every converted page renders back with the app's Pebble settings.

Measured

The ship-time order with this branch's step in place of #49's, each other step at the commit the item-B build used. Then: 4-seed re-mint, verify, rebuild_database.py --pack Content,TooltipTexts --page-size 4096, brotli -q 11 -w 24.

Build	.br download	File
Item-B ship-time build	67,133,725	85,708,800
This branch	66,204,977	84,803,584
Saved	928,748	905,216

• Input: the conversion started from the item-B build's own pre-conversion state (sha256 ac373dca…). The full order reproduces that state byte for byte.
• Render check: 12,827 of 12,828 pages byte-identical, as before. R.styleable is over the render limit.
• Re-mint verify: 22,744 of 22,744 items and 1,650 of 1,650 templates.
• Templates: 1,650 (1,119 includes, 524 bundles). The largest bundle is 7,998 bytes; the largest template is still ADFA-3305: Store the a/ reference pages as JSON plus shared templates #49's android-inherited-406.peb (810,416).

What a page compiles

Bundle source a page compiles on its first load (exact; the same 6,441 texts):

	First-use order	Grouped by page
Bundles	164	524
Median page	15,419	2,653
95th percentile	61,211	14,411
Largest	290,523	144,127
TextView	290,523 (38 bundles)	121,032 (32)
EmojiButton	39,260 (5)	4,454 (2)
VerticalGridView	15,914 (2)	1,385 (1)
Pair	23,927 (3)	7,972 (2)
Bundles a page, median / 95th percentile	2 / 8	1 / 5

The full build with first-use packing saved 856,064 file and 867,547 .br bytes. Grouping costs no measurable storage: the difference is within the re-mint's seed-to-seed spread, about 0.25 MB.

Why these settings

• 5 pages: measured with the first prototype, which filled bundles in first-use order. Full builds at other thresholds saved less:
  • 2 pages: 614,400 file, 799,100 .br, in 1,047 bundles;
  • 20 pages: 643,072 file, 588,189 .br.
• 8,000 bytes per bundle: offline with this packing.
  • 16,000-byte bundles put TextView's texts in 28 bundles instead of 32, for about the same source (122,038 bytes against 121,032).
  • 32,000-byte bundles made TextView compile 174,317.
• 8,000 bytes per bundle in the merge rule: offline with this packing.
  • 2,000 compiled less per page (median 958), in 938 bundles that stored about 22 KB more.
  • 32,000 compiled more (median 7,973).
• Other packings tried:
  • sorting by page set alone changed nothing;
  • one bundle per identical page set gave TextView 164 bundles;
  • merging by Jaccard similarity, or giving the heaviest pages bundles of their own, each did worse on per-page source or on storage.

On a phone

Pixel 6 Pro, build C-d-1020-0417 (versionCode 261020082: phase B plus CodeOnTheGo #2168's final lexer, e86384013), with the Bookshelf plugin 1.0.1 installed. The two databases:

• this branch's ship-time build (1151c74's packing, sha256 c7de1d8a…);
• the item-B build without shared texts (4c2005de…).

They alternated cycle by cycle in one session. Each page was loaded in a fresh process, two runs each, timed with timepage.mjs as in phase B.

• The plugin: it rewrites the installed database at every start, taking about 4.4 s. Each run waited for that to finish, then touched the override, and timed only after logcat showed the app had swapped to it. Every run did, so none is excluded.

Times in ms:

Page	Item-B, first	This branch, first	Difference	Item-B, second	This branch, second
EmojiButton	931, 982	1,026, 1,039	+76	287, 273	289, 295
TextView	1,197, 1,231	1,481, 1,592	+323	923, 699	885, 759
VerticalGridView	1,137, 1,488	1,196, 1,618	+95	331, 377	373, 393
Pair	250, 262	277, 344	+55	77, 107	65, 101

• First loads are 55-323 ms slower with shared texts, TextView the most; second loads are about the same.
• Off-device bytes: 0, the app UID's count stayed at 189,621,756.

Tests

• 25 new tests in scripts/test_android_pages_to_templates.py:
  • shared texts serve the same bytes, and the holders convert;
  • the 5-page threshold;
  • texts of 32 characters shared and of 31 not;
  • bundles fill to the limit and no further, and may reach it exactly: at a test cap and at the real 8,000, and one byte over stays inline;
  • several bundles serve the same bytes;
  • a text longer than a bundle stays inline;
  • link holes and fragments are never shared;
  • only interned holes get the alternative;
  • the dry run writes nothing;
  • a database holding bundles refuses;
  • an unused bundle is not written;
  • texts of the same pages share a bundle and other pages' texts don't;
  • merges when the pages served outweigh the waste, none when they don't, none when the gain is exactly zero, one that fills a bundle exactly, none that would overflow one;
  • order of first use within and across bundles;
  • a merge offered before its group grew is not taken;
  • a merged group can still merge with a lower-numbered one;
  • texts holding Pebble syntax ({{, {%, {#, unclosed tags, {%-, a trailing {) or CRLF, LS, PS, NEL or non-ASCII are shared, and their pages convert and serve the same bytes.
• 62 tests pass under pytest (Python 3.11 and 3.14) and as a plain script. CI doesn't run scripts/.
• Mutants: 46 of the new code (20 of the interning, 14 of the packing, and the review's 7 plus 5 more boundary cases); 45 are killed. The survivor lets the merge pre-check allow one byte more than the exact lower bound. That's equivalent: the exact check when a merge is taken still refuses anything over the cap.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

[davidschachterADFA]

Independent Claude review (head 713ab3e)

The output is correct. Every page the step writes renders back byte for byte, and every claimed guarantee held on real data. The one real bug is in the packing: it skips merges that the docstring and this PR say it makes. The rest are test gaps and docs that have drifted from the code.

apt = scripts/android_pages_to_templates.py, test = scripts/test_android_pages_to_templates.py.

ID	Severity	Where	Issue	Failure scenario
R56-01	Medium	apt:595, 601	A heap entry stores the ids as (min, max) but the versions as (a, b)	After a merge, every merge with a lower-numbered group is thrown away
R56-02	Low	test:1085+	7 mutants of the new code survive the 55 tests	An 8,001-byte bundle is written, and the tests stay green
R56-03	Low	test:851+	No test sends Pebble syntax through a bundle	A regression shows up only as pages quietly staying HTML
R56-04	Low	apt:108, 155	The docstring and constant comments quote superseded figures	8,000 is justified by first-use packing data
R56-05	Nit	apt:45, CLAUDE.md:192	"Links stay in the pages" covers only the link holes	2,460 bundle texts hold an <a href>
R56-06	Nit	PR body	"List indexing, which the step already used"	#49 indexes no list

R56-01:

• The bug: line 595 pushes (-g, min(a, b), max(a, b), version[a], version[b]), and line 601 compares (version[min], version[max]). A merge keeps the lower id and raises its version, so an entry re-offered against a lower-numbered neighbour is always discarded.
• Output stays correct: no stale entry can be accepted, because versions only grow. But "until no merge lowers it" is false.
• Small case: X uses pages 0-11, A pages 0-9 and 12, B pages 0-9 and 13. The code gives [X] and [A, B], leaving a merge that fits with a gain of 79,588; fixed, it gives one bundle.
• Real data:
  • 45 merges that fit and have a positive gain are left; the fix leaves none.
  • Bundles go from 541 to 524, and the objective drops by 0.73%.
  • The 95th percentile goes from 14,672 to 14,411, EmojiButton from 5,046 to 4,454 and Pair from 8,140 to 7,972. TextView gets worse, from 116,987 in 31 bundles to 121,032 in 32.
• Tests: all 55 pass either way.
• Fix: push version[lo], version[hi] and add the X/A/B regression test. Then re-measure every published figure.

R56-02:

• Survived (12 targeted mutants):
  • the single-text size check without the tag overhead: an 8,001-byte bundle gets written, and the only test uses a 9,608-byte text;
  • >= INTERN_MIN_CHARS changed to >, and 32 changed to 33;
  • the merge pre-check changed from <= to <;
  • the pop size check changed from > to >=;
  • the gain test changed from > 0 to >= 0;
  • the R56-01 fix.
• Fix: add boundary tests: a bundle at exactly the cap and one byte over, texts of 31 and 32 characters, a merge exactly at the cap, a zero-gain pair, and the R56-01 case.

R56-03:

• Real data never reaches this path: 0 of the 6,441 texts hold {{, {% or {#.
• The code handles it: 21 edge texts through intern_texts and the real Render.java all rendered byte-identical. They included unclosed tags, {%-, a trailing {, CRLF, LS, NEL and PS line breaks, and non-ASCII.
• Fix: add a test whose pages share such texts and that asserts they convert. The HTML fallback would hide a failure from a byte-identity check.

R56-04:

• apt:108-121 still gives ADFA-3305: Store the a/ reference pages as JSON plus shared templates #49's template figures: 12,135,203 bytes against this branch's 13,389,918.
• apt:155-157 gives the prototype's 839,680 / 850,755 as this build's figures.
• apt:161-162 justifies 8,000 with first-use packing data, and calls its median "13 KB" where the measured median was 15,419.
• Fix: label these as first-use or prototype figures, or re-measure them.

R56-05: only the h hole and the *# fields are excluded, and 2,460 interned texts contain href=". They are all rooted, so drop_unread_rows.py still resolves them, but it would refuse to rewrite a link that sits inside a template.

• Fix: say "link holes excepted".

R56-06: #49's main template indexes no list; the app's layout.pebble does. This is harmless; reword it.

Checked and sound:

• Figures: the PR's step-level figures reproduce exactly on ac373dca.
• The written database:
  • every [n, j] points at a written bundle and branch;
  • bundles are numbered 0-540 with no gaps, and none is unused;
  • no interned text is on fewer than 5 converted pages.
• Rendering: bundle texts are literals and print with | raw, so autoescaping can't change them. Newline trimming after tags is handled, and the include's with scope is clean.
• Determinism: Python 3.13 with PYTHONHASHSEED=777 and 7 workers gives byte-identical Templates and a/ rows.
• Refusals: a second run exits 1 and changes nothing.
• Links: h and h# are never interned, and interned texts hold no relative or fragment-only links.
• Speed: interning takes 2.0 s of the 47 s step.
• Not verified: the .br and file savings, and the phone timings.

[davidschachterADFA]

Response to the independent review (head 713ab3e, now 592a1f6)

All six findings are addressed (713ab3e..592a1f6).

ID	What changed	Commit
R56-01	Heap entries carry the versions of lo, hi = min(a, b), max(a, b); the X/A/B regression test	af27f50
R56-02	The merge pre-check allows for the 7 bytes a merge saves, so a merge may fill a bundle exactly; boundary tests	9388896, a20f1e3
R56-03	A run test shares ten Pebble-like texts across five pages and asserts they convert	1151c74
R56-04	Docstring and constant comments give the fixed packing's figures, or say which earlier code measured them	592a1f6
R56-05	"Link holes stay in the pages", in the docstring, CLAUDE.md and the body	592a1f6, body
R56-06	The body says include ... with comes from #49 and list indexing from the app's layout.pebble	body

• R56-01: your X/A/B case is now a test. On 713ab3e it gets [X] and [A, B] (2 bundles); now 1.

• R56-02:

  • The pre-check: it compared the two bundles' sizes as they stand with the cap. A merged bundle is up to 7 bytes smaller than its parts: one {% endif %} (11) and one {% if j == 0 %} (15) go, and one {% elseif j == n %} (19 or more) comes. So a merge that would just fit was never offered. It now allows for those 7 bytes, and the exact check when a merge is taken is unchanged.
  • New tests:
    • a single text whose bundle is exactly the cap is shared, and one a byte longer is not, at a 300-byte cap and at the real 8,000;
    • 32 characters are shared, 31 are not;
    • a merge that fills a bundle exactly is taken;
    • a zero-gain merge is not, and with a bundle worth one byte more it is.
  • The split tests: now that the remainder of a split can merge into a bundle with room, the two tests of the split itself run with a bundle costing 0, so they still test only the split.

• R56-03: the texts hold {{, {%, {#, unclosed tags, {%-, a trailing {, CRLF, LS, PS, NEL and non-ASCII. The test asserts:

  • the pages convert and serve the same bytes;
  • no text stayed in a page;
  • a bundle uses verbatim.

  The list itself landed in a20f1e3 by mistake, with literal LS and PS characters; 1151c74 adds the test that uses it and writes them as escapes.

• R56-04:

  • the shared-texts paragraph now gives this branch's build (below);
  • the page-count comment says its figures are the first prototype's, with bundles filled in first-use order;
  • the bundle-size and bundle-cost comments give offline figures with the fixed packing;
  • the measured block for the shapes and includes says it predates the shared texts.

• R56-05: a shared text may still hold a link in its markup, as you found. The docstring says so, and that every such link in the a/ pages is rooted.

Proof

• Without the fixes: on 713ab3e, the R56-01 test fails (2 bundles, not 1) and so does the exact-fill merge test (bundles of 176 and 181 bytes, not one of 350). The other new tests pin behaviour 713ab3e already had, so mutants are their proof:
  • the single-text check without its tags, >= as > and 33 for the length, the cap at 7,999, and the gain >= 0 each fail one of them;
  • writing a raw text into a bundle instead of its Pebble source fails the R56-03 test.
• Mutants: 46 run: your 7 surviving kinds plus 5 more boundary cases, the earlier 20 of the interning and 14 of the packing. 45 are killed. The survivor lets the pre-check allow one byte more than the exact lower bound. That's equivalent: the exact check when a merge is taken still refuses anything over the cap.
• Tests: 62, passing under pytest (Python 3.11 and 3.14) and as a plain script.
• Full ship-time pipeline with the fixed packing (1151c74), from the item-B build's pre-conversion state (ac373dca…), each other step at the item-B build's commits:

Build	.br download	File
Item-B ship-time build	67,133,725	85,708,800
This branch	66,204,977	84,803,584
Saved	928,748	905,216

• Render check: 12,827 of 12,828 pages byte-identical (R.styleable is over the render limit).
• Re-mint verify: 22,744 of 22,744 items and 1,650 of 1,650 templates.
• Templates: 1,650: 1,119 includes and 524 bundles, the largest 7,998 bytes. The main template is 39,012 characters.
• Against 713ab3e's build (66,172,076 / 84,754,432): 32,901 bytes more .br. Its re-mint stored 44 KB more, inside the seed-to-seed spread of about 0.25 MB.
• Per-page compile, exact, the same 6,441 texts:

	713ab3e	Now
Bundles	541	524
Median page	2,653	2,653
95th percentile	14,672	14,411
Largest	145,235	144,127
TextView	116,987 (31)	121,032 (32)
EmojiButton	5,046 (2)	4,454 (2)
VerticalGridView	1,385 (1)	1,385 (1)
Pair	8,140 (2)	7,972 (2)
Bundles a page, median / 95th percentile	1 / 5	1 / 5

These match your figures for the fixed packing. The database is kept for the phone run, against item-B with the final lexer build.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3