Repository navigation
ADFA-3305: Store the a/ reference pages as JSON plus shared templates - #49
davidschachterADFA wants to merge 17 commits into
Conversation
android_pages_to_templates.py turns every a/ text/html page (except a/devsite/) into JSON rendered through android-reference.peb. Repeated structures become records of fixed shapes, and each inherited-member block header becomes one shared include, android-inherited-<n>.peb, with every row behind a guard and the rows a page's class overrides listed in its JSON. Every page is rendered back with Pebble 4.1.1 and compared byte for byte before anything is written; one that does not match, or is over the render limit, stays HTML. Templates are written as plain text for the 3.0.0 migration to compress. On the schema-3 test database: 12,827 of 12,828 pages convert (R.styleable stays HTML), with 1 main template and 1,119 includes. After template compression and a 4-seed re-mint the file is 7.4 MB (7.5%) smaller and the .br 7.5 MB (9.4%). 32 tests; 28 of 30 mutants are caught, and the 2 left are equivalent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
Independent Claude review (head 12b9920)The conversion is faithful. In the 3.0.0 ship-time build, each of the 12,827 converted pages, served the way the app serves it, is byte-identical to the same page before the conversion. That check joins the chunks by the app's rule, decodes against the build's dictionary, decodes the templates (it is a 3.0.0 file) and renders with the app's engine settings. The 415
R49-01. The step never reads R49-02. R49-03. Pebble 4.1.1's default template cache is an unbounded
A median page uses no include. A widget page pulls in up to 1.2 MB of include text: R49-04. I ran 37 mutants; 29 were killed. Two of the 8 survivors are real gaps:
Fix: one test for each. R49-05. R49-06. #46's Checked and sound
🤖 Generated with Claude Code |
The step never read DocumentationDatabaseVersion. Run after the 3.0.0 migration, it exited 0 having added android-reference.peb and its includes as plain UTF-8 beside the compressed templates, and its closing line sent the user to the migration, which then refuses the file. The app decodes every template from 3.0.0 on, so every converted a/ page would fail. It now refuses a database whose newest DocumentationDatabaseVersion row, ordered as the app orders them (changeTime, then rowid), declares major 3 or later: one error: line, exit 1, before the renderer starts and before anything is written, in a dry run too. The helpers have the names and semantics of #50's guard in the other template writers (2950b1b) and #48's declared_major, so the guards read alike. Tests: a 3.0.0 database refuses, written or dry; the version row is read by changeTime then rowid (a later change in an older row, and a tie); no version table, an empty one and version 2 still convert. The first two fail on 12b9920 (exit 0, pages converted); the third passes there, as it must. 12 mutants of the guard (row order, max(major), threshold, an empty or missing table read as 3, exit code, dry run, no guard), all killed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
Two guards survived the review's mutants. find_blocks could accept a header and rows without the style's footer, and a render error other than the size limit could be reported as "renders differently" instead of "does not render". Neither changes what is stored: the render check keeps such a page as HTML either way. But the first costs size and the second gives the wrong reason. One test now has a block of each style with a wrong footer, alone and followed by a whole block of the same style; only the whole one is found, at its own offsets. Another renders one page through a renderer that fails with an error other than the limit; that page stays HTML, named with the renderer's first line, and the others convert. Mutants killed: the footer check dropped, a footer accepted anywhere later in the page, the not-ok branch removed, and every failure read as the limit (4 of 4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
_PEBBLE_NEWLINE and _VERBATIM_STRIPS held U+2028, U+2029 and 13 Unicode spaces (U+1680, U+2000-U+200A but U+2007, U+205F, U+3000) as literal characters, and the test file held U+2028 and U+2029 the same way. They show as blanks in a diff or an editor, so a reader could not see which characters the sets hold, or that U+2007 is rightly missing; and str.splitlines(), and tools built on it, break those lines at U+2028 and U+2029. They are \u escapes now. No behaviour change: both files parse to the same AST as before, the strip set holds the same 26 characters, and a dry run on the ship-time input gives the same output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
ProcessDocs/AndroidDocs/AndroidDocsToJson/load_android_json_db.py (ADFA-5548, on main) also replaces the scraped a/android/ and a/androidx/ rows with JSON, through its own android-class.peb, android-package.peb and android-index.peb. build-android-docs.yaml runs it on the master, by hand only; it has no runs so far. Neither the docstring nor CLAUDE.md said so. Both now do. This step converts scraped HTML on the ship-time copy and leaves a row already in another template as it is, so after that pipeline has run it converts almost nothing and saves almost nothing. Which of the two ships is still open. No behaviour change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
#46's drop_unread_rows.py says it runs after every step that writes page content, links or tooltips. This step writes page content, and its docstring put it after drop_unread_rows.py, which also listed the row drops among the steps that write a/ pages. The end-to-end build used that order and verified it, but the two docstrings contradicted each other. The docstring and the CLAUDE.md entry now name the order as an exception to #46's rule, and say why it is safe: every page this step converts renders back byte for byte, so it links to exactly what the row drops already checked in it as HTML. The includes do hold links (41,153 rooted ones, 40,991 of them under /a/, and about 15,400 absolute ones in the ship-time build's), but none that its pages did not hold before. drop_unread_rows.py reads templates and templated rows' JSON too, so it could also run after this step. No behaviour change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
Response to the reviewFive findings are fixed here (12b9920..48a3734). R49-03 is in CodeOnTheGo, so it's left for a decision outside this PR.
🤖 Generated with Claude Code |
The a/ pages ship converted on the ship-time copy by android_pages_to_templates.py. build-android-docs.yaml and build-android-docs-local.yaml ran ADFA-5548's pipeline on the master instead, replacing the same a/android/ and a/androidx/ rows with JSON through other templates; after one ran, this step would have converted almost nothing. Neither has ever run. The run-build-android-docs-with-act.sh driver goes with them, and build-kotlin-docs.yaml no longer says it shares the Drive file's concurrency group with the Android workflow. The pipeline's scripts and tests stay. The README, CLAUDE.md and this step's docstring now say which pipeline ships. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
|
New commit 0765f83 retires ADFA-5548's Android docs workflows, as decided on ADFA-3305: this step's conversion on the ship-time copy is the one that ships. It removes 🤖 Generated with Claude Code |
R49-03 measured on a Pixel 6 ProThe app's compiled-template cache, measured with the ADFA-3305 debug build and the 3.0.0 test database. The method: heap dumps after a forced GC. A template's retained size is the bytes reachable from GC roots minus those still reachable once the cache is taken away.
Conclusion: no change in this PR. The worst case is about 6% of the heap, and reaching it takes viewing about 741 particular pages in one process. Caching also saves recompiling large shared includes on every page that uses them. If low-RAM devices show memory pressure, the fix belongs in CodeOnTheGo: a least-recently-used The heavy session (741 pages, every include) was not run on the phone. The installed build predates ADFA-6589, so each page view still triggers Safe Browsing lookups off the device. The all-includes figure is therefore an extrapolation from the 52 includes measured, which are 25% of the include text, the largest among them. 🤖 Generated with Claude Code |
CodeOnTheGo's compress_docdb.yml only Brotli-compresses the master from Drive and release.yml bundles that file, so today's release keeps the master's 256 KiB dictionary. The Kotlin and Java workflows don't call the re-mint; only the Android pair does, and #49 removes it. CLAUDE.md now says the new default applies to a re-mint run by hand and to the ship-time pipeline once one exists, and its 2026-08-20 paragraph about the master is back to the master's 256 KiB. (R53-01) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
Found on the phone: the first load of a widget page takes about 100 sOn a Pixel 6 Pro, with the 3.0.0 item-B database, the first load of EmojiButton in a fresh process takes 93-114 s, and TextView takes 22-24 s. Later loads take 0.3-0.4 s. A bisect of the app PRs (#2164 alone, plus #2161, plus #2162) cleared all three: #2164 alone is just as slow. Cause: Pebble 4.1.1's lexer is quadratic on Android, and this PR's largest shared includes are big enough to hit it.
Proposed fix in this PR: cap the include size, splitting a large inherited block into several includes of a few KB each. Lexing cost grows with about the square of the size, so splitting the 807 KB include about 50 ways should cut it about 50-fold. That's an estimate to measure on the phone, along with the cost to the size saving. The app needs no change. The alternatives:
Details: the ADFA-3305 bisect notes of 2026-10-19. 🤖 Generated with Claude Code |
…dles
After the shapes and inherited-member includes, the a/ JSON still repeats
whole texts across pages: description paragraphs, inherited docs, table
heads, annotations, parameter lists and types. Each page stored its own copy,
and the shared dictionary could not hold them all.
A text of 32 or more characters that 5 or more pages hold, in a raw segment
or a hole other than a link, now goes into a bundle template
android-text-<n>.peb of at most 8,000 bytes of source, one branch per text of
{% if j == 0 %}...{% elseif j == 1 %}...{% endif %}. The page stores
[<n>, <j>] under "<hole>@" ("r@" for a raw segment), and only the holes that
hold one get the include alternative in the main template. Links stay in the
pages: a same-page link's fragment often equals a member's id, and the
prototype that interned it lost the page's URL. A text too long for a bundle
stays in its pages, and a bundle no converted page reads is not written. A
database holding bundles refuses a second run, as one holding the step's
other templates does.
The thresholds come from full ship-time builds (4-seed re-mint, packed,
brotli -q 11 -w 24) of the 2026-10-19 build: 5 pages saved 839,680 file and
850,755 .br bytes, against 614,400 and 799,100 at 2 pages and 643,072 and
588,189 at 20. Against 8,000 bytes, bundles of 16,000 or 32,000 stored 11 or
14 KB less but doubled or quadrupled the bundle source the median page
compiles (13 KB).
Every converted page still renders back byte for byte before anything is
written. 11 new tests; 22 mutants of the new code, all killed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
…e build The docstring quoted the prototype's run. This branch's full ship-time build, from the same input as the item-B build (its conversion backup has the same sha256), interned the same 6,441 texts into 164 bundles and saved 856,064 file and 867,547 .br bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
A page compiles every bundle it uses. Filled in first-use order, a bundle
held texts from unrelated pages, so on the phone every first load got slower:
TextView's 575 references fell in 38 bundles, 290,523 bytes of source to
compile, and its first load took about 0.7 s longer.
Texts that the same pages use now start in one group. Two groups then merge,
best first, while the merge lowers the sum over pages of compiled bundle source
plus 8,000 bytes for each bundle: the pages that use both save a bundle each,
and every other page using one of them compiles the other for nothing. A
merge must still fit one bundle. Bundles are numbered, and texts ordered
within them, by first use.
The interned set does not change: the same 6,441 texts. Offline, on the
2026-10-19 ship-time build's a/ pages, against first-use packing:
first use by pages
bundles 164 541
compiled, median page 15,419 2,653
compiled, 95th pctile 61,211 14,672
compiled, largest 290,523 145,235
TextView 290,523 116,987 (38 -> 31 bundles)
bundles a page, p95 8 5
Bundles of 16,000 bytes compiled about as much (TextView in 27 bundles).
Merging by Jaccard similarity of page sets, or giving the heaviest pages
bundles of their own, did worse on one of these or on storage.
7 new tests; 14 mutants of the packing and 20 of the interning, all killed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
The full ship-time build with 8c3ffc8's packing, from the item-B build's own pre-conversion state: 541 bundles, 954,368 file and 961,649 .br bytes saved against the item-B build (84,754,432 and 66,172,076), against 856,064 and 867,547 with first-use packing. The difference is within the re-mint's seed-to-seed spread (about 0.25 MB), so the grouping costs no measurable storage. 12,827 of 12,828 pages render back byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
…R56-01) A heap entry stored its groups' ids as (min, max) but their versions as (a, b), the group being offered first. A merge keeps the lower id and raises its version, so an entry offered from the merged group against a lower-numbered neighbour never matched at the pop and was always dropped: "until no merge lowers it" was false. Output stayed correct, since a stale entry is still never accepted. The versions now follow the ids. In the review's case, X (pages 0-11), A (0-9 and 12) and B (0-9 and 13): A and B merge first, and AB then merges with X into one bundle; before, two. The new test fails on 713ab3e. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
The merge pre-check compared the two bundles' sizes as they stand with the
cap, but a merged bundle is up to 7 bytes smaller than its parts: one
"{% endif %}" (11) and one "{% if j == 0 %}" (15) go, and one
"{% elseif j == n %}" (19 or more) comes. So a merge that would just fit was
never offered. The pre-check now allows for those 7 bytes, and the exact
check when a merge is taken still refuses one that does not fit.
A new test merges two groups into a bundle of exactly the cap; on 713ab3e
it gets two bundles. Now that the remainder of a split can merge into a
bundle with room, the two tests of the split itself run with a bundle costing
0, so that only the split is under test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
The review's mutants of the size, length and gain checks survived the tests. Four tests now pin the boundaries: - a single text whose bundle is exactly the cap is shared, and one a byte longer stays in its pages, at a 300-byte cap and at the real 8,000; - a text of 32 characters is shared, and one of 31 is not; - a merge that gains exactly nothing is not taken, and with a bundle worth one byte more it is. They pin behaviour 713ab3e already had, so the mutants are their proof: the single-text check without its tags, ">=" as ">" and 33 for the length, 7,999 for the cap and ">= 0" for the gain each fail one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
…56-03)
No real a/ text holds "{{", "{%" or "{#", so nothing tested that a shared
text is written into its bundle as literal text. A failure there would show
only as pages quietly staying HTML, which a byte-identity check alone does
not catch. The new run test shares ten such texts across five pages:
"{{", "{%" and "{#", unclosed tags, "{%-", a trailing "{", CRLF, LS, PS,
NEL and non-ASCII. It asserts that the pages convert and serve the same
bytes, that no text stayed in a page, and that a bundle uses verbatim.
It pins existing behaviour: writing the raw text into the bundle instead of
its Pebble source fails it. The PEBBLE_LIKE list came in with a20f1e3 by
mistake; this commit adds the test that uses it and writes its LS and PS as
escapes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
…link holes (R56-04, R56-05) R56-04: the docstring and constant comments quoted figures from earlier code. Now: - the shared-texts paragraph gives this branch's full ship-time build: 524 bundles, the largest 7,998 bytes; 905,216 file and 928,748 .br bytes saved against the item-B build (84,803,584 and 66,204,977); the median page compiles 2,653 bytes, the 95th percentile 14,411, the largest 144,127, TextView 121,032 in 32 bundles; - the page-count comment says its figures are the first prototype's, with bundles filled in first-use order; - the bundle-size and bundle-cost comments give figures measured offline with the fixed packing, in place of first-use packing's; - the measured block for the shapes and includes says it predates the shared texts. R56-05: only the link holes stay in the pages, not every link: a shared text may hold a link in its markup (all of them in the a/ pages are rooted). The docstring and CLAUDE.md now say link holes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
ADFA-3305: Store texts that five or more a/ pages repeat once, in bundles grouped by page
ADFA-3305
Adds
scripts/android_pages_to_templates.py, a ship-time step that stores the Android reference pages undera/as JSON plus shared Pebble templates. After the 3.0.0 migration compresses the templates and the final re-mint, documentation.db is 7.4 MB (7.5%) smaller and its.br7.5 MB (9.4%).Stacked on #43, whose Pebble render checker it uses. It needs the 3.0.0 migration to compress the templates (stacked on #45) and an app that reads compressed templates (CodeOnTheGo, stacked on #2163). All of these ship together as schema 3.0.0.
Includes #56, merged into this branch on 2026-10-20: the step also stores once every text that 5 or more
a/pages repeat, in 524 bundle templates. That saves a further 928,748 bytes of download and 905,216 of file. Its 25 tests bringtest_android_pages_to_templates.pyto 62. See #56 for the change, measurements and phone timings; the figures below are this PR's own.How
android-inherited-<n>.peb, with every row behind{% if x contains i %}. A page stores the rows its class overrides, which is 0 for the median block. The widgets each carried about 0.5 MB of View's rows.android-reference.pebloops over a page's segments.Checks before writing
a/row in another template, a page that is not UTF-8 or holds a NUL, anda/devsite/are left as they are.a/row, a missing renderer, and a database that already holds this step's templates (it does not run twice). The run uses one transaction, a backup first, VACUUM after, and--dry-run.DocumentationDatabaseVersionrow is read as the app reads it (changeTime, then rowid), with the helper names of ADFA-3305: Compress every template in the 3.0.0 migration #50's and ADFA-3305: Re-mint and verify compressed templates in schema 3 #48's guards. From 3.0.0 every template is compressed and the app decodes every one, so this step's plain templates would fail every converted page. On a copy of the ship-time input migrated to 3.0.0 with ADFA-3305: Compress every template in the 3.0.0 migration #50's script, the step before this guard exited 0 and left 1,120 plain templates beside the 6 compressed ones; it now prints oneerror:line, exits 1, and the file's sha256 is unchanged.The other Android pipeline (retired)
ADFA-5548's
ProcessDocs/AndroidDocs/AndroidDocsToJson/load_android_json_db.pydoes the same job another way: it replaces the scrapeda/android/anda/androidx/rows with JSON from the committed source tree, through its ownandroid-class.peb,android-package.pebandandroid-index.peb. A row already rendered through another template is left alone by this step, so after that pipeline had run, this step would convert almost nothing and the saving below would not happen. This step is the one that ships: 0765f83 removes the two workflows that ran that pipeline on the master (build-android-docs.yaml,build-android-docs-local.yaml, neither ever run) andrun-build-android-docs-with-act.sh, and drops the Kotlin workflow's note about sharing their concurrency group. The pipeline's scripts and tests stay; its README, CLAUDE.md and this step's docstring say which ships.Results
On the schema-3 test database (98,201,600 bytes):
R.styleable(4.46 M characters) stays HTMLSizes are measured against the same database with only its templates compressed. Both copies were re-minted with 4 seeds (templates sampled and recompressed with the rows), then packed with
rebuild_database.py --pack Content,TooltipTexts --page-size 4096(#38), and the.brmade withbrotli -q 11 -w 24:.brOn the end-to-end input of 2026-10-19, measured in the review (1-seed re-mint, migrated and packed the same way): file 98,770,944 -> 90,927,104 (-7.9%),
.br80,704,825 -> 72,669,139 (-10.0%). In that build, all 12,827 converted pages render byte-identical to the pages just before this step, read independently of this step's code.Seed-to-seed spread within each re-mint: at most 0.4 MB. Render time on a desktop JVM, JSON parse included: median 0.15 ms per page; the largest page,
R.attr(2.1 M characters), took 13 ms. Not yet measured on a phone.Pebble rules found on the way
\uescapes.{% verbatim %}, Pebble strips all leading whitespace (Java'sCharacter.isWhitespaceplus U+0085, measured one character at a time), so that whitespace is written before the block. No real page needed this today, and the tests cover it.{before a tag is printed from a string literal. Text that would end a verbatim block is never made template text.Tests
scripts/test_android_pages_to_templates.py: 37 tests on synthetic Doclava and Dackka pages. They pass under pytest (Python 3.11 and 3.14) and as a plain script. Tests that render needjavaand the Pebble jars, astest_hm_pages_to_templates.pydoes; neither file runs in CI (python-tests.yamldoes not runscripts/).max(major), the threshold, an empty or missing table read as 3, the exit code, the dry run, no guard) and 4 of the footer and render-failure paths.Pipeline position
On the ship-time copy only, never on the master database the other tools edit:
a/pages,drop_unread_rows.py(ADFA-3305: Drop documentation rows nothing reads, and tooltip details that repeat the summary #46), and the Kotlin trim.Coming after
drop_unread_rows.pyis an exception to that step's rule that it runs after every step writing page content. It is safe because this step adds no link: every converted page renders back byte for byte, so it links to exactly what the row drops already checked in it as HTML.drop_unread_rows.pyreads templates and templated rows' JSON too, so it could also run after this step.Not covered
DocumentationContentSourcesets none. On a desktop JVM, all 1,119 includes compiled hold 33 MB of heap and the largest 2.35 MB; a widget page pulls in up to 1.2 MB of include text. Not measured on a phone. That is CodeOnTheGo code, so it is left to a decision outside this PR.Device check
order) as the debug override. Ten pages gave the same text as on the schema-3 database before this step, including
the converted Android reference page TextView, which loaded in 451 ms; androidx EmojiButton (converted) 122 ms;
R.styleable (still HTML, 1.8 M characters) 1,992 ms. A 2.x database still rendered its templated Kotlin and
Help+Manual pages.
🤖 Generated with Claude Code
https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3