Skip to content

ADFA-3305: Store the a/ reference pages as JSON plus shared templates - #49

Open
davidschachterADFA wants to merge 17 commits into
proto/ADFA-6618from
fix/ADFA-3305-android-templates
Open

davidschachterADFA wants to merge 17 commits into
proto/ADFA-6618from
fix/ADFA-3305-android-templates

Conversation

@davidschachterADFA

@davidschachterADFA davidschachterADFA commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

ADFA-3305

Adds scripts/android_pages_to_templates.py, a ship-time step that stores the Android reference pages under a/ as JSON plus shared Pebble templates. After the 3.0.0 migration compresses the templates and the final re-mint, documentation.db is 7.4 MB (7.5%) smaller and its .br 7.5 MB (9.4%).

Stacked on #43, whose Pebble render checker it uses. It needs the 3.0.0 migration to compress the templates (stacked on #45) and an app that reads compressed templates (CodeOnTheGo, stacked on #2163). All of these ship together as schema 3.0.0.

Includes #56, merged into this branch on 2026-10-20: the step also stores once every text that 5 or more a/ pages repeat, in 524 bundle templates. That saves a further 928,748 bytes of download and 905,216 of file. Its 25 tests bring test_android_pages_to_templates.py to 62. See #56 for the change, measurements and phone timings; the figures below are this PR's own.

How

  • Records: a page is cut into raw HTML and records of 21 fixed shapes: summary rows, detail headers, parameter rows, table heads, in Doclava and Dackka markup. One shape text gives both the regex that reads a record and the template branch that writes it back. The fixed head and footer variants, shared by at least 5 pages, become shapes too.
  • Shared includes: each inherited-member block ("From class X" plus X's rows) becomes one include per exact header, android-inherited-<n>.peb, with every row behind {% if x contains i %}. A page stores the rows its class overrides, which is 0 for the median block. The widgets each carried about 0.5 MB of View's rows.
  • Main template: android-reference.peb loops over a page's segments.

Checks before writing

  • Every page is rendered with Pebble 4.1.1, built as the app builds it with the 4 Mi-character limit, and compared byte for byte. A page that does not match, or is over the limit, stays HTML and is named.
  • An a/ row in another template, a page that is not UTF-8 or holds a NUL, and a/devsite/ are left as they are.
  • Exit 1 always means nothing was written. That covers a database that already declares documentation version 3 or later, an unreadable a/ row, a missing renderer, and a database that already holds this step's templates (it does not run twice). The run uses one transaction, a backup first, VACUUM after, and --dry-run.
  • Version 3 or later refuses before the renderer starts and before anything is written, in a dry run too. Its newest DocumentationDatabaseVersion row is read as the app reads it (changeTime, then rowid), with the helper names of ADFA-3305: Compress every template in the 3.0.0 migration #50's and ADFA-3305: Re-mint and verify compressed templates in schema 3 #48's guards. From 3.0.0 every template is compressed and the app decodes every one, so this step's plain templates would fail every converted page. On a copy of the ship-time input migrated to 3.0.0 with ADFA-3305: Compress every template in the 3.0.0 migration #50's script, the step before this guard exited 0 and left 1,120 plain templates beside the 6 compressed ones; it now prints one error: line, exits 1, and the file's sha256 is unchanged.

The other Android pipeline (retired)

ADFA-5548's ProcessDocs/AndroidDocs/AndroidDocsToJson/load_android_json_db.py does the same job another way: it replaces the scraped a/android/ and a/androidx/ rows with JSON from the committed source tree, through its own android-class.peb, android-package.peb and android-index.peb. A row already rendered through another template is left alone by this step, so after that pipeline had run, this step would convert almost nothing and the saving below would not happen. This step is the one that ships: 0765f83 removes the two workflows that ran that pipeline on the master (build-android-docs.yaml, build-android-docs-local.yaml, neither ever run) and run-build-android-docs-with-act.sh, and drops the Kotlin workflow's note about sharing their concurrency group. The pipeline's scripts and tests stay; its README, CLAUDE.md and this step's docstring say which ships.

Results

On the schema-3 test database (98,201,600 bytes):

Pages 12,827 of 12,828 converted; R.styleable (4.46 M characters) stays HTML
Includes 1,119, covering 13,039 of 13,042 blocks; 1 header kept inline (its pages disagree on row order)
Templates as text 12,135,203 bytes (compressed by the 3.0.0 migration)
End to end 23,844 items read back (chunks joined) serve the same bytes; 12,827 rendered through the stored templates

Sizes are measured against the same database with only its templates compressed. Both copies were re-minted with 4 seeds (templates sampled and recompressed with the rows), then packed with rebuild_database.py --pack Content,TooltipTexts --page-size 4096 (#38), and the .br made with brotli -q 11 -w 24:

Baseline This step Saved
File 98,271,232 90,857,472 7,413,760 (7.5%)
.br 80,139,063 72,644,272 7,494,791 (9.4%)

On the end-to-end input of 2026-10-19, measured in the review (1-seed re-mint, migrated and packed the same way): file 98,770,944 -> 90,927,104 (-7.9%), .br 80,704,825 -> 72,669,139 (-10.0%). In that build, all 12,827 converted pages render byte-identical to the pages just before this step, read independently of this step's code.

Seed-to-seed spread within each re-mint: at most 0.4 MB. Render time on a desktop JVM, JSON parse included: median 0.15 ms per page; the largest page, R.attr (2.1 M characters), took 13 ms. Not yet measured on a phone.

Pebble rules found on the way

  • Pebble drops one line break after any tag, including U+0085 (NEL), U+2028 and U+2029. Text that starts with one gets that character doubled. The source writes these characters, and the Unicode spaces below, as \u escapes.
  • Inside {% verbatim %}, Pebble strips all leading whitespace (Java's Character.isWhitespace plus U+0085, measured one character at a time), so that whitespace is written before the block. No real page needed this today, and the tests cover it.
  • A final { before a tag is printed from a string literal. Text that would end a verbatim block is never made template text.

Tests

  • scripts/test_android_pages_to_templates.py: 37 tests on synthetic Doclava and Dackka pages. They pass under pytest (Python 3.11 and 3.14) and as a plain script. Tests that render need java and the Pebble jars, as test_hm_pages_to_templates.py does; neither file runs in CI (python-tests.yaml does not run scripts/).
  • The review fixes add 5:
    • three for the version guard: 3.0.0 refuses, written or dry; the newest row is read by changeTime, then rowid (a later change in an older row, and a tie); no version table, an empty one and version 2 still convert. The first two fail on 12b9920 (exit 0, pages converted).
    • a block whose footer is wrong, alone and followed by a whole block of the same style;
    • a render that fails with an error other than the limit stays HTML, named with the renderer's first line.
  • Mutants: the first version's 30, of which 28 fail a test; the 2 that pass are equivalent: no explicit rollback (closing the connection discards the transaction), and a match that does not advance (every shape starts with fixed text). The review's two real survivors (the footer check, the not-ok branch) are now killed. The fixes add 16 more, all killed: 12 of the version guard (row order, max(major), the threshold, an empty or missing table read as 3, the exit code, the dry run, no guard) and 4 of the footer and render-failure paths.

Pipeline position

On the ship-time copy only, never on the master database the other tools edit:

  1. The steps that write a/ pages, drop_unread_rows.py (ADFA-3305: Drop documentation rows nothing reads, and tooltip details that repeat the summary #46), and the Kotlin trim.
  2. This step.
  3. The 3.0.0 migration, which compresses every template.
  4. The final 4-seed re-mint, which re-mints templates too.
  5. Packing.

Coming after drop_unread_rows.py is an exception to that step's rule that it runs after every step writing page content. It is safe because this step adds no link: every converted page renders back byte for byte, so it links to exactly what the row drops already checked in it as HTML. drop_unread_rows.py reads templates and templated rows' JSON too, so it could also run after this step.

Not covered

  • The app keeps every compiled template: Pebble 4.1.1's default cache is unbounded and DocumentationContentSource sets none. On a desktop JVM, all 1,119 includes compiled hold 33 MB of heap and the largest 2.35 MB; a widget page pulls in up to 1.2 MB of include text. Not measured on a phone. That is CodeOnTheGo code, so it is left to a decision outside this PR.

Device check

  • Done on 2026-10-19 (Pixel 6 Pro, Android 17), with the CodeOnTheGo app PR ADFA-3305: Read compressed templates from documentation version 3 CodeOnTheGo#2164 built and the ship-time database (every step, in
    order) as the debug override. Ten pages gave the same text as on the schema-3 database before this step, including
    the converted Android reference page TextView, which loaded in 451 ms; androidx EmojiButton (converted) 122 ms;
    R.styleable (still HTML, 1.8 M characters) 1,992 ms. A 2.x database still rendered its templated Kotlin and
    Help+Manual pages.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

android_pages_to_templates.py turns every a/ text/html page (except a/devsite/)
into JSON rendered through android-reference.peb. Repeated structures become
records of fixed shapes, and each inherited-member block header becomes one
shared include, android-inherited-<n>.peb, with every row behind a guard and the
rows a page's class overrides listed in its JSON.

Every page is rendered back with Pebble 4.1.1 and compared byte for byte before
anything is written; one that does not match, or is over the render limit, stays
HTML. Templates are written as plain text for the 3.0.0 migration to compress.

On the schema-3 test database: 12,827 of 12,828 pages convert (R.styleable
stays HTML), with 1 main template and 1,119 includes. After template
compression and a 4-seed re-mint the file is 7.4 MB (7.5%) smaller and the .br
7.5 MB (9.4%). 32 tests; 28 of 30 mutants are caught, and the 2 left are
equivalent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
@davidschachterADFA

Copy link
Copy Markdown
Collaborator Author

Independent Claude review (head 12b9920)

The conversion is faithful. In the 3.0.0 ship-time build, each of the 12,827 converted pages, served the way the app serves it, is byte-identical to the same page before the conversion. That check joins the chunks by the app's rule, decodes against the build's dictionary, decodes the templates (it is a 3.0.0 file) and renders with the app's engine settings. The 415 a/ HTML rows the step leaves alone are identical too. The render harness matches DocumentationContentSource setting for setting, and the body's counts and size saving reproduce on the end-to-end input. The findings are at the edges: no version guard, a second pipeline for the same pages that the PR doesn't mention, the heap the compiled includes hold in the app, and small test and source-hygiene gaps.

ID Sev file:line Finding
R49-01 low step:459 No version guard: on a 3.0.0 file it adds 1,120 plain templates, exit 0
R49-02 low body; load_android_json_db ADFA-5548's workflow converts the same pages; after it, this does none
R49-03 low CoGo content source Compiled includes stay cached: 33 MB of heap if all are used
R49-04 nit step's tests Untested: a block's footer, and a render failure other than the limit
R49-05 nit step:194, :199-200 U+2028, U+2029 and 13 Unicode spaces written literally, not escaped
R49-06 nit docstring; #46 #46 says row drops follow every page-writing step; this says before it

step is scripts/android_pages_to_templates.py.

R49-01. The step never reads DocumentationDatabaseVersion. I ran it on a copy of the pre-conversion database after migrate_tooltip_schema.py (#50) had already declared 3.0.0. It exited 0, having added android-reference.peb and 1,119 includes as plain UTF-8 beside the 6 compressed templates. Its closing line says "Next: the 3.0.0 migration (compresses the templates)". The migration then refuses ("1,120 of 1,126 template(s) do not decode ... a step run after the conversion wrote them; nothing was changed"), so that advice leads nowhere. brotli -d -D on the stored main template gives corrupt input. The app would fail every converted a/ page in the same way. #48's re-mint also refuses such a file, so the documented order still catches it before shipping. A run after the re-mint would not be caught. #50's 2950b1b added this guard to every other template writer and named this step as unguarded. Fix: refuse a database whose newest version row (by changeTime, then rowid) declares major 3 or later, before the backup, with an error: line. Add a test.

R49-02. ProcessDocs/AndroidDocs/AndroidDocsToJson/load_android_json_db.py (ADFA-5548, on main) also replaces the scraped a/android/ and a/androidx/ HTML rows with JSON. It writes android-class.peb, android-package.peb and android-index.peb, and build-android-docs.yaml runs it against the master on Drive (workflow_dispatch; no runs yet). After it has run, this step leaves every such row as "rendered through another template" and converts almost nothing, so the body's 7.4 MB saving would not happen. Nothing breaks: the step skips those rows safely. Neither the body nor the docstring mentions the other pipeline. Fix: say which of the two ships, and either retire the other or state that this step is meant for scraped HTML only.

R49-03. Pebble 4.1.1's default template cache is an unbounded ConcurrentMapTemplateCache (from the bytecode of PebbleEngine$Builder.build). DocumentationContentSource sets no cache, so every include a page compiles stays in memory until the next database swap or clear-cache. On a desktop JVM I compiled the shipped templates and measured the heap:

  • all 1,119 includes hold 33,080,032 bytes;
  • the largest, android-inherited-406.peb (810,416 bytes of text, used by 126 pages), holds 2,351,240.

A median page uses no include. A widget page pulls in up to 1.2 MB of include text: a/androidx/leanback/widget/VerticalGridView.html uses 7, and TextView 6 (346 KB). Before this step, a/ pages compiled nothing. Not measured on the phone. Fix: measure the app's heap on a device after opening many widget pages (dumpsys meminfo). If it matters, give the engine a bounded PebbleCache in CodeOnTheGo.

R49-04. I ran 37 mutants; 29 were killed. Two of the 8 survivors are real gaps:

  • find_blocks accepting a header and rows without the expected footer. The render check would still keep such a page as HTML, so this costs size, not fidelity.
  • Removing the elif not ok: branch, which sends a render error other than the size limit to the "renders differently" message.

Fix: one test for each.

R49-05. _PEBBLE_NEWLINE (line 194) and _VERBATIM_STRIPS (199-200) hold U+2028, U+2029 and 13 Unicode spaces (U+1680, U+2000-U+200A except U+2007, U+205F, U+3000) as literal characters. The test file has U+2028/U+2029 literally too, at 328 and 333. They show as blanks in the diff and in an editor, so a reviewer can't see which characters the set holds; the missing U+2007, which is correct, is invisible. Python's str.splitlines(), and tools built on it, break those lines at U+2028/U+2029. Fix: write them as \u escapes.

R49-06. #46's drop_unread_rows.py docstring (e31aa00) says it runs "after every step that writes page content, links or tooltips". This step writes page content, and its docstring and CLAUDE.md entry put it after drop_unread_rows.py. The end-to-end build used that order and verified it, and the links don't change: the a/ includes hold absolute https://developer.android.com/... links. So the order works, but the two docstrings contradict each other. Fix: let one of them name the order exception.

Checked and sound

  • Fidelity, end to end:

    • I compared /home/david/adfa-e2e/O1/final.db (3.0.0: re-minted and packed after this step and the migration) with o.db.backup-20261019-000940 (the same build just before this step). Of 13,242 a/ HTML rows, the 12,827 templated ones render byte-identical to the originals, with 0 render failures. The 415 untemplated rows (414 a/devsite/ pages and R.styleable) are identical.
    • Both databases were read independently of the step's own code: chunks joined by DocumentationContentSource.readChunks's rule, and templates decoded from the version row.
  • Harness parity (Render.java against #2164's DocumentationContentSource and DatabaseTemplateLoader):

    • Pebble 4.1.1 with the builder defaults: autoescaping on (every print is | raw), strict variables off (s.k is null for raw segments), newline trimming on.
    • maxRenderedSize of 4 Mi characters.
    • Gson 2.12.1 with LONG_OR_DOUBLE. That makes x contains 3 compare Longs, and makes "android-inherited-" ~ s.i produce ...-5.peb, not ...-5.0.peb.
    • Names resolved exactly, resolveRelativePath the identity, and output as UTF-8 bytes.
  • ART:

    • Pebble's line-break rule is an explicit list (Syntax.POSSIBLE_NEW_LINE, including U+0085, U+2028 and U+2029), so it is the same on ART.
    • Verbatim stripping is StringUtils.ltrim (Character.isWhitespace), which can differ on ART. The shipped templates hold no verbatim block and no whitespace-control tag, so nothing depends on that.
    • The 292 line breaks right after a tag (291 \n, 1 \r) are doubled correctly, and the fidelity check covers them.
  • Counts: the e2e log and both of my runs give 12,828 pages found and 12,827 converted (R.styleable, at 4,455,003 characters, is over the limit). They also give 13,042 blocks under 1,120 headers, 1,119 includes covering 13,039 of them, and 12,135,203 bytes of template text. All match the body.

  • Size: I ran the end-to-end input (pre-conversion) without this step and with it. Each run re-minted (1 seed, before the migration), migrated, packed with --pack Content,TooltipTexts --page-size 4096, and compressed with brotli -q 11 -w 24:

    • file: 98,770,944 -> 90,927,104 (-7,843,840, -7.9%);
    • .br: 80,704,825 -> 72,669,139 (-8,035,686, -10.0%).

    The body's -7.4 MB / -7.5 MB was measured on another input with 4 seeds.

  • Cost: on a desktop JVM, the median render is 0.14 ms per page. R.attr took 36 ms in my run (one render, early in the run); the body says 13 ms. The first compile takes 24 ms for the main template and 35 ms for the largest include. A dry run on real data takes 79 s with 6 workers, peaks at 3.7 GB RSS, and leaves the file unchanged.

  • Tests: 32 pass under pytest and as a plain script. Besides R49-04's two, the mutation survivors are equivalent:

    • no explicit rollback;
    • a match that does not advance;
    • the 12-character prefilter;
    • MIN_SHARED_PAGES at 4 or 6 (the test reads the constant);
    • a page that begins with the head mark.

    These kill their mutants: the doubled line break, the final {, the endverbatim refusal, the verbatim lead, self-link fragments, row splitting, cycles, the second-run refusal, the a/devsite/ exclusion, other-template, NUL and UTF-8 rows, the byte compare, unused includes, the dry run, templateId, chunk rows, the VACUUM, and storing the JSON compressed.

  • Sibling sweep:

  • Not checked: the device check in the body (I did not use the phone); the body's size table on its schema-3 test database, which is not available; the heap on ART.

  • Outside this PR: the a/ pages carry off-device links (class="external-link" href="https://developer.android.com/..."), now in 505 of the 1,119 includes. That content is unchanged by this step.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

davidschachterADFA and others added 5 commits October 19, 2026 09:01
The step never read DocumentationDatabaseVersion. Run after the 3.0.0
migration, it exited 0 having added android-reference.peb and its
includes as plain UTF-8 beside the compressed templates, and its closing
line sent the user to the migration, which then refuses the file. The
app decodes every template from 3.0.0 on, so every converted a/ page
would fail.

It now refuses a database whose newest DocumentationDatabaseVersion row,
ordered as the app orders them (changeTime, then rowid), declares major 3
or later: one error: line, exit 1, before the renderer starts and before
anything is written, in a dry run too. The helpers have the names and
semantics of #50's guard in the other template writers (2950b1b) and
#48's declared_major, so the guards read alike.

Tests: a 3.0.0 database refuses, written or dry; the version row is read
by changeTime then rowid (a later change in an older row, and a tie);
no version table, an empty one and version 2 still convert. The first two
fail on 12b9920 (exit 0, pages converted); the third passes there, as it
must. 12 mutants of the guard (row order, max(major), threshold, an empty
or missing table read as 3, exit code, dry run, no guard), all killed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
Two guards survived the review's mutants. find_blocks could accept a
header and rows without the style's footer, and a render error other
than the size limit could be reported as "renders differently" instead
of "does not render". Neither changes what is stored: the render check
keeps such a page as HTML either way. But the first costs size and the
second gives the wrong reason.

One test now has a block of each style with a wrong footer, alone and
followed by a whole block of the same style; only the whole one is
found, at its own offsets. Another renders one page through a renderer
that fails with an error other than the limit; that page stays HTML,
named with the renderer's first line, and the others convert.

Mutants killed: the footer check dropped, a footer accepted anywhere
later in the page, the not-ok branch removed, and every failure read as
the limit (4 of 4).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
_PEBBLE_NEWLINE and _VERBATIM_STRIPS held U+2028, U+2029 and 13 Unicode
spaces (U+1680, U+2000-U+200A but U+2007, U+205F, U+3000) as literal
characters, and the test file held U+2028 and U+2029 the same way. They
show as blanks in a diff or an editor, so a reader could not see which
characters the sets hold, or that U+2007 is rightly missing; and
str.splitlines(), and tools built on it, break those lines at U+2028 and
U+2029. They are \u escapes now.

No behaviour change: both files parse to the same AST as before, the
strip set holds the same 26 characters, and a dry run on the ship-time
input gives the same output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
ProcessDocs/AndroidDocs/AndroidDocsToJson/load_android_json_db.py
(ADFA-5548, on main) also replaces the scraped a/android/ and
a/androidx/ rows with JSON, through its own android-class.peb,
android-package.peb and android-index.peb. build-android-docs.yaml runs
it on the master, by hand only; it has no runs so far. Neither the
docstring nor CLAUDE.md said so.

Both now do. This step converts scraped HTML on the ship-time copy and
leaves a row already in another template as it is, so after that
pipeline has run it converts almost nothing and saves almost nothing.
Which of the two ships is still open. No behaviour change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
#46's drop_unread_rows.py says it runs after every step that writes
page content, links or tooltips. This step writes page content, and its
docstring put it after drop_unread_rows.py, which also listed the row
drops among the steps that write a/ pages. The end-to-end build used
that order and verified it, but the two docstrings contradicted each
other.

The docstring and the CLAUDE.md entry now name the order as an
exception to #46's rule, and say why it is safe: every page this step
converts renders back byte for byte, so it links to exactly what the row
drops already checked in it as HTML. The includes do hold links (41,153
rooted ones, 40,991 of them under /a/, and about 15,400 absolute ones in
the ship-time build's), but none that its pages did not hold before. drop_unread_rows.py
reads templates and templated rows' JSON too, so it could also run
after this step. No behaviour change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
@davidschachterADFA

Copy link
Copy Markdown
Collaborator Author

Response to the review

Five findings are fixed here (12b9920..48a3734). R49-03 is in CodeOnTheGo, so it's left for a decision outside this PR.

ID Fixed in Change
R49-01 66249e1 Refuses a database declaring version 3 or later: one error: line, exit 1, before anything
R49-02 1fc3fb9 Docstring, CLAUDE.md and body name ADFA-5548's pipeline for the same pages
R49-03 - Not changed: CodeOnTheGo code. In the body's "Not covered"
R49-04 621e78e Tests for a block's footer and a render that fails with another error
R49-05 94b9d29 U+2028, U+2029 and the 13 spaces written as \u escapes, in the step and its tests
R49-06 48a3734 Docstring and CLAUDE.md name the order after drop_unread_rows.py as an exception, and why
  • R49-01: the guard runs before the renderer starts and before the backup, in a dry run too. It reads the newest DocumentationDatabaseVersion row by changeTime, then rowid, with the helper names and semantics of ADFA-3305: Compress every template in the 3.0.0 migration #50's 2950b1b and ADFA-3305: Re-mint and verify compressed templates in schema 3 #48's declared_major.
    • Real data: I migrated a copy of the ship-time input (o.db.backup-20261019-000940) to 3.0.0 with ADFA-3305: Compress every template in the 3.0.0 migration #50's script. On that copy, 12b9920 exited 0 and left 1,120 plain templates beside the 6 compressed ones, still declaring 3.0.0; none of the 1,120 decodes against the dictionary. The fixed step prints one error: line and exits 1, dry or not, with the sha256 unchanged and no backup.
    • Tests: three new ones. A 3.0.0 database refuses, written or dry. The newest row is read correctly: a later changeTime in an older row, and a tie broken by the later rowid. No version table, an empty one and version 2 still convert. The first two fail on 12b9920 (exit 0, pages converted); the third passes there, as it must.
    • Mutants: 12 of the guard, all killed. They cover rowid alone, changeTime alone, a tie broken by the older rowid, max(major), a threshold of 2 or 4, <=, an empty or missing table read as 3, exit 0, the guard skipped in a dry run, and no guard.
  • R49-02: documentation only. build-android-docs.yaml has no runs (total_count 0). Which pipeline ships is still open.
  • R49-03: not changed here. The default cache, and whether to bound it, is CodeOnTheGo code. The body's "Not covered" now gives your figures (33 MB for all 1,119 includes on a desktop JVM, 2.35 MB for the largest, not measured on a phone).
  • R49-04: the two tests pin behaviour the step already had, so the evidence is mutants. Your two survivors are now killed: the footer check dropped, and the elif not ok: branch removed. So are two more I added: a footer accepted anywhere later in the page (the test puts a whole block after the broken one), and every failure read as the limit. 4 of 4.
  • R49-05: both files parse to the same AST as before, and the strip set holds the same 26 characters. A dry run on the ship-time input gives byte-identical stdout and stderr before and after: 12,828 pages found, 12,827 to convert, 1,119 includes, 12,135,203 bytes of templates. The é in two test pages is still literal, since it's visible and is test data.
  • R49-06: the reason is not that the includes hold only absolute links. In the ship-time build, the 1,119 includes hold 41,153 rooted links (40,991 under /a/) and about 15,400 absolute ones. The order is safe because every converted page renders back byte for byte, so it links to exactly what the row drops already checked in it as HTML. drop_unread_rows.py also reads templates and templated rows' JSON, so it could run after this step too. ADFA-3305: Drop documentation rows nothing reads, and tooltip details that repeat the summary #46's docstring could name this exception from its side as well; this PR doesn't edit ADFA-3305: Drop documentation rows nothing reads, and tooltip details that repeat the summary #46.
  • Tests: 37, passing under pytest (Python 3.11 and 3.14) and as a plain script. python-tests.yaml doesn't run scripts/, so CI doesn't cover them.
  • Outside this PR: the off-device links you noted: about 15,400 absolute links in the includes, to developer.android.com and 10 other hosts. They came with the pages, and this step doesn't change them.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

The a/ pages ship converted on the ship-time copy by
android_pages_to_templates.py. build-android-docs.yaml and
build-android-docs-local.yaml ran ADFA-5548's pipeline on the master
instead, replacing the same a/android/ and a/androidx/ rows with JSON
through other templates; after one ran, this step would have converted
almost nothing. Neither has ever run. The run-build-android-docs-with-act.sh
driver goes with them, and build-kotlin-docs.yaml no longer says it shares
the Drive file's concurrency group with the Android workflow.

The pipeline's scripts and tests stay. The README, CLAUDE.md and this
step's docstring now say which pipeline ships.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
@davidschachterADFA

Copy link
Copy Markdown
Collaborator Author

New commit 0765f83 retires ADFA-5548's Android docs workflows, as decided on ADFA-3305: this step's conversion on the ship-time copy is the one that ships. It removes build-android-docs.yaml and build-android-docs-local.yaml (neither has ever run) and run-build-android-docs-with-act.sh, and drops build-kotlin-docs.yaml's note that it shares the documentation-db concurrency group with the Android workflow; the group itself is unchanged. load_android_json_db.py and the rest of that pipeline's scripts and tests stay, and its README now says they are not run in CI. CLAUDE.md, this step's docstring and the body's "The other Android pipeline" section are updated. The step's 37 tests pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

@davidschachterADFA

Copy link
Copy Markdown
Collaborator Author

R49-03 measured on a Pixel 6 Pro

The app's compiled-template cache, measured with the ADFA-3305 debug build and the 3.0.0 test database. The method: heap dumps after a forced GC. A template's retained size is the bytes reachable from GC roots minus those still reachable once the cache is taken away.

Measure Value
20 a/ widget pages in one session +4.53 MB retained (meminfo's heap growth agrees within 2%); the session's 51 templates hold 7.79 MB
Per page in that session about 226 KB
Per include (52 measured) mean 154 KB, median 11.3 KB; compiled is 2.65 x the text (2.39 to 2.98)
Largest include, android-inherited-406.peb 2,234,152 bytes (desktop: 2,351,240)
All 1,119 includes, estimate (12.1 MB of text x 2.65) about 32.1 MB (28.9 to 36.0), matching the review's 33.08 MB on a desktop JVM
Heap limit 512 MB (largeHeap)

Conclusion: no change in this PR. The worst case is about 6% of the heap, and reaching it takes viewing about 741 particular pages in one process. Caching also saves recompiling large shared includes on every page that uses them. If low-RAM devices show memory pressure, the fix belongs in CodeOnTheGo: a least-recently-used PebbleCache capped by template size, with no new dependency.

The heavy session (741 pages, every include) was not run on the phone. The installed build predates ADFA-6589, so each page view still triggers Safe Browsing lookups off the device. The all-includes figure is therefore an extrapolation from the 52 includes measured, which are 25% of the include text, the largest among them.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

davidschachterADFA added a commit that referenced this pull request Oct 19, 2026
CodeOnTheGo's compress_docdb.yml only Brotli-compresses the master from
Drive and release.yml bundles that file, so today's release keeps the
master's 256 KiB dictionary. The Kotlin and Java workflows don't call the
re-mint; only the Android pair does, and #49 removes it. CLAUDE.md now
says the new default applies to a re-mint run by hand and to the
ship-time pipeline once one exists, and its 2026-08-20 paragraph about
the master is back to the master's 256 KiB. (R53-01)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
@davidschachterADFA

Copy link
Copy Markdown
Collaborator Author

Found on the phone: the first load of a widget page takes about 100 s

On a Pixel 6 Pro, with the 3.0.0 item-B database, the first load of EmojiButton in a fresh process takes 93-114 s, and TextView takes 22-24 s. Later loads take 0.3-0.4 s. A bisect of the app PRs (#2164 alone, plus #2161, plus #2162) cleared all three: #2164 alone is just as slow.

Cause: Pebble 4.1.1's lexer is quadratic on Android, and this PR's largest shared includes are big enough to hit it.

  • Profile: simpleperf puts 99% of the request thread's samples in PebbleEngine.getPebbleTemplate -> LexerImpl.tokenize, mostly memmove and System.arraycopy. This runs while each include is compiled for the first time.
  • Why it's quadratic: LexerImpl calls Pattern.matcher(...) on its TemplateSource at about 19 sites. TemplateSource.toString() copies everything not yet lexed. On Android, each new Matcher turns its input into a String and copies it again into ICU, so every match costs time in proportion to the rest of the template. Desktop Java matches on the CharSequence directly, which is why the render check here was fast.
  • EmojiButton pulls in android-inherited-406.peb (807,447 characters, the inherited View members, shared by 126 widget pages), -408 (122,658) and -411. Other large includes: -39 (334,588), -373 (334,406), -407 (264,704) and -370 (230,631).

Proposed fix in this PR: cap the include size, splitting a large inherited block into several includes of a few KB each. Lexing cost grows with about the square of the size, so splitting the 807 KB include about 50 ways should cut it about 50-fold. That's an estimate to measure on the phone, along with the cost to the size saving. The app needs no change.

The alternatives:

  • patching Pebble's lexer in the app, which fixes every template but means carrying a library patch;
  • precompiling at database open, which only moves the CPU time, and keeps the compiled templates' ~30 MB of heap (R49-03).

Details: the ADFA-3305 bisect notes of 2026-10-19.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

davidschachterADFA and others added 4 commits October 19, 2026 22:11
…dles

After the shapes and inherited-member includes, the a/ JSON still repeats
whole texts across pages: description paragraphs, inherited docs, table
heads, annotations, parameter lists and types. Each page stored its own copy,
and the shared dictionary could not hold them all.

A text of 32 or more characters that 5 or more pages hold, in a raw segment
or a hole other than a link, now goes into a bundle template
android-text-<n>.peb of at most 8,000 bytes of source, one branch per text of
{% if j == 0 %}...{% elseif j == 1 %}...{% endif %}. The page stores
[<n>, <j>] under "<hole>@" ("r@" for a raw segment), and only the holes that
hold one get the include alternative in the main template. Links stay in the
pages: a same-page link's fragment often equals a member's id, and the
prototype that interned it lost the page's URL. A text too long for a bundle
stays in its pages, and a bundle no converted page reads is not written. A
database holding bundles refuses a second run, as one holding the step's
other templates does.

The thresholds come from full ship-time builds (4-seed re-mint, packed,
brotli -q 11 -w 24) of the 2026-10-19 build: 5 pages saved 839,680 file and
850,755 .br bytes, against 614,400 and 799,100 at 2 pages and 643,072 and
588,189 at 20. Against 8,000 bytes, bundles of 16,000 or 32,000 stored 11 or
14 KB less but doubled or quadrupled the bundle source the median page
compiles (13 KB).

Every converted page still renders back byte for byte before anything is
written. 11 new tests; 22 mutants of the new code, all killed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
…e build

The docstring quoted the prototype's run. This branch's full ship-time build, from
the same input as the item-B build (its conversion backup has the same
sha256), interned the same 6,441 texts into 164 bundles and saved 856,064
file and 867,547 .br bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
A page compiles every bundle it uses. Filled in first-use order, a bundle
held texts from unrelated pages, so on the phone every first load got slower:
TextView's 575 references fell in 38 bundles, 290,523 bytes of source to
compile, and its first load took about 0.7 s longer.

Texts that the same pages use now start in one group. Two groups then merge,
best first, while the merge lowers the sum over pages of compiled bundle source
plus 8,000 bytes for each bundle: the pages that use both save a bundle each,
and every other page using one of them compiles the other for nothing. A
merge must still fit one bundle. Bundles are numbered, and texts ordered
within them, by first use.

The interned set does not change: the same 6,441 texts. Offline, on the
2026-10-19 ship-time build's a/ pages, against first-use packing:

                         first use    by pages
  bundles                      164         541
  compiled, median page     15,419       2,653
  compiled, 95th pctile     61,211      14,672
  compiled, largest        290,523     145,235
  TextView                 290,523     116,987  (38 -> 31 bundles)
  bundles a page, p95            8           5

Bundles of 16,000 bytes compiled about as much (TextView in 27 bundles).
Merging by Jaccard similarity of page sets, or giving the heaviest pages
bundles of their own, did worse on one of these or on storage.

7 new tests; 14 mutants of the packing and 20 of the interning, all killed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
The full ship-time build with 8c3ffc8's packing, from the item-B build's own
pre-conversion state: 541 bundles, 954,368 file and 961,649 .br bytes saved
against the item-B build (84,754,432 and 66,172,076), against 856,064 and
867,547 with first-use packing. The difference is within the re-mint's
seed-to-seed spread (about 0.25 MB), so the grouping costs no measurable
storage. 12,827 of 12,828 pages render back byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
davidschachterADFA and others added 6 commits October 20, 2026 08:49
…R56-01)

A heap entry stored its groups' ids as (min, max) but their versions as
(a, b), the group being offered first. A merge keeps the lower id and raises
its version, so an entry offered from the merged group against a
lower-numbered neighbour never matched at the pop and was always dropped:
"until no merge lowers it" was false. Output stayed correct, since a stale
entry is still never accepted.

The versions now follow the ids. In the review's case, X (pages 0-11), A
(0-9 and 12) and B (0-9 and 13): A and B merge first, and AB then merges
with X into one bundle; before, two. The new test fails on 713ab3e.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
The merge pre-check compared the two bundles' sizes as they stand with the
cap, but a merged bundle is up to 7 bytes smaller than its parts: one
"{% endif %}" (11) and one "{% if j == 0 %}" (15) go, and one
"{% elseif j == n %}" (19 or more) comes. So a merge that would just fit was
never offered. The pre-check now allows for those 7 bytes, and the exact
check when a merge is taken still refuses one that does not fit.

A new test merges two groups into a bundle of exactly the cap; on 713ab3e
it gets two bundles. Now that the remainder of a split can merge into a
bundle with room, the two tests of the split itself run with a bundle costing
0, so that only the split is under test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
The review's mutants of the size, length and gain checks survived the
tests. Four tests now pin the boundaries:

- a single text whose bundle is exactly the cap is shared, and one a byte
  longer stays in its pages, at a 300-byte cap and at the real 8,000;
- a text of 32 characters is shared, and one of 31 is not;
- a merge that gains exactly nothing is not taken, and with a bundle worth
  one byte more it is.

They pin behaviour 713ab3e already had, so the mutants are their proof:
the single-text check without its tags, ">=" as ">" and 33 for the length,
7,999 for the cap and ">= 0" for the gain each fail one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
…56-03)

No real a/ text holds "{{", "{%" or "{#", so nothing tested that a shared
text is written into its bundle as literal text. A failure there would show
only as pages quietly staying HTML, which a byte-identity check alone does
not catch. The new run test shares ten such texts across five pages:
"{{", "{%" and "{#", unclosed tags, "{%-", a trailing "{", CRLF, LS, PS,
NEL and non-ASCII. It asserts that the pages convert and serve the same
bytes, that no text stayed in a page, and that a bundle uses verbatim.

It pins existing behaviour: writing the raw text into the bundle instead of
its Pebble source fails it. The PEBBLE_LIKE list came in with a20f1e3 by
mistake; this commit adds the test that uses it and writes its LS and PS as
escapes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
…link holes (R56-04, R56-05)

R56-04: the docstring and constant comments quoted figures from earlier
code. Now:
- the shared-texts paragraph gives this branch's full ship-time build:
  524 bundles, the largest 7,998 bytes; 905,216 file and 928,748 .br bytes
  saved against the item-B build (84,803,584 and 66,204,977); the median
  page compiles 2,653 bytes, the 95th percentile 14,411, the largest
  144,127, TextView 121,032 in 32 bundles;
- the page-count comment says its figures are the first prototype's, with
  bundles filled in first-use order;
- the bundle-size and bundle-cost comments give figures measured offline
  with the fixed packing, in place of first-use packing's;
- the measured block for the shapes and includes says it predates the
  shared texts.

R56-05: only the link holes stay in the pages, not every link: a shared
text may hold a link in its markup (all of them in the a/ pages are
rooted). The docstring and CLAUDE.md now say link holes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3
ADFA-3305: Store texts that five or more a/ pages repeat once, in bundles grouped by page
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants