[davidschachterADFA]

ADFA-3305

Keeps javadoc's split index and drops its one-page copy. j/html/api/index-all.html (23 MB decoded) repeats the
split index index-files/index-1.html to index-27.html, and every javadoc page's Index link opened it. On a Pixel 6
Pro, in one timepage.mjs run per page with the cache disabled, it took 5,033 ms to its load event, against 537 ms
for index-1.html and 832 ms for index-19.html. Stacked on #46, whose step it extends.

The change

• A fifth drop_unread_rows.py group, javadoc-index-all (on by default): every quoted href or src that resolves
  to a one-page index, its file name matched in any letter case, is pointed at index-files/index-1.html, keeping its
  own relative or rooted prefix; a fragment goes to the split page holding its anchor (index-1.html first when
  several do). Then the one-page index (and its chunk row) is deleted.
• The reference check reads the rewritten rows, so a link the rewrite does not reach - unquoted, built by a script,
  in an attribute whose name only ends in href or src (xlink:href, data-href), in a templated row's JSON, or
  in a template (decoded first from version 3), none of which is rewritten - refuses the run. So do a fragment no
  split page holds, a missing or unreadable split index, and a page named index-all.html that is not a javadoc
  index.
• Everything ADFA-3305: Drop documentation rows nothing reads, and tooltip details that repeat the summary #46 guarantees holds: one transaction, backup, VACUUM, --dry-run, exit 1 means nothing written, a
  second run does nothing.
• rebuild_database.py --pack has to follow it. Each rewritten row stores a few bytes more (+52,829 over 5,222 rows)
  and VACUUM keeps row order, so in a file that was already packed they move to new pages: on the ship-time build the
  step took the file from 90,783,744 to 99,475,456 bytes, and the pack brought it to 89,542,656.

Real data (the 2026-10-19 ship-time build, every ADFA-3305 step, 90,783,744 bytes)

• 5,224 links in 5,222 javadoc pages pointed at the split index; nothing else referred to the one-page index. All
  are lower-case, quoted hrefs with no fragment.
• Packed with rebuild_database.py --pack Content,TooltipTexts --page-size 4096 and compared with the same database
  repacked: file 1,241,088 bytes smaller (90,783,744 -> 89,542,656), .br (brotli -q 11 -w 24) 1,250,482 smaller
  (72,685,042 -> 71,434,560). Repacking alone moves the .br by about 0.14 MB.
• A dry run decodes all 1,126 compressed templates and finds none linking to index-all.html. A copy whose
  page.peb links to it refuses (template page.peb links to j/html/api/index-all.html); before the rebase onto
  ADFA-3305: Drop documentation rows nothing reads, and tooltip details that repeat the summary #46's template decoding, it exited 0.
• An independent check that shares no code with the step: only index-all.html and its chunk row are gone, every
  changed page equals its original but for the link, every new link resolves to a stored row, and no kept row,
  template or tooltip mentions index-all in any letter case. A second run reports nothing to do.

Tests

52 tests (15 new), passing under pytest (Python 3.11 and 3.14) and as a plain script. All 15 fail on #46's head.

• The review fixes add 5. One pins the letter-case rule: a row whose only link is INDEX-ALL.HTML is rewritten; it
  fails on 5cf9c35, where the row was left as it was. Four pin paths no test covered, so they pass on 5cf9c35 and
  were checked by mutants instead: upper-case HREF/SRC and file names, xlink:href/data-href left to the
  reference check, a percent-encoded fragment, and an id on several split pages.
• Mutants: the review's 24 plus 2 of the letter-case rule, 26 in all; 23 fail a test. The 3 survivors are
  equivalent: dropping the cheap row filter (it skips only rows the rewrite cannot change), dropping the text/html
  filter on split pages (the ship-time build's index-files/ holds only its 27 HTML pages), and dropping the \b
  in the anchor pattern (it would also match an attribute such as valid="..."; on those 27 pages it finds the same
  ids either way).

Pipeline position

Unchanged from #46: after every step that writes page content or links, then rebuild_database.py --pack, which
must follow it.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

[davidschachterADFA]

Rebased onto #46's e31aa00 (R50-05, which decodes version 3's templates before checking their links), so this branch was force-pushed: b6bcd09 -> 5cf9c35. The rebase conflicted in the docstring's check list and in CLAUDE.md; both sides' text is kept.

New commit 5cf9c35: the rewrite changes only Content rows, so a template's link to index-all.html must refuse. A new test checks that, with a compressed template at 3.0.0 and a plain one at 2.1.0; it fails on b6bcd09. On a copy of the ship-time build whose page.peb links to the one-page index, the step refuses; before, it exited 0. The 10 tests this PR adds fail on #46's new head. 10 mutants, all killed. The body is updated.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

[davidschachterADFA]

Independent Claude review (head 5cf9c35)

The group does what the body says. On a copy of the 3.0.0 ship-time build it pointed 5,224 links in 5,222 javadoc pages at index-files/index-1.html, deleted index-all.html and its chunk row, and changed nothing else. A check that shares no code with the step confirms it:

• each changed row differs from its original only in index-all.html becoming index-files/index-N.html;
• every new link resolves to a stored row;
• no kept row, template or tooltip mentions index-all in any case;
• every other table is identical.

The file and .br savings reproduce after repacking, and a second run has nothing to do. The findings are at the edges:

• the tests don't pin the fragment and attribute-name paths, which the real data never uses;
• the rewrite loosens a packed file until it is repacked;
• two body figures rest on one run, or reproduce only approximately.

ID	Sev	file:line	Finding
R51-01	low	step:163-165, :273-289	Tests don't pin upper-case links, xlink:href, encoded fragments, shared ids
R51-02	nit	step:111-117	On a packed file the step adds 8.7 MB until the --pack rebuild runs
R51-03	nit	body; step:38-58	Timings are one run per page; the .br saving reproduces 2,335 B lower

step is scripts/drop_unread_rows.py.

R51-01. I ran 24 mutants of the new code against the 47 tests. 16 were killed. Four of the 8 survivors change behaviour that no test pins:

• No re.I on _INDEX_LINK: HREF="index-all.html" is then left alone and refuses, instead of being rewritten.
• No lookbehind: xlink:href and data-href links are then rewritten, where today they are left to the reference check and refuse. Either choice is safe, but the tests don't say which one is intended.
• No unquote on the fragment: index-all.html#I%3AA then refuses instead of going to the page holding I:A.
• The last split page wins an id (assignment instead of setdefault, or reversing the page order): a link to an id on several split pages then goes to the last page instead of the first.

The other 4 survivors don't change behaviour on the fixtures: dropping the index-all.html in raw precheck, the text/html filter on split pages, and the \b in _SPLIT_ANCHOR, plus the page-order mutant on an anchor held once.

The precheck is case-sensitive, but _INDEX_LINK is not. So an upper-case INDEX-ALL.HTML, which the case-sensitive server never resolved, is left as it is in a row with no lower-case link, and rewritten to a working link in a row that has one. That is harmless either way, but the behaviour is untested.

The real data uses none of these forms: all 5,224 links are lowercase, quoted hrefs with no fragment. So today's output doesn't depend on them, but a later javadoc build or minifier could make it depend on them. Fix: one test per case, through point_at_split_index the way test_a_fragment_goes_to_the_split_page_that_holds_its_anchor does it:

• an upper-case HREF, and an upper-case file name;
• an xlink:href, asserting whichever behaviour is intended;
• a percent-encoded fragment;
• an id held by two split pages.

R51-02. The rewritten rows are about 10 bytes larger stored (+52,829 over 5,222 rows), and the VACUUM at the end keeps rowid order. On a copy of the ship-time build, which rebuild_database.py --pack had packed, the file went from 90,783,744 to 99,475,456 bytes:

• Content went from 17,691 pages with 218,195 unused bytes to 19,813 pages with 10,110,787 unused;
• a plain VACUUM of the untouched copy stays at 90,783,744, so the growth comes from the rewrite;
• rebuild_database.py --pack Content,TooltipTexts --page-size 4096 then brings it to 89,542,656.

The documented order (row drops before rebuild_database.py) already avoids this, and the PR's own measurement repacks. But the docstring says only that the step VACUUMs, and run after the pack, as the measurement was, it makes the shipped file larger. Fix: say in "Pipeline position" that the --pack rebuild has to follow, because the rewrite loosens a packed file (+8.7 MB on the ship-time build).

R51-03. Two figures:

• The timings rest on one run each. "5.0 s to load on a Pixel 6 Pro, against 0.5-0.8 s for a split page" comes from one timepage.mjs run per page, with the cache disabled: index-all.html 5,033 ms to the load event, index-1.html 537 ms, index-19.html 832 ms. That is 2 of the 27 split pages. The direction is clear (23,264,909 decoded bytes against at most about 1 MB), but the text reads as a repeated measurement.
• The .br saving reproduces only approximately. Repacked the same way and compressed with brotli -q 11 -w 24, I got 72,685,042 -> 71,434,560, a saving of 1,250,482 bytes, against the body's and docstring's 1,252,817. The file saving reproduces to the byte (1,241,088).

Fix: say "one run each", and give the .br figure for the commit it was measured at (or this one).

Checked and sound

• The rewrite (point_at_split_index):
  • Prefixes: relative ../ at every depth, rooted /j/... and http://localhost:6174/... all keep their own prefix, and the relative path to the split page comes from the index's directory. Both quote styles are tested.
  • Links outside j/ (the fixture's i/apis.html) are rewritten, and a k/ page named index-all.html is neither deleted nor checked.
  • A link with a ?query, an unquoted link, a link built in a script, a link in a templated row's JSON, and a link in a template, plain or compressed at 3.0.0, are not rewritten. The reference check reads the rewritten rows, so each of them refuses; the four that the tests cover are tested.
  • A link whose .. goes past the root refuses: the reference check finds that it names the deleted file. I called point_at_split_index and References directly for the ?query, upper-case and past-the-root cases.
• Anchors on real data: on all 27 split pages, _SPLIT_ANCHOR finds exactly the ids an HTML parser finds. The only ids on several pages are navbar chrome (navbar-top, search-input, reset-button, skip-navbar-top, navbar-top-firstrow).
• Ship-time build, real run (copy of final.db, --only javadoc-index-all, exit 0):
  • Deleted: index-all.html and index-all.html-1, 1,293,715 stored bytes; 23,264,909 bytes decoded, as the body says ("23 MB").
  • Changed: 5,222 j/ HTML rows, with type, language and templateId unchanged.
  • Each rewritten row decodes against the dictionary in place. None starts with a large-window header, so these are the normal 24-bit streams the app's decoder takes.
  • Tables other than Content are identical.
• Size, both copies packed with ADFA-3305: Convert animated GIFs to VP9 WebM and pack documentation.db #38's rebuild_database.py (c947a46): file 90,783,744 -> 89,542,656 (-1,241,088). Repacking alone moves the .br by +138,977, as the body says (about 0.14 MB).
• Re-runs: the dry run reports the same counts and writes nothing, and a dry run on the result says "Nothing to do" for all five groups.
• Dictionary: rows are recompressed with BrotliCodec against CompressionDictionary row 1 of the same file, so no re-mint is needed for them to decode. In the pipeline the re-mint recompresses them anyway.
• Outside the database: nothing in CodeOnTheGo origin/stage or the local addons checkout (c109270) mentions index-all.
• Tests: 47 pass under pytest (Python 3.14 and 3.11) and as a plain script. On ADFA-3305: Drop documentation rows nothing reads, and tooltip details that repeat the summary #46's head (e31aa00) the 10 new ones fail: 3 with KeyError: 'javadoc-index-all' or AttributeError, since the group isn't there yet, and 7 on their assertions. The 16 killed mutants include the prefix, the fragment, the first-page default, the templated-row exclusion, the j/ restriction, the index-page check, the missing and unreadable split index, rows rewritten but not written, rows stored plain, and the scan reading the old text.
• Pipeline position: row drops come after the steps that write pages and before the re-mint and pack. The step reads both template encodings, so it can run on either side of the 3.0.0 migration.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

[davidschachterADFA]

Response to the review

All three findings are addressed (5cf9c35..8e807f4).

ID	Fixed in	Change
R51-01	5612ba3, 52ef08d	Tests for each untested link form; the row filter now ignores letter case, as the rewrite does
R51-02	c27103b	Docstring and CLAUDE.md say rebuild_database.py --pack has to follow the rewrite
R51-03	8e807f4, body	Timings given as one run per page; the .br saving is the one that reproduces

• R51-01, tests (5612ba3): four tests through point_at_split_index and the step:

  • upper-case HREF and SRC, and an upper-case file name, are rewritten;
  • xlink:href, data-href and data-src are left alone, and the reference check refuses the run;
  • #I%3AB goes to the page holding I:B;
  • an id on two split pages goes to index-1.html.

  They pin behaviour the step already had, so they pass on 5cf9c35. The evidence is mutants: they kill "no re.I", "no lookbehind", "no unquote", "last page wins" and "first not first".

• R51-01, letter case (52ef08d): I picked one rule: every link the rewrite's pattern matches is rewritten, in any letter case. The row filter, index-all.html in raw, was case-sensitive; it now uses the pattern's own case rule.

  • The other choice, rewriting none, would have needed a case-sensitive pattern too. That would also stop an upper-case HREF attribute from being rewritten. The only links this rule adds are ones that never resolved on the case-sensitive server, and now do.
  • The new test, a row whose only link is INDEX-ALL.HTML, fails on 5cf9c35: the row keeps its link. Two mutants of the filter (case-sensitive again, no re.I) fail it. Dropping the filter is now equivalent, since it skips only rows the pattern cannot match.

• R51-02 (c27103b): "Pipeline position" and the CLAUDE.md entry say the pack has to follow. On a copy of the ship-time build at this head: +52,829 stored bytes over the 5,222 rewritten rows, file 90,783,744 -> 99,475,456 after the step, and 89,542,656 once packed. These are your figures, reproduced.

• R51-03 (8e807f4): the docstring and body give the timings as one timepage.mjs run per page, cache disabled: index-all.html 5,033 ms to the load event, index-1.html 537 ms, index-19.html 832 ms. The .br saving at this head is 1,250,482 bytes (72,685,042 -> 71,434,560), the same as yours, so both now say that. The file saving is unchanged at 1,241,088.

• Real data at this head:

  • 5,224 links in 5,222 rows, as before; every real link is lower-case, so the letter-case change rewrites nothing more there.
  • Your independent check, rerun: only index-all.html and its chunk row are gone; every changed row differs only in the link; every new link resolves; no kept row, template or tooltip mentions index-all.
  • A second run has nothing to do. final.db's sha256 is unchanged.

• Mutants: your 24, plus 2 of the letter-case rule. 23 of 26 fail a test. The 3 survivors are equivalent:

  • dropping the row filter;
  • dropping the text/html filter on split pages (index-files/ holds only its 27 HTML pages);
  • dropping the \b in the anchor pattern (on those 27 pages it finds the same ids either way).

• Tests: 52 (15 from this PR), passing under pytest (Python 3.11 and 3.14) and as a plain script. All 15 fail on ADFA-3305: Drop documentation rows nothing reads, and tooltip details that repeat the summary #46's head, e31aa00.

• Noted, not changed: after index-1.html, split pages are read in name order, so index-10.html comes before index-2.html. Only an id on several pages that leaves out index-1.html would notice. On the ship-time build the only shared ids are navbar ones, which every page holds, so they go to index-1.html.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

The base branch was changed.