[davidschachterADFA]

ADFA-3305

Two new pipeline steps make documentation.db 6.5 MB smaller (5.6 MB in the .br download) by deleting rows nothing
reads and emptying tooltip details that only repeat their summary.

• scripts/drop_unread_rows.py deletes four groups of Content rows, with their chunk rows:
  • javadoc-use: javadoc Use pages and module graphs, 4,956 rows. Only other Use pages link to them.
  • pdfjs-cmaps: pdf.js CMaps, 169 rows. No stored PDF has a font that needs one.
  • pdfjs-extras: pdf.js no-WebAssembly fallbacks and scripting sandbox, 5 rows. The step also sets
    enableScripting to false in p/web/viewer.mjs, which is the only way the viewer loads the sandbox.
  • x-placeholders: the x/ "Placeholder for XML layout attribute" pages, 1,238 rows. x.html and x.pdf stay.
• scripts/clear_repeated_tooltip_detail.py sets Tooltips.detail to '' where it is exactly the summary:
  1,605 tooltips (1,592 Kotlin, 13 Java).

Sizes

Base: the 2026-10-18 database after ADFA-6627's first build (#44 at a3c444e; 110,895,104 bytes). Each result, and the base, was packed with
rebuild_database.py --pack Content,Tooltips --page-size 4096 (from the ADFA-3305 branch) and compressed with
brotli -q 11 -w 24. Packed base: 110,895,104 bytes, .br 87,091,418.

Change	Step, group	Rows	File saved	.br saved
D01	drop_unread_rows, javadoc-use	4,956	5,042,176	4,330,908
D04	drop_unread_rows, pdfjs-cmaps	169	860,160	851,736
D05	drop_unread_rows, pdfjs-extras	5	339,968	346,809
D06	clear_repeated_tooltip_detail	1,605	151,552	10,049
D07	drop_unread_rows, x-placeholders	1,238	122,880	103,920
All	all five, in this order		6,520,832	5,635,912

All five together: 104,374,272 bytes, .br 81,455,506. Repacking alone moves the .br by up to about 0.2 MB, so the
D06 and D07 .br figures are within that. Neither step needs a dictionary re-mint: the deletions remove rows, the
viewer edit changes one word, and D06 changes Tooltips, which are not Brotli-compressed.

What users see

Only D06 changes anything visible. The first tooltip popup is unchanged. All 1,605 tooltips have buttons, so See More
still shows; its popup used to show the summary again, a blank line, then the links, and now shows only the links. The
info dialog and the feedback e-mail show an empty Detail.

Safety checks

drop_unread_rows.py writes nothing, and exits 1 with the list, if:

• any kept text row (HTML, CSS, JS, SVG, JSON contexts and the rest, decoded, chunks joined), template, tooltip
  button, tooltip text or Bookshelf entry links to a row it would delete, or names a file only deleted rows have.
  Three references are expected and allowed, and printed: the worker's two fallback imports, which run only when
  WebAssembly cannot start, and the viewer's sandbox path, dead once scripting is off;
• a row to delete is not what its group holds (for example, an x/ page that is not a placeholder);
• a stored PDF has a font pdf.js would fetch a CMap for (font names are read from /BaseFont and /FontName), or
  its fonts cannot all be read: it is encrypted, or a stream other than an image or cross-reference stream cannot be
  decoded (Flate, ASCII85, ASCIIHex and RunLength are decoded, in chains);
• the tooltip tables are in neither documentation schema (2 or 3);
• the viewer does not hold exactly one enableScripting default, or a kept row, a template, a row to delete or a
  PDF cannot be read.

From documentation version 3 (the newest DocumentationDatabaseVersion row, ordered as the app orders them) every
template is stored Brotli-compressed against the dictionary, so the step decodes templates before reading their
links; below 3 it reads them as plain text. On the 2026-10-19 ship-time build (3.0.0, 1,126 compressed templates) the
link-like strings it reads from templates went from 0 to 29,210; decoding them takes about 1.1 s. That build has
already had its rows dropped, so a dry run there finds nothing to do. On a copy given one x/ placeholder row and a
link to it in page.peb, the step refuses (template page.peb links to x/probe.html); before this fix it exited 0.

On the real database, a separate link checker that shares no code with the step found no remaining row linking to
any of the 6,368 deleted paths.

PDF rendering, in headless Chrome 140 on loopback only: all 15 stored PDFs, plus synthetic form, JPX and CCITT PDFs,
were opened in the stored pdf.js viewer before and after (D05 alone, and all five). Page counts, text and rendered text
layers matched on every page; nothing requested a deleted file; no new 404 or console message appeared. The form PDF
loaded the sandbox before the change and not after, and its field still showed its value. The JPX and CCITT PDFs
used openjpeg.wasm and jbig2.wasm, never a fallback. Chrome 140 lacks Map.prototype.getOrInsertComputed and
Math.sumPrecise, which pdf.js 6.1.200 calls, so the check defined both on both sides; ADFA-6636 tracks that.

Tests

48 tests (scripts/test_drop_unread_rows.py, scripts/test_clear_repeated_tooltip_detail.py), runnable under pytest
or as plain scripts. Each guard in the two steps was broken once, in a copy of the scripts: 92 mutants (49 for the
first version, 26 for the review fixes, 17 for the template decoding), each failing at least one test.

Pipeline position

• clear_repeated_tooltip_detail.py runs after every step that writes tooltip text and before ADFA-3305: Migrate the tooltip tables to keyed tables (documentation schema 3.0.0) #45's
  migrate_tooltip_schema.py, which moves detail out of Tooltips; it refuses a migrated database.
• drop_unread_rows.py runs after every step that writes page content, links or tooltips. It reads both tooltip
  schemas and decodes version 3's compressed templates, so it may run on either side of the migration.
• Both run before rebuild_database.py. Both back the database up, write in one transaction, VACUUM, support
  --dry-run, and change nothing on a second run.

Not covered

• No test on a device or emulator, so WebAssembly in the app's WebView is untested.
• The 7 books of the current Bookshelf plugin (appdevforall/addons) need no CMap (checked with pdffonts); books a
  plugin installs on the phone later are not checked.
• CI (python-tests.yaml) does not run scripts/test_*.py, so these tests run only by hand.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

[davidschachterADFA]

Independent Claude review (head 87b8b62)

Six findings: two medium, four low. None of them deletes a row that something still reads.

ID	Severity	file:line	Finding
R46-01	medium	drop_unread_rows.py:456	Crashes on a schema-3 DB, which its stated position allows
R46-02	medium	clear_repeated_tooltip_detail.py:1	No PEP 723 block: the documented uv run dies, no brotli
R46-03	low	drop_unread_rows.py:353	Chinese-font rule reads /BaseFont; pdf.js reads /FontName
R46-04	low	drop_unread_rows.py:335	PDF streams it cannot decode are skipped, not refused
R46-05	low	drop_unread_rows.py:284	Links with '..' past the root are dropped; browsers clamp
R46-06	low	test_drop_unread_rows.py	Mutants of the guards at :403, :421, :474 pass all 24 tests

R46-01. run() always reads TooltipButtons.uri and Tooltips.summary/detail. Schema 3 (#45's migrate_tooltip_schema.py) removes both: buttons get uriId/descriptionId and the text moves to TooltipTexts. The migration writes Tooltips, so both placement rules allow this step after it: the docstring's "after every step that writes page content or links" and the PR body's "...or Tooltips". On a migrated copy, drop_unread_rows.py --dry-run scanned for about a minute and then failed with sqlite3.OperationalError: no such column: uri at line 456; the database was unchanged. The tooltip step's docstring also says "after every step that writes Tooltips" while the PR body says it runs before the migration (it does refuse schema 3 cleanly).

R46-02. clear_repeated_tooltip_detail.py has no # /// script block, but it imports optimize_db_media, which imports brotli and PIL at module level. uv run scripts/clear_repeated_tooltip_detail.py --help fails with ModuleNotFoundError: No module named 'brotli'. The tests hide it because the test file declares both itself.

R46-03. The stored pdf.js 6.1.200 takes the font name from the FontDescriptor's /FontName when it differs from /BaseFont (pdf.worker.mjs:37103-37121), and that name decides the GBK-EUC-H fetch (:36486-36500). cmap_needs checks /BaseFont only: a non-embedded TrueType font with /BaseFont /SimSunX and /FontName /#CB#CE#CC#E5 returns []. A string /FontName is missed too. No stored PDF has this today.

R46-04. pdf_parts silently skips any stream zlib cannot inflate: an object stream filtered [/ASCII85Decode /FlateDecode], or every stream of an encrypted PDF. A crafted object stream holding /Encoding /90ms-RKSJ-H behind ASCII85 + Flate returns []. The docstring says any sign of a CMap need refuses the run, and an undecodable text row does refuse; here it neither sees the font nor refuses. No stored PDF or plugin book is encrypted; their undecoded streams are images.

R46-05. link_paths discards a resolved path that starts with ../. A browser resolves ../../x/a.html from i/clamp.html to /x/a.html, since .. stops at the root, and the file-name fallback does not fire when a kept row shares the name. With the fixture plus i/sub/a.html and that link, the run exits 0 and deletes x/a.html. 117 of the 1,238 x/ file names are shared with kept rows; no such link exists today.

R46-06. Three mutants pass all 24 tests: (a) line 403 without not viewer_changed prints "Nothing to do" and leaves scripting on when the sandbox rows are already gone; (b) line 421 treats an undecodable row to delete as fine; (c) line 474 treats an unreadable PDF as needing no CMap. The PR's "49 mutants, each failing at least one test" does not hold for these three.

Checked and sound

• A dry run leaves the file unchanged and writes no backup. A real run on a copy deletes exactly 6,368 rows (javadoc group: 4,672 class-use, 224 package-use, 60 module graphs); the only changed row is p/web/viewer.mjs, line 698 only. A second run of each step prints "Nothing to do".
• No kept row, template, tooltip or button refers to a deleted path, and no app or addons code builds these URLs. All 13,613 XML tooltip buttons point at a/.
• enableScripting is the only way the viewer loads pdf.sandbox.mjs; the no-WebAssembly fallbacks load only when WebAssembly fails.
• The 7 Bookshelf-plugin books in appdevforall/addons need no CMap, so the "Not covered" item about plugin books can be closed for the current plugin.
• The tooltip figures (1,605: 13 Java, 1,592 Kotlin) and "What users see" match ToolTipManager.
• Chunk rows, rollback, backup and VACUUM are handled correctly; all 33 tests pass and write nothing into the checkout.
• On a 112,283,648-byte copy of the input, the packed file shrank by 6,733,824 bytes and the .br by 5,764,408, in line with the PR's figures.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

[davidschachterADFA]

Response to the review

All six findings are fixed (87b8b62..6c33cac). Each new test failed on 87b8b62 for the reason it is named for.

ID	Fixed in	Change
R46-01	238a1d4	Drop step reads tooltip links in schema 2 or 3, refuses any other shape before the scan
R46-02	cf65e44	PEP 723 block on the tooltip step; uv run ... --help works
R46-03	dd4424f	Chinese-font rule reads /BaseFont and /FontName, as a name, literal or hex string
R46-04	20c7b37	Streams decoded through Flate, ASCII85, ASCIIHex, RunLength chains; unreadable PDFs refuse
R46-05	315366d	.. stops at the root when resolving a link
R46-06	6c33cac	Tests for the three surviving mutants

• R46-01: the tooltip step now says it runs before migrate_tooltip_schema.py and refuses a migrated database by name. On 87b8b62 the new tests crashed with no such column: uri.
• R46-04: image and cross-reference streams are skipped. An encrypted PDF, an unknown filter, a predictor or corrupt data refuses the run by name. endstream no longer reads as the start of a stream. All 6 new cases exited 0 on 87b8b62.
• Tests: 44 (33 + 11), passing under pytest and as plain scripts. Mutants: 75 of 75 killed (the original 49, two re-targeted where the fixes rewrote the code, plus 26 for the new guards).
• Real data: on schema 2 the output is identical to the 87b8b62 run (6,368 rows deleted, 1,605 details emptied). On schema 3 (tooltip step, migration, then the drop step) it now runs instead of crashing, deleting the same 6,368 rows. All 15 stored PDFs decode fully, and none needs a CMap.
• PR body: I corrected the test and mutant counts, the pipeline position and the safety checks, and closed the plugin-books item for the current Bookshelf plugin.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

[davidschachterADFA]

New commit e31aa00 answers R50-05 from the review on #50: from documentation version 3, the step decodes each template before reading its links. A template that doesn't decode refuses the run. Below 3, templates are read as plain text, as before.

• 4 new tests (48 in all). The two refusal tests fail on 6c33cac. The other two pass on it, as they should: one checks plain templates below 3, the other that a 3.0.0 database still loses the expected rows.
• 17 new mutants, all killed (92 in all).
• On the ship-time build, the link-like strings read from templates go from 0 to 29,210. A copy whose page.peb links to an x/ row refuses; before, it exited 0.

The body is updated. #51 is rebased onto this commit.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3