ADFA-3305: Merge duplicate media and drop unlinked rows

[davidschachterADFA]

ADFA-3305

Two more drop_unread_rows.py groups, from the item-B size analysis (G05 and G09): duplicate-media merges byte-identical media copies, pointing their links at the kept one, and unlinked deletes 31 rows nothing reads, from a closed list. On the 2026-10-19 item-B ship-time build they save 983,040 bytes of file and 259,386 of .br. Stacked on #51, whose step they extend.

duplicate-media also does item 2 of ADFA-6645 (the Help+Manual zoom images identical to their small ones), as #52 does its item 3.

duplicate-media

• Copies: rows of an image (other than SVG), video or audio type with the same type, language and stored bytes (chunks joined), that no other group deletes.
• The copy kept: the one the most rows link to (kept text rows, templates, tooltip buttons, Bookshelf entries), then the shortest path, then the lowest id.
• Links: every link to a dropped copy is pointed at the kept one, keeping its own directory prefix (relative, rooted or http://localhost:6174/) and its percent-encoding. That covers kept text rows, the JSON of templated rows (the Help+Manual toggles' href, src, data-src0 and data-src1), and tooltip button uris (TooltipButtonUris at schema 3, TooltipButtons.uri at 2). At schema 3, where each distinct uri is stored once, a button whose new uri already has a row is pointed at that row and the old row is deleted; otherwise the uri is rewritten in place.
• Refusals: templates are not rewritten, so a template link to a dropped copy refuses the run, as does any link the rewrite misses (the reference check reads the rewritten rows) and a templated row whose JSON would stop parsing.
• Not merged: PDFs, because the Bookshelf placeholder books share their bytes (the seven bookshelf/… PDFs and t/textbookPlaceholder.pdf) but are separate entries, and deleting one fires DeleteBook; text rows, since the only identical ones are the jQuery licence notices; and SVGs, because an SVG is a document whose relative references resolve against its own URL, so a copy kept in another directory could load other files. None of the shipped copies is an SVG.
• Real data: 53 copies in 52 sets, 881,962 stored bytes; 98 links in 13 rows pointed at the kept copies; no tooltip uri named a copy. 44 are Help+Manual _zoom25/_zoom10 images, 8 are a/devsite images, and i/images/closehelp.webp goes in favour of ZoomClose.webp, which two scripts name.

unlinked

• The list: UNLINKED names each of the 31 rows and why it is unread. The group deletes those and nothing else, and the reference check still refuses the run if any kept row links to one.

• How it was made: from the analysis's 113 candidates (its 41 pdf.js rows are item C and stay). A check that shares no code with the step read every text row, template, tooltip text and button uri of the item-B build. It kept a candidate out if anything not itself being deleted named it:

  • by file name (other than an absolute URL to another host);
  • by path without the extension;
  • by file name without the extension.

  It repeated that until nothing changed. None of the 31 is among the app's own documentation URLs (CodeOnTheGo stage, grepped).

• Why the stricter rules: nav.peb and page.peb build their links as /{{ node.id }}.html from page ids with no .html in the data. So 20 of the analysis's 21 Kotlin topic pages are linked from the sidebar after all, and the 21st from kotlin-brand-assets.html. Help+Manual's hm_webhelp.js loads its scripts and images by name. Its Embed strings are hmWebHelp.doFullEmbedWindow, not a load of hmEmbedHelp.js.

• What goes:

  • five Help+Manual embedded-popup files;
  • four Help+Manual skin files, including the 7-byte GoogleFonts.css stub ADFA-4480 left;
  • four javadoc doc-files images no page shows;
  • five javadoc files: the overview-summary.html redirect stub, jQuery 3.6.1 (the pages load 3.7.1), jdk-default.css, the JDWP spec page and java_keywords.html;
  • six Dokka disambiguation pages;
  • two Kotlin topic pages outside the sidebar;
  • four Dokka package-list files;
  • k/not-found-version.html.

• Left out: 65 candidates the check could not prove unread, among them the sidebar's Kotlin pages, the Help+Manual scripts and images loaded by name, and k/navigation.html. Also left out by policy:

  • the pages of WebServer's disabled playground endpoint;
  • the PUCC example data;
  • x.html and x.pdf, which ADFA-3305: Drop documentation rows nothing reads, and tooltip details that repeat the summary #46 keeps;
  • the t/ placeholder book;
  • the licence notices;
  • the two index pages.

Real data (the 2026-10-19 item-B ship-time build, 85,708,800 bytes)

Each result and the baseline packed with rebuild_database.py --pack Content,TooltipTexts --page-size 4096 (#38) and compressed with brotli -q 11 -w 24. The baseline, the same build repacked, is 85,708,800 bytes, .br 67,290,641.

Group	Rows	File saved	.br saved
duplicate-media	53	892,928	165,045
unlinked	31	94,208	86,465
Both	84	983,040	259,386

• The independent check (no code shared with the step) passes on the both-groups run:

  • only the 53 copies and 31 rows are gone;
  • exactly one copy of each set remains;
  • each of the 13 changed rows equals its original with the links to dropped copies renamed to the kept copy, and nothing else;
  • every renamed link resolves;
  • no kept row, template or tooltip names a deleted row;
  • every other table is unchanged.

  It catches a run without unlinked and a row with one link left broken.

• Re-runs: a dry run of all seven groups on the result reports nothing to do.

• The whole ship-time order with this step in place of ADFA-3305: Point javadoc's Index links at the split index; drop index-all.html #51's, from the 2026-10-18 database: 84,828,160 bytes, .br 67,163,662. The step found the same 53 copies and 31 rows there, early in the order, and refused nothing. The re-mint verified 22,720 of 22,720 items and 1,126 of 1,126 templates. Against the item-B build (85,708,800 and 67,133,725) that is 880,640 bytes less file but 29,937 more .br: the re-mint kept another seed (20822, not 20819), whose compressed rows store 27,790 bytes more, so the two full builds differ by more than these groups. The table above, on one dictionary, is the like-for-like measure.

Tests

70 tests (18 new), passing under pytest (Python 3.11 and 3.14) and as a plain script. The first 15 fail on #51's head, where neither group exists, so the mutants carry the proof for them:

• duplicate-media: 28 mutants, 26 killed. They cover:

  • the kept-copy order and what counts as a link;
  • same bytes in full, same type and same language;
  • the excluded types, and rows other groups delete;
  • the prefix, the encoding, the lookbehind and the lookahead;
  • the JSON rewrite and its check;
  • the uri rewrite, its check and its write.

  The 2 survivors are equivalent: each removes a prefilter that skips rows not naming a copy.

• unlinked: 4 mutants (selecting by prefix, by file name, nothing, or losing a path), 4 killed.

• The review fixes add 3 tests and strengthen 1:

  • byte-identical SVGs in two directories, each beside its own pic.webp, stay unmerged. This fails on 5b7f910, which merges them;
  • at schema 3, a button uri rewritten onto a string another row already holds is pointed at that row, and the old row goes. This fails on 5b7f910, which leaves two rows holding one string;
  • a uri whose new string isn't stored yet is still rewritten in place, keeping its id (passes on 5b7f910, and pins that path);
  • the other-groups test now also uses a WebP that unlinked deletes beside an identical WebP elsewhere. Its SVG case stopped reaching the guard once SVGs were excluded.

• Mutants after the fixes: the review's 25 (one adjusted to the changed line) and 10 of the fixes, 35 in all: 34 killed. The survivor, MAX(id) instead of MIN(id) in the uri lookup, is equivalent, since TooltipButtonUris holds each string once.

Real data, after the review fixes

On a copy of the item-B build, the fixed step deletes the same 84 rows and points the same 98 links in 13 rows, and the result is byte-identical to 5b7f910's (sha256 e3f59f64...). No shipped copy is an SVG, and no shipped button uri names a copy. A dry run on the result has nothing to do.

Pipeline position

Unchanged from #46: after the steps that write page content, links or tooltips, then rebuild_database.py --pack, which must follow. Both rewrites change row sizes, which loosens a packed file.

Not covered

• No device check. A Help+Manual toggle now swaps an image for itself, and the a/devsite pages show the kept copy of each identical image.
• The list was proven on the item-B build. On a database that later changes, the reference check is what refuses a row someone has since linked.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

[davidschachterADFA]

Independent Claude review (head 5b7f910)

On today's data the two groups do what the body says, and nothing they delete or repoint breaks a page, image, script or tooltip.

• The run: on a copy of the item-B build, the step deleted 84 rows (53 media copies, 31 unlinked rows) and rewrote 98 links in 13 rows; no tooltip URI changed.
• Independent checker (brotli CLI with the database's dictionary, its own link resolution, no code shared with the step):
  • Every other table is identical.
  • In each of the 13 rows, only path tokens changed. Each old token resolves to a deleted copy, each new one to a stored row with the same bytes, and every templated row is still valid JSON.
  • No kept row, template, tooltip text or button URI names a deleted row by file name, encoded name or stem.
• Chrome: I rendered all 13 pages from servers for the before and after databases, with the Help+Manual pages rendered through the ODT Pebble harness. Screenshots are pixel-identical. Requests for deleted paths go from 52 before to 0 after, and there are no new 404s.
• The body's figures reproduce to the byte:
  • both groups 983,040 file and 259,386 .br;
  • duplicate-media alone 892,928 and 165,045;
  • unlinked alone 94,208 and 86,465.
• Mutants: 25 of my own, all killed.

The two findings are latent: a type the merge rule admits but shouldn't (SVG), and a lookup-table invariant at schema 3.

ID	Sev	file:line	Finding
RG-01	low	step:245, :781-787	SVG counts as media: copies in different directories merge, changing what their relative references load
RG-02	nit	step:1023-1026	A schema-3 URI rewrite can leave two TooltipButtonUris rows with one string

step is scripts/drop_unread_rows.py.

RG-01. DUPLICATE_MEDIA_PREFIXES is ("image/", "video/", "audio/"), so image/svg+xml rows are merged like pixels. An SVG is a document: shown through <object>, as i/prefs-editor.html shows the i/ SVGs, or opened directly, its relative href, xlink:href and url() resolve against its own URL.

Probe, through the test file's own fixtures:

• two byte-identical SVGs, a/media/logo.svg and k/art/logo.svg, each holding <image xlink:href="pic.webp">;
• a different pic.webp in each directory;
• a/page.html and k/page.html each linking their own SVG.

--only duplicate-media exits 0, deletes a/media/logo.svg, and rewrites a/page.html to media/../../k/art/logo.svg. Opened as a document, the Android page's SVG now draws the Kotlin picture. Nothing refuses: the dropped copy's own references are never read, and the kept copy's resolve.

None of the 53 shipped copies is an SVG, and no shipped SVG has a relative reference (all 196 checked), so nothing changes today. A later build with identical SVGs in two directories would ship this silently.

Fix: take image/svg+xml out of the merge, or merge an SVG only within one directory or when it makes no relative reference, with a test like the probe.

RG-02. At schema 3, a rewritten URI is changed in place: UPDATE TooltipButtonUris SET uri = ? WHERE uri = ?. If the kept copy's URI already has its own row, two rows end up holding the same string.

Probe, schema 3, buttons on a/devsite/media/arch.webp (id 1) and a/devsite/media/arch-overview.webp (id 2): after the run, ids 1 and 2 both hold a/devsite/media/arch-overview.webp.

The lookup tables are meant to store each distinct string once. They have no UNIQUE index, and writers look a string up before inserting it (docs/documentation-database.md; PluginTooltipTables.kt). It's harmless to the app, which reads by id. No shipped button URI names a copy.

Fix: point the buttons at the existing row's id and delete the old row, or update only when the new string isn't already there, with a schema-3 test.

Checked and sound

• Real run (copy of /home/david/adfa-e2e-b/final.db, sha256 4c2005de... unchanged afterwards):
  • The dry run left the file byte-identical and reported what the real run did: 53 copies (881,962 stored bytes), 31 unlinked rows (86,348), 98 links in 13 rows, 0 tooltip URIs.
  • The copies: 44 _zoom25/_zoom10 Help+Manual images, 8 a/devsite/media images, and closehelp.webp going in favour of ZoomClose.webp. All sets are in one directory.
  • No row was added. No row's id, type, template or language changed.
  • Every other table (Templates, the tooltip tables, Bookshelf, the dictionary) is identical. All 7 Bookshelf entries still name a stored row.
  • Nothing under p/ and no PDF was deleted, so pdf.js (item C) and the DeleteBook trigger are untouched.
• Repointed links. Each of the 13 rows was compared token by token, before against after:
  • Text outside path tokens is identical.
  • Every changed token resolved from the row's own directory (or the page's, for templated JSON) to a deleted copy, and its new form resolves to a stored row with the same bytes.
  • The 8 templated Help+Manual rows still parse.
  • Prefixes are kept as written: relative (./images/), rooted, and http://localhost:6174/.
  • The lookahead keeps a following ?query or #fragment outside the match.
• Nothing still names a deleted row. I searched every decoded text row, template, tooltip text and button URI of the result for each deleted row's file name, its percent-encoded form and its stem. The only hits:
  • alt="..._zoom25" attribute text on 10 toggles;
  • pdf.js's --page-bg-color CSS variable.
• Loads by name:
  • hmWebHelp.extFuncs loads ./js/<func>.js, and every literal name it's called with is a kept script.
  • Popups load ./js/hmSmartPopup.js, which is kept.
  • _hmXpopup, _hmXtopic, hmEmbedHelp, hmEmbeddedPopups and hm_popup appear nowhere.
  • The toggle script reads data-src0 and data-src1 from the page data. No template or script builds a _zoom name.
  • nav.peb builds /{{ node.id }}.html, and page.peb builds /{{ prev.id }}.html and /{{ next.id }}.html. No id names an unlinked Kotlin page.
  • The javadoc pages load jquery-3.7.1.min.js; nothing names 3.6.1, overview-summary, jdk-default.css, jdwp-protocol or java_keywords.
  • GoogleFonts.css is 7 bytes.
• Outside the database:
  • None of the 84 deleted file names appears in CodeOnTheGo origin/stage (git grep over .kt, .java, .xml, .json, .html, .js and .md files). The only hit was a Kotlin import kotlin.coroutines.cancellation.CancellationException.
  • None appears in the local addons checkout (c109270). Its one link into the core docs, i/layout-top.html, is kept.
• Chrome (headless 140, as compare_svg_renders.py isolates it: dead proxy on 127.0.0.1:9, every name but 127.0.0.1 unresolvable, pages served on loopback from each database):
  • All 13 pages are pixel-identical before and after.
  • The before pages requested 52 now-deleted paths; the after pages requested none.
  • The only 404, i/images/banner_company.jpg, happens on both, so it predates this branch.
  • NetLog: the only non-loopback sockets were Chromium's IPv6 probe (UDP connect to [2001:4860:4860::8888]:443, local-address lookup, no send event).
• Sizes, both copies packed with ADFA-3305: Convert animated GIFs to VP9 WebM and pack documentation.db #38's rebuild_database.py --pack Content,TooltipTexts --page-size 4096 and compressed with brotli -q 11 -w 24:
  • baseline (the same build repacked): 85,708,800 / 67,290,641;
  • both groups: 84,725,760 / 67,031,255;
  • each group alone matches the body's table to the byte.
  • Unpacked, the result is 84,779,008: the --pack the docs require costs little to skip here, but it still applies.
• Exclusions: the body is right that the only byte-identical non-media rows are 8 PDFs (the seven Bookshelf books and t/textbookPlaceholder.pdf) and two pairs of jQuery licence notices.
• Re-runs: a dry run of all seven groups on the result reports "Nothing to do" and leaves the file unchanged. In the author's full ship-time run (e2e-drop.log) the step ran before the 3.0.0 migration, on a 2.x database, and found the same 53 copies and 31 rows. It touched 14 rows there, against 13 on the item-B build; the body doesn't claim a row count for that run.
• Tests: the 67 tests pass. 25 mutants of mine, all killed, covered:
  • the relative path to the kept copy, the encoded name and the encoding kept;
  • the kept-copy order and every source it counts;
  • type, language and the excluded types;
  • rows other groups delete;
  • the JSON check;
  • the URI table in both schemas, URIs not written, and URIs checked as stored;
  • the lookbehind and the lookahead;
  • templated JSON not rewritten;
  • an unlinked entry dropped, and selection by file name.
• Not checked:
  • a device;
  • clicking the Help+Manual toggles in Chrome (their data-src1 targets were checked by resolution and bytes instead);
  • how the 113 analysis candidates were reduced to 31;
  • the full-order run's re-mint figures (22,720 items, seed 20822).

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

[davidschachterADFA]

Response to the review

Both findings are fixed (5b7f910..84a3126).

ID	Fixed in	Change
RG-01	f6306d4, 84a3126	SVGs are left out of duplicate-media entirely; the other-groups test now reaches the guard
RG-02	627c1e2	At schema 3, a rewritten button uri already stored is pointed at that row; the old row goes

• RG-01: image/svg+xml is now excluded (DUPLICATE_MEDIA_EXCLUDED), rather than merged within one directory only. That's simpler, and none of the 53 shipped copies is an SVG.
  • The new test is your probe: two identical SVGs, a/media/logo.svg and k/art/logo.svg, each beside its own pic.webp. It fails on 5b7f910 because the SVG is merged.
  • A side effect I fixed: test_copies_ignore_rows_other_groups_delete proved the "rows other groups delete are not copies" guard with an SVG, the module graph. Once SVGs were excluded, that case no longer reached the guard, and the mutant that drops path not in doomed survived. The test now adds a WebP that unlinked deletes, beside an identical WebP elsewhere; without the guard, the second one is deleted too (84a3126).
• RG-02: for each rewritten uri at schema 3, the step looks the new string up in TooltipButtonUris.
  • If a row holds it, the buttons on the old row are pointed at that row and the old row is deleted. Otherwise the string is rewritten in place, as before. Schema 2's TooltipButtons.uri is unchanged.
  • The reuse test is your probe (buttons on arch.webp and arch-overview.webp). It fails on 5b7f910 with ids 1 and 2 both holding arch-overview.webp.
  • The in-place test checks that a uri whose new string isn't stored yet keeps its id. It passes on 5b7f910 and pins that path.
• Mutants: your 25, with "rows other groups delete are copies too" adjusted to the changed line, plus 10 of the fixes: 35 in all, 34 killed.
  • The fixes' mutants: SVG exclusion emptied, exclusion not checked, all images excluded, no uri lookup, the lookup at schema 2 too, looking up the old string, buttons not repointed, old row not deleted, rewritten in place even when stored.
  • The survivor, MAX(id) instead of MIN(id) in the lookup, is equivalent: the table holds each string once.
• Real data: on a copy of the item-B build, the fixed step deletes the same 84 rows and points the same 98 links in 13 rows. The result is byte-identical to 5b7f910's (sha256 e3f59f64...), with the same log apart from the backup name. A dry run on the result has nothing to do, and final.db's sha256 is unchanged.
• Tests: 70 (18 new), passing under pytest on Python 3.11 and 3.14, and as a plain script on both.
• Base: 93e9f96 merges the updated ADFA-3305: Point javadoc's Index links at the split index; drop index-all.html #51 (which took main's ADFA-3305: Re-encode the images inside SVG rows as WebP #52 through ADFA-3305: Drop documentation rows nothing reads, and tooltip details that repeat the summary #46). Only CLAUDE.md conflicted, and both entries are kept.
• Body: corrected to match.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3