ADFA-3305: Migrate the tooltip tables to keyed tables (documentation schema 3.0.0)

[davidschachterADFA]

ADFA-3305

Adds scripts/migrate_tooltip_schema.py, which converts a documentation.db's tooltip tables, in one transaction,
to keyed tables (documentation schema 3.0.0). The file shrinks by 5.2 MB and a tooltip's buttons become a key
lookup instead of a scan. The app side is appdevforall/CodeOnTheGo#2163.

Schema 3.0.0 will also mean every template is stored Brotli-compressed (ADFA-3305 decision, one switch for both).
That part comes in a PR stacked on this one, which has to merge with it.

Schema

• Tooltips(id, categoryId, tag): WITHOUT ROWID, PRIMARY KEY (categoryId, tag), the key the app looks a tooltip
  up by. The 613-page UNIQUE (categoryId, tag) index, a second copy of every tag, goes.
• TooltipTexts(id INTEGER PRIMARY KEY AUTOINCREMENT, summary, detail): keeps every id and the old sequence.
• TooltipButtons(tooltipId, buttonNumberId, descriptionId, uriId): WITHOUT ROWID, PRIMARY KEY (tooltipId, buttonNumberId), so a tooltip's buttons are one key range, already in display order.
• TooltipButtonDescriptions (711 rows, most used first) and TooltipButtonUris (20,453, in text order) hold each
  string once. "View full documentation" was stored 43,703 times. Neither has a UNIQUE index, which would cost
  about as much as the table.
• Tooltips.id and TooltipButtons.tooltipId are declared foreign keys to TooltipTexts(id).
• DocumentationDatabaseVersion becomes one row, 3.0.0.

The text is not in the WITHOUT ROWID table on purpose. Measured that way the rebuilt file was 1.18 MB larger:
such a table is an index B-tree, whose interior pages hold whole rows (340 pages), and its 264 rows over 1,002
bytes spill into mostly empty overflow pages. Its .br was 0.14 MB smaller, which is at noise level.

Sizes and timings

Base: the 2026-10-18 database after ADFA-6627's first build (#44 at a3c444e), packed as the pipeline packs it. .br is brotli -q 11 -w 24.

Copy	File	.br	File saved	.br saved
Base	110,895,104	86,918,141
Base repacked, no other change (control)	110,895,104	87,091,418	0	-173,277
This step	106,254,336	86,658,719	4,640,768	259,422
This step, then --pack Content,TooltipTexts	105,697,280	86,693,653	5,197,824	224,488

The file saving is real. The .br saving is at noise level: repacking alone moved it by 0.17 MB.

Desktop SQLite 3.46.1, warm, one pinned core, 2,000 keys: a tooltip's buttons took 0.0033 ms instead of 1.36 ms
median (a scan and sort of all 54,799 rows); the tooltip lookup itself is unchanged, 0.0056 ms against 0.0057.
SQLite 3.46.1, 3.44.4, 3.31.1 and 3.22.0 (Android 9's version, desktop build) plan the app's tooltip and button
queries as key searches only.

The step

• Backs up first, VACUUMs after, has --dry-run (converts and checks, then rolls back), and does nothing to a
  converted database declaring 3 or later.
• Before the commit, reads every tooltip and its buttons back through the new tables twice: by key, and with the
  app's own two queries by (category name, tag). It also compares both tables row for row, and their row counts.
  Any difference rolls everything back.
• Tooltips.id must be unique and name a TooltipTexts row; writers take it from the TooltipTexts insert. No
  UNIQUE (id) index enforces it (it would cost about the 613 pages saved), so the step checks it.
• Refuses, with exit 1 and the file untouched: a database not declaring major 2; a converted database declaring less
  than 3, or no version; tooltip tables in neither shape;
  data the new keys cannot hold (NULLs, a button with no tooltip, two buttons with one number, duplicate keys or
  ids); indexes, triggers or views on these tables. PR ADFA-6597: Index TooltipButtons by (tooltipId, buttonNumberId) #39's TooltipButtonsByTooltip is dropped instead, since the
  new key is the same index.

Pipeline and release order

• Where it runs: on the copy that ships, after every step that writes tooltips (TooltipManager.py, docdb-studio,
  sync_kdoc_json_to_db.py and the one-off scripts all write the old shape), and never on the master database they
  edit: check-tools, which docdb-regression-test.yaml runs daily at 02:00 UTC, selects the old columns, and nothing
  converts back.
• Then: rebuild_database.py --pack Content,TooltipTexts (ADFA-3305: Convert animated GIFs to VP9 WebM and pack documentation.db #38). --pack Tooltips errors, because a
  WITHOUT ROWID table has no rowids. That rebuild's comparison passed.
• How the database reaches users: CodeOnTheGo's compress_docdb.yml runs daily at 11:00 UTC. It downloads
  documentation.db from Google Drive (DOCUMENTATION_DB_FILE_ID) and, when its md5 has changed, uploads
  documentation.db.br to Drive. release.yml runs daily at 12:00 UTC: it merges stage into main, downloads
  that documentation.db.br into assets/release/common/database/, and builds the release APK from main, which
  bundles it. generate_assets.yml, run by hand, bundles the same Drive file into the asset zips. Only local debug
  builds take dev-assets.
• Why app and database switch in the same build: the app from ADFA-3305: Read and write the keyed tooltip tables (documentation schema 3.0.0) CodeOnTheGo#2163 refuses tooltips from
  a database declaring less than 3, so with a 2.x file it shows no built-in tooltips. The current app reading a 3.0.0
  database loses its tooltips too, and from reading its code, a plugin install records the plugin as installed with
  no rows. So every daily build has to pair the new reader with a 3.0.0 file, or the current one with 2.x.
• Ways to switch (not yet chosen):
  • Same day: after one day's 12:00 UTC release and before the next day's 11:00 UTC compression, put the 3.0.0 file
    on Drive and merge #2163 into stage; the next 12:00 UTC build takes both. Putting the file there between 11:00
    and 12:00 UTC needs compress_docdb.yml run by hand. No off-cycle release may run in between.
  • Pause: disable release.yml's schedule until both are in place, then run compress_docdb.yml and release.yml
    by hand.
• Not checked: whether ODT's master (GOOGLE_DRIVE_FILE_ID, which build-kotlin-docs.yaml calls "the production
  documentation.db" and writes back to) is the same Drive file as CodeOnTheGo's DOCUMENTATION_DB_FILE_ID; both IDs
  are secrets. If it is, putting the 3.0.0 file there breaks ODT's tools on their next run, so the shipped file would
  need a Drive file of its own.

Plugins, in the app PR

A plugin's tooltip text goes into TooltipTexts first, whose AUTOINCREMENT id keys its Tooltips row, so ids are
never reused. Button text is looked up, or added, 500 strings per query. Removal deletes by category, then deletes
button text no button uses. The app refuses a database declaring less than 3.

Conflicts

• ADFA-6597: Index TooltipButtons by (tooltipId, buttonNumberId) #39: its index duplicates the new key. This step drops it, but ADFA-6597: Index TooltipButtons by (tooltipId, buttonNumberId) #39's --operation index would add it back to a
  converted database. ADFA-6597: Index TooltipButtons by (tooltipId, buttonNumberId) #39's test and buttons query assume the old columns.
• CLAUDE.md conflicts with ADFA-6597: Index TooltipButtons by (tooltipId, buttonNumberId) #39, ADFA-6627: Minify the documentation's text without changing what pages show #44, ADFA-3305: Drop documentation rows nothing reads, and tooltip details that repeat the summary #46 and ADFA-3305: Trim the Kotlin API pages' JSON to what layout.pebble reads #47; python-tests.yaml with ADFA-3305: Trim the Kotlin API pages' JSON to what layout.pebble reads #47 (both add a test file to the same
  pytest step). ADFA-3305: Convert animated GIFs to VP9 WebM and pack documentation.db #38 merges cleanly.
• ADFA-3305: Convert animated GIFs to VP9 WebM and pack documentation.db #38's usage line and --pack help say --pack Content,Tooltips, which exits 1 on a converted database; after this
  step it is Content,TooltipTexts.

Tests

scripts/test_migrate_tooltip_schema.py: 56 tests, passing under pytest (Python 3.11 and 3.14) and as a plain script;
python-tests.yaml now runs them. 49 mutants: 48 fail a test; the other (no explicit ROLLBACK on an exception) is
equivalent, since closing the connection discards the transaction. In the app PR, 18 of 20 mutants are
caught. The 2 that pass: taking the id from the Tooltips insert, which is equivalent there because
last_insert_rowid() still holds the TooltipTexts id; and dropping the 500-string chunking, which Robolectric's
SQLite does not catch because it binds over 33,000 variables.

Still to check on a device

• Done on 2026-10-18 (Pixel 6 Pro, with ADFA-3305: Read and write the keyed tooltip tables (documentation schema 3.0.0) CodeOnTheGo#2163): built-in tooltips in the ide and kotlin
  categories (text, button order, links), and a 2.x debug database refused with the log line and no crash.
• Still open: the xml and java categories; plugin install, uninstall and reinstall into a 3.0.0 database; query plans
  and the chunking on a device (checked on a desktop SQLite 3.22.0 only).

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

[davidschachterADFA]

Independent Claude review (head 97c0fd9)

On the real database the conversion is sound: every tooltip and every button comes back byte-identical and in order. No critical issue; the biggest risk is the rollout plan in the PR body, not the script.

ID	Sev	file:line	Finding
R45-01	high	PR body: Pipeline and release order	Stated order ships a build with no built-in tooltips
R45-02	medium	CLAUDE.md:172 (also :35, :149)	No rule keeps the 3.0 file off the master; tools break
R45-03	low	migrate_tooltip_schema.py:370	"Already converted" exits 0 on a 3.0 shape declaring 2.x
R45-04	low	test_migrate_tooltip_schema.py:412	9 of 40 guard mutants pass all 37 tests; claim overstated
R45-05	low	migrate_tooltip_schema.py:146,320	Read-back never runs the app's tooltip query
R45-06	low	migrate_tooltip_schema.py:99	Tooltips.id no longer unique; a shared id loses text
R45-07	low	#38 rebuild_database.py:14,514	#38 says --pack Content,Tooltips; conflict list incomplete
R45-08	low	python-tests.yaml:71 (on main)	The 37 tests are not in CI

R45-01. The body says to release the app first, then upload the database, and names only dev-assets. CodeOnTheGo's release.yml runs daily at 12:00 UTC, merges stage into main, and downloads documentation.db.br from Google Drive, made from Drive's documentation.db by compress_docdb.yml (daily, 11:00 UTC); the APK bundles that file. So the first daily build after #2163 merges pairs the new reader with the 2.x database: requireKeyedTooltips throws, ToolTipManager catches it, and no built-in tooltips show. Converting Drive first breaks the current app the same way. A working switch converts Drive's file and merges #2163 in the same window (between 11:00 and 12:00 UTC), or pauses the cron; the plan should name Drive.

R45-02. Nothing says the 3.0 output must stay off the database ODT's tools edit, and no script converts 3.0 back. On a converted copy, check-tools/main.py (run daily by docdb-regression-test.yaml) crashes with no such column: detail; build-kotlin-docs.yaml edits the production documentation.db on Drive, and its step 5 (sync_kdoc_json_to_db.py:289) selects TooltipButtons.uri; TooltipManager.py and docdb-studio write the old shape. If Drive's file is the one compress_docdb.yml ships, the 3.0 file has to go there (both file IDs are secrets, so this was not checked). CLAUDE.md's schema section has no 3.0.0 callout, and lines 149-151 still point tooltip work at the old tables.

R45-03. With a converted copy set back to version 2.0.0, the script printed "already in the new shape; nothing to do" and exited 0, though the app refuses that file. #29's populate_db.declare_database_version rewrites the version row to 2.0.0 unconditionally, which is one way to get there.

R45-04. Disabling one guard at a time (40 mutants), 9 passed all 37 tests: Tooltips' exact column check (only TooltipButtons' has a test, so an extra Tooltips column would be dropped silently); all four new-shape conditions (TooltipTexts' shape, the lookup tables, WITHOUT ROWID on each keyed table); and the row count and the two row digests, each caught only in combination. This contradicts the claim that disabling each of 24 guards failed a test named for it.

R45-05. read_back looks tooltips up with KEY_QUERY (by categoryId); APP_TOOLTIP_QUERY is never run, and a test only checks it equals the test file's copy, neither pinned to TooltipQuery.kt. Both match #2163's head (5cc8d2bc) today, and the app's lookup by category name for all 46,128 keys gave 0 mismatches, so this is about what the docstring and body claim.

R45-06. Tooltips.id was the primary key; now nothing stops two rows sharing one. In the new DDL, two Tooltips rows with id 7 (a built-in and a plugin key): running the app's delete for the plugin's category removed the built-in's text, and the app query then returned nothing. Uniqueness is checked only during the conversion. A UNIQUE(id) index would cost about the 613 pages this saves, so at least document the rule for writers.

R45-07. #38's usage line and --pack help suggest --pack Content,Tooltips, which fails on a converted copy ("a WITHOUT ROWID table has no rowids to renumber", exit 1). By git merge-tree, CLAUDE.md also conflicts with #46 and #47, not only #39 and #44.

R45-08. The body says so. #47 adds its test to the pytest path list; this file needs only the standard library, so it is a one-line addition.

Checked and sound

• Real database (46,128 tooltips, 54,799 buttons): 2.1 s; 0 differences in tooltips, buttons in order, and the app's queries old against new, on raw bytes and value types. The 16 other tables, schema and pragmas unchanged; the sequence (93,791) carries over to TooltipTexts; integrity and foreign-key checks clean. A dry run leaves the file byte-identical.
• Plugin rows from stage's 2.x writer (0-based button numbers, empty uri, a tag listed twice, an uninstalled plugin): 0 differences; on the converted copy the new writer reinstalls with a new id (93,796), uninstalls cleanly, built-ins untouched.
• ADFA-3305: Convert animated GIFs to VP9 WebM and pack documentation.db #38 --pack Content,TooltipTexts passes its own comparison; --pack Tooltips is refused. ADFA-3305: Drop documentation rows nothing reads, and tooltip details that repeat the summary #46 gives identical results on 2.x and 3.0; clear_repeated_tooltip_detail refuses 3.0, matching this PR's order.
• Query plans on SQLite 3.22.0 (desktop library, same version as Android 9) and 3.31.1: the app's tooltip and button queries are key lookups with no sort; only the unused-text cleanup and the 500-string lookup scan, over 11 and 322 pages.
• Sizes match the PR to the byte (4,640,768; 5,197,824 after packing); the .br changes are noise, as the PR says. Buttons 2.56 ms to 0.0055 ms median; the tooltip lookup is unchanged (0.0089 vs 0.0090 ms), where the PR's ranges imply a small gain.
• The current app still reads Content from a 3.0 database. All 37 tests pass and write only to temp directories.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

[davidschachterADFA]

Response to the review

All eight findings are addressed (97c0fd9..ce12ffc). On real data the converted database is identical to 97c0fd9's (same content digest of every table, schema and sequence).

ID	Fixed in	Change
R45-01	PR body	Release section rewritten from compress_docdb.yml and release.yml: Drive, 11:00/12:00 UTC, two ways to switch
R45-02	390d0a0	CLAUDE.md: a Schema 3.0.0 callout; the step runs on the shipped copy, never on the master
R45-03	4559eac	A converted database declaring less than 3, or no version, is refused (exit 1, nothing written)
R45-04	c4add20	17 tests, each named for a guard the mutants showed untested
R45-05	15d8174	Read-back also runs the app's QUERY_TOOLTIP and QUERY_TOOLTIP_BUTTONS for every key
R45-06	390d0a0	The id rule documented in the docstring and CLAUDE.md, and still checked by the step
R45-07	PR body	Conflicts list completed; #38's --pack Content,Tooltips text noted (#38's own fix)
R45-08	36bdb20	python-tests.yaml runs the file; standard library only

• R45-03: refusing rather than declaring 3.0.0, as every other mismatch in the script is refused. Only a later step can have rewritten the version row, and the script can't tell what else that step changed.
• R45-05: the copied queries match TooltipQuery.kt at ADFA-3305: Read and write the keyed tooltip tables (documentation schema 3.0.0) CodeOnTheGo#2163's 5cc8d2bc9, and the comments now say so. On real data it costs 0.4 s more (2.44 s against 2.87 s).
• R45-01: your claims check out against CodeOnTheGo's origin/stage. compress_docdb.yml lives in CodeOnTheGo. The choice between the two ways to switch is open.
• Tests: 56, passing under pytest (Python 3.11 and 3.14) and as a plain script. 49 mutants: 48 fail a test, and the one survivor (no explicit ROLLBACK) is equivalent.
• Timing (ce12ffc): re-measured on one pinned core. The tooltip lookup is unchanged (0.0056 against 0.0057 ms), and buttons take 0.0033 ms instead of 1.36 ms. The docstring and body no longer imply a lookup gain.
• Also new in the body: schema 3.0.0 will also mean every template is stored compressed. That comes in a stacked PR that merges with this one.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CMZQuXT3GT7nEct2w4tg3

[davidschachterADFA]

Merge pair: schema 3.0.0 now also means compressed templates, which #50 adds to this migration; #50 merges with this PR, and #48 (re-minting templates) with both. The app side is appdevforall/CodeOnTheGo#2163 plus appdevforall/CodeOnTheGo#2164.

🤖 Generated with Claude Code