Retire the legacy target store on the Thesis ledger branch

[MaxGhenis]

Retires the legacy target store on codex/thesis-ledger-facts, and with it the step in the required "Arch checks" job that ran arch load all on every push and every resolver append PR.

Why

The "Build target input database" step ran arch --db /tmp/arch-targets-ci.db init, load all and stats. load all executes every db/etl_*.py loader, including the state SOI loaders. The audit in #348 (docs/us-legacy-etl-cleanup.md on that PR) found those loaders write invented values presented as IRS figures. This branch's db/etl_soi_state.py, db/etl_soi_deductions.py and db/etl_soi_income_sources.py are byte-identical to main's audited copies at a5280e1. Nothing on this branch read the tables those loaders filled, apart from the stack being removed here.

What the append path executes (task 1)

Checked by reading the code, the workflows and the Thesis resolver, not from the audit summary:

• Append gate / Trusted base append gate (.github/workflows/thesis-facts-append.yml): runs the BASE commit's scripts/check_thesis_facts_append.py (which loads scripts/receipt_pins.py) in the base's uv sync --locked --no-dev environment. The pytest step runs tests/test_policyengine_ledger.py, test_release_chain.py, test_thesis_append_adversarial.py, test_thesis_append_shim_isolation.py, test_receipt_shim_transparency.py and test_production_tsa_pins.py. These import arch.core (stdlib only), receipt and the scripts.
• Thesis resolver (ThesisInstitute/thesis scripts/resolve_pending.py, _regenerate_series_catalog and CATALOG_REGENERATION_INPUTS): fetches scripts/build_series_catalog.py, check_thesis_facts_append.py and receipt_pins.py plus the ledger/ catalog inputs from the base commit, and runs the generator with Thesis's own interpreter.
• chronicle.institute (scripts/refresh-data.sh): reads ledger/official_observations.jsonl, ledger/immutable_prefix.json and releases/ from this branch. It builds its bundle from main, not from this branch.
• The scripts import only the stdlib, receipt and each other. build_series_catalog.py imports check_thesis_facts_append lazily. The new test_append_path_scripts_load_nothing_from_the_repository_packages enforces this: it imports all six append-path scripts in an isolated interpreter and asserts that no arch, db, micro, calibration, packages, sqlmodel, sqlalchemy or supabase module loads.

What changes

Removed, each with its tests:

• all 19 db/etl_*.py (etl_targets_supabase.py plus 18 others, including UK etl_hmrc, etl_obr and etl_ons) and db/tests/;

• arch load, arch stats and arch query; the Stratum, StratumConstraint, Target, TargetType and GeographicLevel models; and query_strata, query_targets, insert_targets_batch and TARGETS_SCHEMA from db/supabase_client.py;

• arch/targets/, arch/normalization/targets.py (TargetBlueprint, as_target, target_kwargs) and scripts/export_to_json.py;

• calibration/ (it pulled rows from targets.db through get_targets; constraints, entropy and loader existed only to fit weights to those rows), plus the micro/us modules that read those rows or the hosted copy:

  • targets, pipeline, district, calibrate and hierarchy_validation;
  • validation_dashboard, whose only caller was hierarchy_validation;
  • hierarchical_pipeline, which reads hosted microplex.targets and falls back to mock targets.

  Main removed calibration/ and micro/ outright in Add Ledger governance and raw fact boundary checks #68.

• scripts/cps_to_policyengine.py --calibrate, which called the deleted micro.us.calibrate (and would otherwise have degraded to a printed warning).

Kept and decoupled:

• arch/jurisdictions/us/soi.py defines AGI_BRACKETS and TABLE_1_1_AGI_LABEL_TO_BRACKET itself. Both are identical in value and order to db/etl_soi.py, checked by AST evaluation of both files. Main's follow-on branch makes the same move.
• arch/jurisdictions/{us,uk}/__init__.py no longer import every loader. Before, the bundle reached all 17 loaders through arch.suite -> arch.jurisdictions.us.soi -> arch.jurisdictions.us -> arch.targets.loaders.
• arch/normalization/transforms.py holds scale_value, convert_units, apply_share and format_derivation, verbatim from the deleted module (checked segment by segment).
• db/cli.py keeps init, load-source-files and source-stats. db/schema.py keeps the source-artifact tables and the Jurisdiction / DataSource enums.

CI (.github/workflows/ci.yml, job arch, display name Arch checks, unchanged, so branch protection needs no edit):

• lint and pytest lists drop the deleted files and add tests/test_legacy_targets_retired.py;
• "Build target input database" becomes "Smoke-test the source-artifact database CLI" (init + source-stats), as main's follow-on does;
• the wheel smoke test imports soi, normalization and micro.us.entities, and asserts the retired modules are not in the wheel.

pyproject.toml changes only the hatch packages list (drops calibration). uv.lock is unchanged (uv lock --check passes). Nothing under ledger/ or releases/ and no gate-surface file changes.

Invariants

• Ledger, release and gate surfaces are untouched: git diff d7e45bc5...HEAD -- ledger releases scripts/check_thesis_facts_append.py scripts/verify_release_chain.py scripts/canonical_json.py scripts/cut_release_manifest.py scripts/receipt_pins.py scripts/build_series_catalog.py .github/workflows/thesis-facts-append.yml uv.lock is empty.
• Append-path import closure contains no repository package and no legacy-stack dependency (tested).
• No remaining file imports a retired module or name. The test walks the AST of every Python file under arch/, db/, micro/, packages/, scripts/, tests/ and data/.
• arch init creates exactly the four source-artifact tables, and load, stats and query exit 2 with "invalid choice" (tested).
• The consumer bundle is unchanged. arch build-bundle --year 2023 was built at d7e45bc and at da5a7eb, and all 745 files match: 548 byte for byte, including consumer_facts.jsonl (10,268 rows, sha256 78ca9a51…) and coverage.json. The rest are equal apart from the run's absolute paths, the arch_builds.created_at build timestamp, and the checksum and size fields in datapackage.json / ro-crate-metadata.json for files already proven equal. A comparison script holds those rules and keeps a per-file sha256 manifest (local evidence folder; the raw bundles were deleted for disk space).

These are exhaustive checks over finite sets (every module, every file, every bundle byte). The branch's dependencies don't include Hypothesis, and there is no input domain to sample.

Verification

• GitHub CI, required "Arch checks": green (run 38324513379) on a probe branch whose tree is identical to this PR's head 29c7ccd (tree 04746dca). The run covers lint, the pytest list, the source-artifact CLI smoke test, the wheel build and the wheel smoke test. The probe was needed because the base was red until Fold seven entity-drift lineages into their economy/aggregate siblings #344 merged.
• Locally, at da5a7eb + 08418cf:
  • the "Arch checks" pytest list plus all six append-gate test files: 283 passed, 1 skipped, 5 xfailed;
  • the whole surviving suite minus the slow bundle tests: 691 passed, 15 skipped, 5 xfailed. The one failure was the stale catalog pin (test_committed_catalog_is_current_and_valid), which Fold seven entity-drift lineages into their economy/aggregate siblings #344 fixed and which this branch now includes;
  • ruff passed on the CI lint list, uv lock --check passed, and the wheel builds with none of the retired files in it.
• Mutation check: adding import db.schema to scripts/canonical_json.py, and a lazy from db.etl_soi import AGI_BRACKETS to arch/core.py, makes the closure test and the import-walk test fail. Both mutations were reverted.
• Independent review (GPT-6.1 Sol): round 1 at da5a7eb requested changes on "targets" left in arch.__all__ (P2) and two docs that still described the deleted loaders as current (P3). Both are fixed in 08418cf, which also replaces a README coverage table that listed BLS and UK sources only those loaders supplied. Round 2 covers the delta.

Merge order and coordination

• Fold seven entity-drift lineages into their economy/aggregate siblings #344 merged at 65822f4 and fixed the stale stripped-segment pin that had kept the base's "Arch checks" red since 598f394. This branch merges that base (29c7ccd), and the merge was clean. The only overlap was pyproject.toml, where Fold seven entity-drift lineages into their economy/aggregate siblings #344 added Hypothesis dev-dependency lines and this PR changed the hatch packages line.
• The session that runs the Thesis side of this lane (Fold seven entity-drift lineages into their economy/aggregate siblings #344, Stage Chronicle's entity-fold seed in every catalog regeneration ThesisInstitute/thesis#342 and IRS SOI Table 1.4: add business, rental, royalty, farm-rental, partnership and S-corp net income/loss legs; give HT2/district 'net income (less loss)' columns their own concept ids #346) raised no objection. It confirmed that the resolver stages only CATALOG_REGENERATION_INPUTS, none of which this PR touches, and that none of its PRs edit ci.yml. Audit period strips by reviewed template, not an exact spelling list #309 needs a rebase onto 65822f4 because of Fold seven entity-drift lineages into their economy/aggregate siblings #344, not because of this PR. Neither Audit period strips by reviewed template, not an exact spelling list #309 nor Resolve catalog supersede links against every row's effective assertion id #306 nor Put every file that decides a gate verdict on the gate surface #308 has a hunk that overlaps this diff. No Thesis hub is registered.
• The resolver merges its append PRs once "Append gate" passes. It doesn't wait for "Arch checks", which is how Record 2 first-print observation(s) via resolve_pending.py #327 merged while that check was red. After this PR, those append PRs no longer run the legacy loaders either way.

axiom: n/a: infra and dead-code removal on the Thesis ledger branch; no policy logic changes.

🤖 Generated with Claude Code