Skip to content

Trim trailing page furniture from BoC press-release extraction - #137

Merged
ethancjackson merged 1 commit into
mainfrom
boc-report-extraction
Jun 23, 2026
Merged

ethancjackson merged 1 commit into
mainfrom
boc-report-extraction

Conversation

@ethancjackson

Copy link
Copy Markdown
Collaborator

What

The BoC press-release HTML extractor grabbed the entire <main> element, so cached rationale artifacts carried trailing page furniture from modern (2021+) announcement pages:

  • the content-type taxonomy line (Content Type(s): Press releases) — on all 139 cached releases;
  • related-content link teasers appended after it;
  • on 2025-01-29 specifically, a block of footnotes plus a bundled same-day operational notice (the deposit-rate/term-repo notice), which is why that file was an outlier at ~9,964 chars.

This is the text the reasoning-alignment evaluator (notebook 03) reads as the "official rationale," and it's the context seam available to the LLMP/agent predictors — so the cruft was participant-facing.

Fix

Truncate the extracted body at the earliest line-anchored footer marker, {Footnotes, Content Type(s)}:

  • Content Type(s) is the universal boundary — drops the taxonomy line and the teasers that follow it on every page.
  • Footnotes is the rare boundary (1/139) that also strips the 2025-01-29 bundled notice, which precedes the taxonomy marker.

Both markers sit alone on their own line and never occur in rationale prose, so the cut is safe and a near no-op on older (2009–2020) pages.

Verified on the regenerated cache (scripts/fetch_boc_press_releases.py --force):

  • no footer marker remains in any of the 139 releases; Content Type(s) appears nowhere;
  • 0 non-sentence-final endings (every release now closes on its rationale sentence);
  • median 3844 → 3484 chars, max 9964 → 5590 (2025-01-29: 9964 → 4590).

Tests

Adds two fixture tests to test_press_releases.py:

  • modern page (footnotes + bundled notice + taxonomy + teasers) → trimmed to the rationale;
  • taxonomy-only page (older releases, no footnotes) → exercises the universal cut independently.

Full BoC suite: 44 passed. ruff clean.

Note: the report cache under data/reports/boc_press_releases/ is gitignored; reviewers regenerate it with uv run python scripts/fetch_boc_press_releases.py --force (HTTP only, no LLM/proxy calls).

🤖 Generated with Claude Code

The HTML extractor grabbed the whole <main>, so cached rationale artifacts
carried trailing page furniture from modern (2021+) BoC announcement pages:
the content-type taxonomy line, related-content link teasers, and — on
2025-01-29 — footnotes plus a bundled same-day operational notice. This is
the text the reasoning-alignment evaluator reads as the "official rationale"
and a context seam for the LLMP/agent predictors, so the cruft mattered.

Truncate at the earliest line-anchored footer marker ({Footnotes,
Content Type(s)}): "Content Type(s)" is universal (drops taxonomy + teasers
on all 139 cached releases), "Footnotes" is the rare boundary that also
strips 2025-01-29's bundled notice. Both sit alone on their own line and
never appear in rationale prose, so the cut is safe and a near-no-op on
older pages. Every release now ends on its closing rationale sentence
(median 3844->3484 chars, 0 non-sentence-final endings across 139 files).

Add fixture tests for both the modern (footnotes + notice + taxonomy +
teasers) and taxonomy-only cases. Regenerate the cache with --force.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@ethancjackson
ethancjackson merged commit 3f66f70 into main Jun 23, 2026
2 checks passed
@ethancjackson
ethancjackson deleted the boc-report-extraction branch June 23, 2026 12:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant