Repository navigation
Trim trailing page furniture from BoC press-release extraction - #137
Merged
Merged
Conversation
The HTML extractor grabbed the whole <main>, so cached rationale artifacts
carried trailing page furniture from modern (2021+) BoC announcement pages:
the content-type taxonomy line, related-content link teasers, and — on
2025-01-29 — footnotes plus a bundled same-day operational notice. This is
the text the reasoning-alignment evaluator reads as the "official rationale"
and a context seam for the LLMP/agent predictors, so the cruft mattered.
Truncate at the earliest line-anchored footer marker ({Footnotes,
Content Type(s)}): "Content Type(s)" is universal (drops taxonomy + teasers
on all 139 cached releases), "Footnotes" is the rare boundary that also
strips 2025-01-29's bundled notice. Both sit alone on their own line and
never appear in rationale prose, so the cut is safe and a near-no-op on
older pages. Every release now ends on its closing rationale sentence
(median 3844->3484 chars, 0 non-sentence-final endings across 139 files).
Add fixture tests for both the modern (footnotes + notice + taxonomy +
teasers) and taxonomy-only cases. Regenerate the cache with --force.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The BoC press-release HTML extractor grabbed the entire
<main>element, so cached rationale artifacts carried trailing page furniture from modern (2021+) announcement pages:Content Type(s): Press releases) — on all 139 cached releases;This is the text the reasoning-alignment evaluator (notebook 03) reads as the "official rationale," and it's the context seam available to the LLMP/agent predictors — so the cruft was participant-facing.
Fix
Truncate the extracted body at the earliest line-anchored footer marker,
{Footnotes, Content Type(s)}:Content Type(s)is the universal boundary — drops the taxonomy line and the teasers that follow it on every page.Footnotesis the rare boundary (1/139) that also strips the 2025-01-29 bundled notice, which precedes the taxonomy marker.Both markers sit alone on their own line and never occur in rationale prose, so the cut is safe and a near no-op on older (2009–2020) pages.
Verified on the regenerated cache (
scripts/fetch_boc_press_releases.py --force):Content Type(s)appears nowhere;Tests
Adds two fixture tests to
test_press_releases.py:Full BoC suite: 44 passed.
ruffclean.🤖 Generated with Claude Code