Repository navigation
[ops] broker — safely retry failed current-main workflow jobs #148
Description
Vision — automated project maintainer
Context
Main 44c80f91aea9e133c42f81df2ed83c3c53448faf remains incompletely validated after #147. CI run 37368971571 and Scorecard run 37368971799 each have jobs with no runner and no executed steps; their annotations say “The job was not acquired by Runner of type hosted even after multiple attempts”. Four executed Python 3.12/3.13 matrix jobs, audit, CodeQL, installed package smoke and product regression passed. The Python 3.14 jobs and Scorecard were cancelled, not passed. GitHub Status reports Actions operational on 2026-10-08. The existing pdctl API allowlist cannot rerun workflows. No source-code failure is established.
References: AGENTS.md, AGENT-EXECUTION-PIPELINE.md ownership amendments, ops/vision/README.md; https://docs.github.com/en/rest/actions/workflow-runs#re-run-failed-jobs-from-a-workflow-run ; https://www.githubstatus.com/api/v2/summary.json .
Goal
Prepare a narrowly validated, broker-only failed-job rerun operation so transient hosted-runner failures need not strand verified development.
Acceptance criteria
- A dedicated
pdctl rerun RUN_ID HEAD_SHA(or equivalently narrow command) checks the fixed repository, exact 40-hex head, completed failed/cancelled run, and current-main identity before requesting only GitHub's rerun-failed-jobs endpoint. - Refuse mismatched/stale SHAs, other repositories, successful or active runs, arbitrary endpoints/options and malformed identifiers; verify event/branch metadata and disallow unreviewed pull-request code.
- Preserve existing serialization/credential boundary. No arbitrary general API allowlist expansion, credentials returned, privileged installation, protection changes, skipping/weakening gates or direct host GitHub use.
- Add meaningful positive/negative mocked broker/client tests and exact locked canonical gates; record commands/exits and an isolated committed bundle.
- Document that owner review, independent prepublication/exact-head verification and root-owned installation remain required; repository merge alone does not update installed policy. No installation is authorized to workers.
Scope boundaries
In scope: smallest ops/vision client/control/tests/documentation change for retrying failed jobs of current-main runs. Vision explicitly authorizes this policy/operational proposal. Existing two-symbol feature delivery and #138 repair remain preserved and paused; do not change their code.
Out of scope: workflow edits, force-push/no-op commits to trigger CI, dispatching arbitrary refs, rerunning unreviewed PRs, releases, product code, dependency/corpus/scorer changes, credential or root-policy access.
Forbidden-territory reminder
Do not install or alter the live root-owned broker; this is only a reviewed repository proposal. Do not weaken tests or protection. Use existing injected/mocked test patterns; run contributed code only in pd-implementer.
Validation commands
uv sync --locked --dev; uv run --locked ruff check src/ tests/ benchmarks/ ops/ .github/scripts/; uv run --locked pyright src/ benchmarks/; uv run --locked pytest --tb=short -q; doctor if a valid worker index is available, otherwise disclose unavailable rather than claim pass. Include focused broker/client tests and git diff --check.
PR template
Refs this issue; exact base/head/tree; checked acceptance with evidence; canonical command exits and limitations; independent review pending; supervisor review: broker authorization surface.
Recovery
Stop after two failed repairs without a new hypothesis. A rejected/unavailable configured openai/gpt-6.1-sol worker route is a blocker, not permission to switch models. Return a committed bundle and JSON decision/handoff to /var/lib/python-docs/exchange/implementation. If unable to finish, checkpoint diagnosis without a passing claim.
Effort estimate
One 900-second isolated worker attempt. Root installation cannot be completed by this role.
Product decision and outcome review
User problem: a transient infrastructure failure prevents trustworthy post-merge validation and delays delivery. Baseline: two current-main failed runs, three unstarted cancelled jobs, no permitted retry. Target: safely retry only those failed current-main jobs without mutating source or reducing gates. Non-goals listed above. Review 2026-10-10; retain only if the narrow retry can be independently audited and improves recovery, revisit/remove if the broker exposes equivalent authorized functionality. No paid calls or product-quality claim.
Activity
ayhammouda commentedon Oct 8, 2026
Vision — automated project maintainer
The narrow recovery proposal is committed at
6745681dd418dc6a09f6184e9533ad35e4fc9269, based on current main44c80f91aea9e133c42f81df2ed83c3c53448faf(tree0ac3d6d15cf35e411f437329d14f897fbf03b681). I imported the verified bundle and statically reviewed all four changed files. No product/workflow/dependency/corpus/scorer or existing-test changes.It admits only an exact current-main failed/cancelled completed push run in this repository, checks both repositories and run metadata, rechecks main immediately before a bodyless failed-jobs-only retry, and preserves the broker lock and general API allowlist. Temporary authentication only: App mode explicitly refuses under its unchanged
actions:readpermission ceiling. A retry acceptance is never reported as passing CI.Fresh developer evidence: 86 focused broker/client tests, 666 full tests, locked lint/type checks and diff checks pass. An initial two-line lint failure was repaired and retained in evidence. Doctor passes on a symbol-only index, not a full documentation index; no product-regression claim follows. The owner ran no contributed code.
Not published, independently approved, installed or used for a live retry. Prepared decision and bundle are checkpointed for mandatory independent prepublication review at the start of the next full review window, followed by exact-head verification and hosted checks. Merge alone cannot update the root-owned broker; authorized operator installation remains separate. Main's two Python 3.14 jobs and Scorecard are still unvalidated, not declared fixed. Existing #138 and two-symbol deliveries remain preserved. Outcome review: 2026-10-10.
ayhammouda commentedon Oct 8, 2026
Vision — automated project maintainer
Mandatory independent prepublication review was attempted on the retained #148 proposal. The broker rejected it with
ValueError: Independent review rejected: incomplete check evidence(exit 1). The unpublished broker-created head is55724bb2538f92d624adc4c162cee6d3fe7afea3, based on44c80f91aea9e133c42f81df2ed83c3c53448faf, and its tree is exactly the reviewed developer tree0ac3d6d15cf35e411f437329d14f897fbf03b681. Verifier session:407c5b13-a617-45b5-aef2-e584f83b3e0d; recorded failure count: 1.No runnable branch/PR was published, no policy was installed, and no CI retry or merge occurred. The exposed broker result does not identify the missing/mismatched/nonzero command evidence, so I am not interpreting this as a reproduced product defect or retrying unchanged content. Exact developer bundle and decision remain preserved for evidence-based recovery.
Live current-main checks remain unchanged: both Python 3.14 jobs and Scorecard were cancelled before runner assignment; executed matrix jobs, product checks, audit and CodeQL pass. Those missing jobs are still not counted as passes. #138 preparation may continue in isolation, but its publication/merge remains gated.
ayhammouda commentedon Oct 10, 2026
Vision — automated project maintainer
Outcome review — 2026-10-10
Target not met; retain #148 as blocked, not delivered. On exact main
44c80f91aea9e133c42f81df2ed83c3c53448faf, CI run 37368971571 still has both Python 3.14 jobs cancelled with zero executed steps; Scorecard run 37368971799 likewise has zero executed steps. The four executed matrix jobs, Product quality, Security Audit and CodeQL passed. No subsequent current-main recovery run exists in the live run listing, and no new user feedback was received since the previous checkpoint.The proposal has not reached independent approval, authorized installation or a live failed-jobs retry, so its operational outcome cannot yet be evaluated. Preserve the existing bundle and rejection evidence; do not repeatedly submit the unchanged rejected tree. The next useful evidence is an authorized sanitized verdict for verifier session
407c5b13-a617-45b5-aef2-e584f83b3e0didentifying the exact head/base, approval/blockers, and commands with exit codes. The installed interface does not expose that diagnostic, and its generic rejection cannot establish which validation predicate failed.Revisit condition: resume after that diagnostic permits a concrete repair hypothesis, or after a new current-main run changes the recovery need. Success still requires independent exact-head review, authorized installation, a narrowly accepted retry and actual passing jobs on the current main SHA; acceptance of a retry alone is not success. If CI recovers before installation, reassess whether this additional broker mutation is worth retaining rather than expanding its scope. Next dated outcome review: 2026-10-12, or earlier on either evidence trigger. No checks, test thresholds, App permissions or credential boundaries change.
ayhammouda commentedon Oct 10, 2026
Implemented in #149 (merged commit77db2df0dab06dcda0a9d19e28ec76da897caeb1), independently verified and installed by the separately authorized trusted operator lane. The fixed-repository/current-main-only failed-job rerun has event/workflow/head restrictions, cooldown/cap and audit; worker credential/installation boundaries and required checks remain intact. Old-main CI37368971571 and Scorecard37368971799 were successfully recovered by authorized operator reruns. See #149 for exact-head verification, tests, deployment limitations and evidence. No package release is claimed.