Skip to content

[rush reporter][R8] Qualify AI reporter auto-selection #5981

Open
Open
[rush reporter][R8] Qualify AI reporter auto-selection#5981
Labels
enhancementThe issue is asking for a new feature or design change

Description

@TheLarkInn
Sean Larkin (TheLarkInn)
opened on Aug 28, 2026

Parent: #5858

Blocked by: #5977, #5978

Goal

Prove that the AI reporter improves agent outcomes before enabling environment-based automatic selection.

Scope

  • Build a deterministic corpus of representative install, configuration, dependency, build, cache, network/auth, and plugin failures.
  • Compare legacy/plaintext/AI output for remediation success, context consumption, determinism, secret handling, and full-log discoverability.
  • Validate built-in COPILOT_CLI detection plus configured agentEnvironmentVariables.
  • Define pass/fail thresholds and store machine-readable gate results.
  • Keep auto-selection disabled until the corpus gates pass.

Acceptance criteria

  • The corpus is deterministic and runnable in CI.
  • AI output never includes classified secrets and does not regress producer privacy.
  • Thresholds for remediation quality, output size, determinism, and log-path correctness are documented and enforced.
  • Passing the gate enables agent auto-selection only on the opt-in reporter path before Rush 6.
  • RUSH_REPORTER=legacy remains authoritative.

DAG

Unblocks R9.

Part of #5858.

Activity

  1. Sean Larkin (TheLarkInn)
    added
    enhancementThe issue is asking for a new feature or design change
    on Aug 28, 2026
  2. Sean Larkin (TheLarkInn)
    added a parent issue on Aug 28, 2026
  3. github-project-automation
    added this to Bug Triageon Aug 28, 2026
  4. github-project-automation
    moved this to Needs triage in Bug Triageon Aug 28, 2026

  5. TheLarkInn commented on Aug 28, 2026

    @TheLarkInn
    Sean Larkin (TheLarkInn)
    on Aug 28, 2026
    MemberAuthor

    R8 qualification integration is reviewed and fixed in #5999 at ac3fd75, but this issue remains blocked.

    The corpus now measures actual reporter output and AI log discoverability, validates exact context values/privacy markers, uses scenario-specific deterministic evidence with aggregate/per-case/compact size gates, preserves honest secret-diagnostic counts, and explicitly requires the separate privacy prerequisite before future AI eligibility. Three standalone runs were byte-identical; all blocking corpus gates passed locally, and reporter/rush-lib/apps-rush validation passed.

    Remaining gates: #5990 must merge and be accepted; stacked base #5997 must merge; #5999 must receive required CI/review and merge with auto-merge still disabled while stacked; the qualification must pass on main; only then may approved opt-in activation consume both prerequisites. RUSH_REPORTER=legacy remains authoritative and no Rush 6/default activation was introduced.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementThe issue is asking for a new feature or design change

    Type

    No type

    Projects

    • Bug Triage
      Status
      Needs triage

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions