Skip to content

fix(eval): avoid passing unknown alternate judge verdicts - #7518

Open
BlueRaddish wants to merge 1 commit into
google:mainfrom
BlueRaddish:fix/oct12-adk-1
Open

BlueRaddish wants to merge 1 commit into
google:mainfrom
BlueRaddish:fix/oct12-adk-1

Conversation

@BlueRaddish

Copy link
Copy Markdown
Contributor

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

No matching existing issue or competing open PR was found for this focused defect.

Problem: The alternate is_the_agent_response_invalid judge field currently maps any unrecognized value to VALID. A response such as unknown therefore receives a passing score of 1.0.

Solution: Parse only the supported true/invalid and false/valid values. Return the existing NOT_FOUND label for anything else, matching the primary judge-field fallback and leaving the existing empty-score behavior intact.

Testing Plan

The same new public-converter regression failed once on unchanged production code (1 failed, 4 passed). The complete final-response evaluator test file passed all 26 tests after the fix on Windows Python 3.13.14. The formatter-only test signature change preserves the AST.

Normal scoped repository hooks and the actual Git pre-commit hook passed without changing the final patch. No model endpoint was called.

Unit Tests:

  • I have added or updated unit tests for my change.
  • All four required full unit-test tox environments pass on hosted Ubuntu.

The required full tox matrix completed successfully for exact commit 785c21753656b2c949f6be46a72027881fbcc773 on four isolated standard Ubuntu runners. Each job used the unchanged repository tox.ini and tox -e py311, py312, py313, or py314, after installing declared extras. The actual import origin, source/configuration hashes and runner-generated untracked dependency lock were verified before/after testing. Standalone uv 0.12.23 was selected through the supported TOX_UV_PATH; no tests or dependencies were excluded.

Python Successful full-suite job Passed Skipped Xfailed Xpassed
3.11 job 18627 91 20 2
3.12 job 18618 92 20 2
3.13 job 18618 92 20 2
3.14 job 18618 92 20 2

The combined validation run also contains four failed jobs for an older, separate draft. All four job links above are successful results for this PR's exact commit. Warnings and normal test-defined skips/xfails/xpasses are retained in the public logs.

Manual End-to-End (E2E) Tests:

The regression exercises the existing public conversion/loading path with deterministic local data. Live model/provider end-to-end behavior was not tested.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have added tests that prove my fix is effective.
  • New and existing unit tests pass with my changes in the full supported tox matrix on hosted Ubuntu.
  • I have manually tested my changes end-to-end.

Additional context

OpenAI Codex assisted with diagnosis, implementation, regression verification and this description. This is a small shared-helper correction with no new dependency or API.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants