Repository navigation
fix(eval): avoid passing unknown alternate judge verdicts - #7518
Open
BlueRaddish wants to merge 1 commit into
Open
BlueRaddish wants to merge 1 commit into
BlueRaddish wants to merge 1 commit into
Conversation
Assisted-by: OpenAI Codex
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
No matching existing issue or competing open PR was found for this focused defect.
Problem: The alternate
is_the_agent_response_invalidjudge field currently maps any unrecognized value toVALID. A response such asunknowntherefore receives a passing score of 1.0.Solution: Parse only the supported true/invalid and false/valid values. Return the existing
NOT_FOUNDlabel for anything else, matching the primary judge-field fallback and leaving the existing empty-score behavior intact.Testing Plan
The same new public-converter regression failed once on unchanged production code (1 failed, 4 passed). The complete final-response evaluator test file passed all 26 tests after the fix on Windows Python 3.13.14. The formatter-only test signature change preserves the AST.
Normal scoped repository hooks and the actual Git pre-commit hook passed without changing the final patch. No model endpoint was called.
Unit Tests:
The required full
toxmatrix completed successfully for exact commit785c21753656b2c949f6be46a72027881fbcc773on four isolated standard Ubuntu runners. Each job used the unchanged repositorytox.iniandtox -e py311,py312,py313, orpy314, after installing declared extras. The actual import origin, source/configuration hashes and runner-generated untracked dependency lock were verified before/after testing. Standalone uv 0.12.23 was selected through the supportedTOX_UV_PATH; no tests or dependencies were excluded.The combined validation run also contains four failed jobs for an older, separate draft. All four job links above are successful results for this PR's exact commit. Warnings and normal test-defined skips/xfails/xpasses are retained in the public logs.
Manual End-to-End (E2E) Tests:
The regression exercises the existing public conversion/loading path with deterministic local data. Live model/provider end-to-end behavior was not tested.
Checklist
Additional context
OpenAI Codex assisted with diagnosis, implementation, regression verification and this description. This is a small shared-helper correction with no new dependency or API.