Repository navigation
Fix mismatched request_id in ITSM scenario 33 - #233
Merged
Merged
Conversation
seed_data/information_required used REQ-SW-066666 (naive employee-suffix pattern) while the ground-truth DB's requests key was the content-hash-based REQ-SW-659dac3f4833, so get_request_status could never resolve it. Align seed_data, information_required, negotiation_behavior, resolution_condition, and the golden expected_trace (including spoken-digit spellouts) to the actual DB key. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The corrected request_id changes what the golden expected_trace/DB require for this scenario, so it affects benchmark outputs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
fanny-riols
marked this pull request as ready for review
September 23, 2026 15:01
katstankiewicz
approved these changes
Sep 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
In the ITSM dataset (
data/itsm_dataset.json), scenarioid: "33"had a mismatchedrequest_id:user_goal.information_required.request_idandseed_data.request_idwere set toREQ-SW-066666ground_truth.expected_scenario_db.requestsonly contains a record keyedREQ-SW-659dac3f4833— same employee, same Salesforce request, different ID string.Fix
Updated scenario 33 so the
request_idis consistent everywhere:seed_data,information_required,negotiation_behavior,resolution_condition, theget_request_statustool call/response and its message inexpected_trace, and the two spoken-utterance digit spellouts (now spelling outsix five nine d a c three f four eight three three).EMP066666references were left untouched since that ID is correct and already matches the DB.Investigation scope
I checked whether this was an isolated authoring mistake or a systemic dataset bug:
information_requiredID-like fields (*_id,*_code) againstground_truth.expected_scenario_dbacross all three datasets:8.2's outdatedclearance_code, where the goal is for the agent to reject it), or destination values for a transfer that legitimately don't exist yet (new_facility_code,new_state_code).request_id. No other scenario in ITSM, HR, or Airline has this issue.Also, on all our past runs, the
get_request_statuscall never once succeeded for that scenario because it always used an id that wasn't present in the DB.