Skip to content

Fix mismatched request_id in ITSM scenario 33 - #233

Merged
fanny-riols merged 2 commits into
mainfrom
pr/fr/fix-itsm-scenario-33-request-id
Sep 23, 2026
Merged

fanny-riols merged 2 commits into
mainfrom
pr/fr/fix-itsm-scenario-33-request-id

Conversation

@fanny-riols

@fanny-riols fanny-riols commented Sep 23, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

In the ITSM dataset (data/itsm_dataset.json), scenario id: "33" had a mismatched request_id:

  • user_goal.information_required.request_id and seed_data.request_id were set to REQ-SW-066666
  • The scenario's own ground_truth.expected_scenario_db.requests only contains a record keyed REQ-SW-659dac3f4833 — same employee, same Salesforce request, different ID string.

Fix

Updated scenario 33 so the request_id is consistent everywhere: seed_data, information_required, negotiation_behavior, resolution_condition, the get_request_status tool call/response and its message in expected_trace, and the two spoken-utterance digit spellouts (now spelling out six five nine d a c three f four eight three three). EMP066666 references were left untouched since that ID is correct and already matches the DB.

Investigation scope

I checked whether this was an isolated authoring mistake or a systemic dataset bug:

  • Scanned all information_required ID-like fields (*_id, *_code) against ground_truth.expected_scenario_db across all three datasets:
    • ITSM (80 scenarios): only scenario 33 had a genuine mismatch.
    • HR (83 scenarios): 34 fields flagged, but all verified false positives — either deliberately wrong/outdated values the scenario is designed to test (e.g. scenario 8.2's outdated clearance_code, where the goal is for the agent to reject it), or destination values for a transfer that legitimately don't exist yet (new_facility_code, new_state_code).
    • Airline (50 scenarios): no matching ID fields at all — no mismatches.
  • Conclusion: this is an isolated authoring bug specific to scenario 33's request_id. No other scenario in ITSM, HR, or Airline has this issue.

Also, on all our past runs, the get_request_status call never once succeeded for that scenario because it always used an id that wasn't present in the DB.

fanny-riols and others added 2 commits September 23, 2026 10:49
seed_data/information_required used REQ-SW-066666 (naive employee-suffix
pattern) while the ground-truth DB's requests key was the content-hash-based
REQ-SW-659dac3f4833, so get_request_status could never resolve it. Align
seed_data, information_required, negotiation_behavior, resolution_condition,
and the golden expected_trace (including spoken-digit spellouts) to the
actual DB key.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The corrected request_id changes what the golden expected_trace/DB
require for this scenario, so it affects benchmark outputs.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@fanny-riols
fanny-riols marked this pull request as ready for review September 23, 2026 15:01
@fanny-riols
fanny-riols added this pull request to the merge queue Sep 23, 2026
Merged via the queue into main with commit 60b9e2e Sep 23, 2026
2 checks passed
@fanny-riols
fanny-riols deleted the pr/fr/fix-itsm-scenario-33-request-id branch September 23, 2026 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants