[u9g]

Adds to SimulationRun what agent-service already knows about each simulation job but couldn't return through the API. Today a client such as lk agent simulate export or a CI check sees only that a job failed and its timings and scores. It can't see why the job failed, how the call ended, or what was said.

SimulationRun.Job.failure_reason (field 19)

A machine-readable cause for a failed job, with a FailureReason enum. Its one cause so far is FAILURE_REASON_AGENT_JOIN_TIMEOUT, meaning the agent under test never joined. agent-service already stores it on every failed job and gives it to the summarizer. Without the field, a client has to match on the human-readable error. The enum numbers match agent-service's internal jobs.FailureReason, so the service copies the stored value straight across. It's field 19 because #1862 claims 17 and 18 on the same message.

SimulationRun.JobMetrics: how the call ran

These are facts the simulator already knows, which until now only reached its logs:

Field	Meaning
end_reason	How the call ended: the caller finished the scenario, the agent ended it, the agent went silent, the turn limit was hit, empty replies, or an agent error. A failing end still also sets the job's error.
caller_language	The simulated caller's language, as a language tag.
caller_listen_language	Audio only: the language the caller transcribed the agent in. "" means the same as caller_language.
impairments	Audio only: whether background noise, low-quality microphone, or packet loss was applied, as a top-level SimulationImpairments with the same three flags as Create.Request. Absent for a clean call.
conversation.backchannel_count	How many acknowledgements ("mm-hm", "right") the caller made over the agent's speech.

SimulationRun.JobMetrics.Turn.text (field 18)

Each turn's words as the simulation's judges read them: the speaker's own words, or, when those are unknown, the words as the other side heard them. The agent's own words are unknown when it has no RemoteSession, e.g. over SIP.

A job already reports one row per turn with timings and scores, but not what was said. So a reader can see that a turn was slow or interrupted but not what it said. They also can't check the judged transcript's shape, such as one caller line per turn with each reply after the line it answers.

Consumers

• agent-service and the simulator fill these in livekit/agents-private#555.
• chore(deps): bump protocol for simulation job failure_reason, call facts and turn text livekit-cli#1042 bumps protocol so lk agent simulate export keeps them.

[changeset-bot]

🦋 Changeset detected

Latest commit: 4b2db4d

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
github.com/livekit/protocol	Patch
@livekit/protocol	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[theomonnom]

Suggested change

	string text = 18;
	string heard_text = 18;

[theomonnom]

Nice, does that means we will also be able to compared the two transcripts for WER

[u9g]

No, because Turn.text holds only one transcript per turn (usually what the speaker said, and what the other side heard only when that's unknown), while WER needs both the said and the heard versions of the same turn.

[theomonnom]

oh I wish we had a shared structs with the Create message

[u9g]

No, in the create message it's a boolean, here it's a string so you can know which one it was.

[u9g]

I changed the background noise to be sent as a boolean and moved it to it's own struct, but we don't have a way to reject out of date cli's from sending the wrong message so we'd have to accept the only version in create in perpetuity, until that mechanism is built first, so I left this as a todo there.