From f5755afabfad6aae6893bc31f34780a515b6c739 Mon Sep 17 00:00:00 2001 From: kevchoi <16812845+kevchoi@users.noreply.github.com> Date: Mon, 12 Oct 2026 16:59:02 +0000 Subject: [PATCH 1/8] feat: propose speech span schema for voice traces Co-Authored-By: Claude Opus 5.5 --- docs/voice-schema/speech-spans/README.md | 208 ++++++ .../speech-spans/examples/cascaded.json | 664 ++++++++++++++++++ .../speech-spans/examples/realtime.json | 608 ++++++++++++++++ .../livekit/voice-schema/speech-spans.test.ts | 74 ++ .../livekit/voice-schema/speech-spans.ts | 92 +++ 5 files changed, 1646 insertions(+) create mode 100644 docs/voice-schema/speech-spans/README.md create mode 100644 docs/voice-schema/speech-spans/examples/cascaded.json create mode 100644 docs/voice-schema/speech-spans/examples/realtime.json create mode 100644 js/src/instrumentation/livekit/voice-schema/speech-spans.test.ts create mode 100644 js/src/instrumentation/livekit/voice-schema/speech-spans.ts diff --git a/docs/voice-schema/speech-spans/README.md b/docs/voice-schema/speech-spans/README.md new file mode 100644 index 000000000..9e6acc112 --- /dev/null +++ b/docs/voice-schema/speech-spans/README.md @@ -0,0 +1,208 @@ +# Voice tracing option: speech spans + +> AI-created or modified and not human-reviewed in its current form; treat this artifact as provisional. Updated 2026-10-12. + +This option logs each utterance as its own `speech` span with framework-neutral metadata. It keeps the shared whole-call recording, `audio.recordings` and `audio.selections`. It changes how speech is identified: Lingua reads `metadata["voice.speech"]` instead of LiveKit span names and `contrib.livekit.*` keys. Turn spans, LLM spans and tool spans stay as they are. + +**Status:** + +| Part | Status | +| ------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | +| Contract, zod schema (`js/src/instrumentation/livekit/voice-schema/speech-spans.ts`), example rows and a test that validates the rows | Implemented | +| Lingua mapping | Implemented in the companion Lingua PR | +| LiveKit and Pipecat emission | Sketched below, not implemented | + +## Customer problem + +Customers want to replay a call and see four things together: + +- what each side said, with the matching audio; +- where time went; +- who interrupted whom; +- which tool calls happened. + +They also need to see where the trace lacks evidence. Today Lingua finds speech by span names that only LiveKit emits (`user_turn`, `assistant_turn`, `livekit.*_speaking`), so a Pipecat trace or a renamed span drops out of the voice view. When those names are missing, Lingua guesses each utterance's LLM step from start times. Interruption appears only as a LiveKit-specific turn flag. + +## Contract + +Every time is in **call ms**: milliseconds since the recording's `timeline.origin_unix_ms`, on the recording clock. A span's `metrics.start` and `metrics.end` are Unix seconds on the same clock: `(origin_unix_ms + call_ms) / 1000`. + +### Call span + +This is the root span: LiveKit `livekit.agent_session`, Pipecat `pipecat.pipeline`. + +| Field | Type | Required | Meaning | +| ---------------------------------- | -------------------------------------------- | ---------------------- | --------------------------------------------------------------------------------------------------------------------- | +| `input.audio[]` | attachment | When audio is captured | Unchanged (`@braintrust/audio`). | +| `metadata["audio.recordings"]` | descriptor[] | When audio is captured | Unchanged. Channel 0 is the caller, channel 1 the agent. | +| `metadata["voice.call"].framework` | `"livekit" \| "pipecat"` | Yes | The integration that emitted the call. | +| `metadata["voice.call"].pipeline` | `"cascaded" \| "realtime" \| "half_cascade"` | Yes | Tells readers which speech evidence to expect. `half_cascade` is a realtime model with text output plus separate TTS. | + +### Turn spans + +`user_turn` and `assistant_turn` task spans are unchanged: + +- `turn.id` is the turn span's own `span_id`. +- On an assistant turn, `turn.reply_to` is the user turn it answers. + +A speech span is a direct child of its turn. In this option turns group speech and steps; they don't carry speech semantics themselves. + +### Speech span + +Each utterance gets one span with `span_attributes: {type: "task", name: "speech"}`. The name stays fixed; Lingua recognizes the span by `voice.speech`, not by name. + +| Field | Type | Required | Meaning | +| -------------------------------------- | --------------------------------------------------------------------------------------- | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `metrics.start` / `metrics.end` | Unix s | Yes | When the utterance's audio started and ended on the recording clock. | +| `metadata["audio.selections"]` | Selection[] | When audio is captured | Unchanged shape. One channel: 0 for user speech, 1 for agent speech. | +| `voice.speech.speaker` | `"user" \| "agent"` | Yes | Who spoke. | +| `voice.speech.turn_id` | span id | No | The owning turn span, which is also the parent. | +| `voice.speech.state` | `"complete" \| "interrupted" \| "incomplete" \| "ignored"` | Yes | `interrupted` means agent playout was cut short. `incomplete` means the call or process ended mid-utterance. `ignored` means user speech the agent heard but did not answer (backchannel, false interruption). | +| `voice.speech.timing_source` | `"audio_samples" \| "vad" \| "provider_vad" \| "playout" \| "span_clock"` | Yes | The evidence behind `metrics.start` and `metrics.end`. `span_clock` means a guess from local spans. | +| `voice.speech.heard_by` | span id[] | User only, optional | The LLM spans whose input first contained this utterance. | +| `voice.speech.produced_by` | span id | Agent only, optional | The LLM span that generated the spoken text. | +| `voice.speech.interrupted_by` | span id | Agent only, optional | The user speech span that cut this one off. Requires `state: "interrupted"`. | +| `voice.speech.played_ms` | ms | Agent only, optional | Audio actually played locally before the end. | +| `voice.speech.provider_played_ms` | ms | Agent realtime only, optional | What the provider was told had played (`conversation.item.truncate` `audio_end_ms`). | +| `voice.speech.end_of_turn_ms` | call ms | User only, optional | When the agent decided the user had finished. | +| `voice.speech.transcript.source` | `"stt" \| "provider_transcription" \| "tts_input" \| "model_output" \| "playback_sync"` | With `transcript` | Where the text came from. | +| `voice.speech.transcript.final` | boolean | With `transcript` | `false` when only partial text arrived before the span ended. | +| `voice.speech.transcript.available_ms` | call ms | No | When the text became available. In realtime this is often after the response started. | +| `voice.speech.transcript.confidence` | 0–1 | No | STT confidence, if the provider reports one. | +| `output.text` | string | Content capture only | The best transcript of what was actually said. | +| `output.planned_text` | string | Content capture only | Agent text that was generated but not fully spoken. | + +**Absence rules:** + +- An omitted field means "not observed". Integrations never write `null` or placeholder text. +- With no transcript, both `transcript` and `output` are absent. +- When content capture is off, `output` is absent but `transcript.source` stays, so readers know a transcript existed. + +### IDs and relationships + +```text +call span ──audio.recordings──► recording segments + ├─ user_turn (turn.id = own span_id) + │ └─ speech {speaker:user, turn_id, heard_by:[llm span ids]} ──audio.selections──► channel 0 + └─ assistant_turn (turn.reply_to = user_turn) + ├─ llm_response ◄──produced_by── speech {speaker:agent, interrupted_by:user speech} + ├─ (tool_call_id) + └─ speech ──audio.selections──► channel 1 +``` + +- All links are `span_id`s in the same trace; Lingua maps them to row ids. +- A link may name a span that ends later than the speech span. Integrations update the earlier span with a merge write; see "late links" below. + +## What each integration captures and when + +| Value | LiveKit cascaded (JS) | LiveKit realtime (JS) | Pipecat cascaded (Python) | Pipecat realtime (Python) | +| ---------------------- | -------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | ----------------------------------------------- | +| User speech start/end | Native `user_speaking` span, clipped to captured input samples at its end (`processor.ts` `onEnd`) | Realtime `input_speech_started`/`stopped` (`realtime.ts` turn tracker) | `UserStartedSpeakingFrame`/`UserStoppedSpeakingFrame` | Same frames, or the service's server-VAD events | +| `end_of_turn_ms` | `AgentActivity.onEndOfTurn` (`EndOfTurnInfo`) | Commit of the input buffer (server VAD) | Turn analyzer stop (`UserStoppedSpeakingFrame` after the end-of-turn decision) | Server VAD commit | +| User transcript | `user_turn` transcript plus `transcription_delay` → `available_ms` | `InputTranscriptionCompleted` → `provider_transcription` | `TranscriptionFrame` (`finalized`) | The service's input transcription frames | +| `heard_by` | The `llm_node` started under the `agent_turn` whose `turn.reply_to` is this turn | The `realtime_inference` started after the commit | The `llm_response` opened by the next `LLMContextFrame` that contains the transcript | The response span after the commit | +| Agent speech start/end | Native `agent_speaking`, placed by `playbackProgressed`/`playbackFinished` | Same | `BotStartedSpeakingFrame`/`BotStoppedSpeakingFrame` at the output transport | Same | +| `produced_by` | The `llm_node` with the same `contrib.livekit.speech_id` | The `realtime_inference` with the same `speech_id`/`response_id` | The `llm_response` whose TTS `context_id` matches | The response that emitted the audio | +| `state`, `played_ms` | `playbackFinished.interrupted`, `playbackPosition` | Same, plus `provider_played_ms` from the truncate call | `InterruptionFrame` while the bot is speaking; position from bot-speaking frames | Same, plus truncation from the service | +| `interrupted_by` | The user speech span active when playout stopped | Same | Same | Same | +| Agent text | `synchronizedTranscript` → `playback_sync`; `tts` input → `planned_text` | Response transcript → `model_output` | `TTSTextFrame`s observed while speaking → `playback_sync` | The service's text frames → `model_output` | + +## What Lingua records and what it infers + +**Recorded as logged:** + +- the speaker, state, timing source and transcript source; +- `interrupted_by`; +- `end_of_turn_time` (origin + `end_of_turn_ms`); +- the selections; +- `step_ids`, taken from `produced_by` for agent speech and `heard_by` for user speech. + +**Inferred:** + +- `Speech.interrupted` (true when `state` is `interrupted`, false for other agent speech); +- the call each speech belongs to (nearest ancestor with recordings); +- speech order (by audio start), and overlap from the selections; +- `step_ids` falling back to the turn span and its step children when no link was logged. + +Latency is not stored. The app computes it from `end_of_turn_time`, step times and the next agent speech start. + +**Compatibility:** traces with no `voice.speech` keep #486's name-based behaviour unchanged. + +## Trajectory and app mapping + +`Trajectory.voice_calls[].speech[]` gains five optional fields: + +- `state` +- `interrupted_by` +- `transcript_source` +- `timing_source` +- `end_of_turn_time` + +The existing fields keep their meaning, so #22034 and #22036 work unchanged: + +- **Voice bar:** windows come from `selections`. +- **Thread view:** clicking a window or "Play from this message" resolves `step_ids` to a turn's `request_id`, `work[].id` or `response_id`. + +The new fields allow, without further schema work: + +- an "interrupted by …" marker; +- a "no transcript" badge when `transcript_source` is absent; +- a "timing estimated" badge for `span_clock`; +- an end-of-turn → first-audio latency bar. + +## Worked example + +`examples/cascaded.json` and `examples/realtime.json` encode the same fictional call: a booking change with `find_slots`, then a barge-in, then `reschedule`. Times are call ms. + +| | Cascaded | Realtime | +| ------------------------------- | ------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | +| Steps under `assistant_turn` 1 | `l1` llm (tool call) → `find_slots` → `l2` llm → `tts` | `r1` realtime response (tool call) → `find_slots` → `r2` response with audio | +| User speech `s1` | `timing_source: vad`; `transcript {stt, available_ms: 3450, confidence: 0.94}`; `heard_by: [l1]` | `timing_source: provider_vad`; `transcript {provider_transcription, available_ms: 4300}`, which arrived after the tool call at 4100; `heard_by: [r1]` | +| Agent speech `s2` | 5300–7900; `played_ms: 2600`; `produced_by: l2`; `output.text` is what was spoken, `planned_text` the full sentence | 5000–7900; `played_ms: 2900` but `provider_played_ms: 2600`, because the local output buffer kept playing for 300 ms after the truncate | +| Barge-in `s3` | `stt` transcript "Nine thirty works." | No transcript: `transcript` and `output` are absent, which the voice view can show as missing | +| End of turn → first agent audio | 3400 → 5300 (1.9 s), split across STT, LLM, tool and TTS spans | 3400 → 5000 (1.6 s), with no STT or TTS spans to split it | + +Lingua output for both files is checked in the companion PR's fixtures. + +The realtime fixture shows one weakness. Because user turn 2 has no transcript, Lingua's turn builder merges both exchanges into one Trajectory turn. `voice_calls` is still correct, but the thread view shows one long turn. + +## Customer usefulness + +| Question | How well this option answers it | +| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | +| What was said | Good. One span per utterance, with text and its provenance. Planned and spoken agent text are kept apart. | +| Play matching audio | Good. Each speech span has exact selections. | +| Diagnose delays | Partial. End of turn and audio start are explicit, but the pipeline breakdown still comes from LLM, TTS and STT spans, and realtime has no breakdown. | +| Interruptions and overlap | Good. `state`, `interrupted_by`, `played_ms` and `provider_played_ms` are explicit; overlap shows in the selections. | +| Follow tools | Indirect. Speech links to LLM steps, and tools sit beside them in the turn. Nothing says that a tool ran while the user was waiting in silence. | +| Missing or uncertain evidence | Good per utterance: `timing_source`, transcript `final`/`source`, and absent fields. There is no call-level summary of gaps. | + +## Edge cases and weaknesses + +- **Barge-in truncation.** Covered by `played_ms` and `planned_text`. The thread still shows the LLM's full planned answer as the turn response, because Lingua's `response_text` comes from the LLM span. +- **Overlap.** Two speech spans overlap in time; the app already prefers the later-starting window (#22036). +- **Late or missing realtime transcripts.** The integration must update `output` and `transcript` after the span ends, or leave them absent. A missing transcript splits turns badly (above). A fix would let a user speech with `heard_by` start a new turn without text; that is sketched, not implemented. +- **Failed or omitted segments.** Selections only point at ready segments, so the speech keeps its timing but has no playable audio. Lingua still lists it, with empty `selections`. +- **Unknown clock offset.** If the recording started late or the origin is wrong, every speech span is wrong by the same amount, and nothing in this option detects it. `timing_source: span_clock` marks only spans that were never aligned to audio. +- **Call in progress.** Speech whose call row isn't logged yet is skipped (#486 behaviour). `incomplete` is set only at shutdown. +- **Content capture off.** Text disappears but every link and timing field stays. +- **Tool cancelled by interruption.** The tool span has the error. The speech has no direct link to it; readers infer it from the turn. +- **Agent handoff.** Speech stays under the turn of the agent that spoke. `voice.call` has no field for the active agent; it would have to come from turn or agent metadata. +- **Late links.** `heard_by` and `interrupted_by` are known only after the speech span ends. Integrations need a second, merge `log()` on an already-ended span, which Braintrust allows but which costs one extra row write per utterance. +- **Strict enums.** Lingua rejects unknown `state`, `timing_source` and `transcript.source` values (the whole `voice.speech` key fails). New values need a Lingua release first. + +## Implementation starting point + +1. **LiveKit (`js/src/instrumentation/livekit/processor.ts`):** + - Rename the exported `livekit.user_speaking`/`livekit.agent_speaking` operations to `speech`, and log `voice.speech` at `onEnd` from the data already tracked (`speakingByTurn`, playback events, `EndOfTurnInfo`). + - Log `voice.call` on `agent_session` start. + - Add `heard_by` when an `llm_node`/`realtime_inference` span starts under a turn with `turn.reply_to`. +2. **Realtime (`realtime.ts`):** create user speech spans for SDK-built turns. Today only the turn is created and `user_speaking` is re-parented under it. +3. **Pipecat (`sdk-python/py/src/braintrust/integrations/pipecat`, `delner/pipecat-voice-recording`):** emit `speech` spans from the speaking and bot-speaking frames listed above. +4. **Lingua:** done in the companion PR. Then add `voice.speech` to `TRAJECTORY_METADATA_FIELDS` after #486 is rebased onto main. + +## Open questions + +- Should user speech that LiveKit merges into one turn (pause ≤ `maxPauseMs`) be one speech span or several? This draft logs one per VAD segment. +- Should `ignored` speech create a Trajectory turn? +- Is a per-utterance late write acceptable for the ingestion volume of long calls? diff --git a/docs/voice-schema/speech-spans/examples/cascaded.json b/docs/voice-schema/speech-spans/examples/cascaded.json new file mode 100644 index 000000000..5715b8a6a --- /dev/null +++ b/docs/voice-schema/speech-spans/examples/cascaded.json @@ -0,0 +1,664 @@ +[ + { + "id": "r-call", + "span_id": "call", + "root_span_id": "call", + "span_parents": null, + "span_attributes": { + "name": "livekit.agent_session", + "type": "task" + }, + "metrics": { + "start": 1791830000.0, + "end": 1791830013.0 + }, + "input": { + "audio": { + "call-0000": { + "type": "braintrust_attachment", + "filename": "call-0000.ogg", + "content_type": "audio/ogg", + "key": "fictional/call-0000.ogg" + } + } + }, + "metadata": { + "voice.call": { + "framework": "livekit", + "pipeline": "cascaded" + }, + "audio.recordings": [ + { + "id": "call-0000", + "recording_group_id": "call", + "state": "ready", + "attachment": { + "span_id": "call", + "ref": "/input/audio/call-0000" + }, + "mime_type": "audio/ogg", + "duration_ms": 13000, + "channel_count": 2, + "sample_rate_hz": 48000, + "sources": [ + { + "channel_index": 0, + "boundary": "session_input" + }, + { + "channel_index": 1, + "boundary": "local_audio_output" + } + ], + "timeline": { + "origin_unix_ms": 1791830000000, + "recording_start_offset_ms": 0, + "basis": "local_input_arrival_and_output_playout" + } + } + ] + } + }, + { + "id": "r-u1", + "span_id": "u1", + "root_span_id": "call", + "span_parents": ["call"], + "span_attributes": { + "name": "user_turn", + "type": "task" + }, + "metrics": { + "start": 1791830000.3, + "end": 1791830003.45 + }, + "input": [ + { + "role": "user", + "content": "Hi, can I move my Thursday cleaning to Friday morning?" + } + ], + "metadata": { + "turn.id": "u1" + } + }, + { + "id": "r-s1", + "span_id": "s1", + "root_span_id": "call", + "span_parents": ["u1"], + "span_attributes": { + "name": "speech", + "type": "task" + }, + "metrics": { + "start": 1791830000.3, + "end": 1791830002.9 + }, + "output": { + "text": "Hi, can I move my Thursday cleaning to Friday morning?" + }, + "metadata": { + "voice.speech": { + "speaker": "user", + "turn_id": "u1", + "state": "complete", + "timing_source": "vad", + "heard_by": ["l1"], + "end_of_turn_ms": 3400, + "transcript": { + "source": "stt", + "final": true, + "available_ms": 3450, + "confidence": 0.94 + } + }, + "audio.selections": [ + { + "recording_span_id": "call", + "recording_id": "call-0000", + "start_offset_ms": 300, + "end_offset_ms": 2900, + "channel_index": 0 + } + ] + } + }, + { + "id": "r-a1", + "span_id": "a1", + "root_span_id": "call", + "span_parents": ["call"], + "span_attributes": { + "name": "assistant_turn", + "type": "task" + }, + "metrics": { + "start": 1791830003.45, + "end": 1791830007.9 + }, + "output": [ + { + "role": "assistant", + "content": "I have nine thirty or eleven on Fri" + } + ], + "metadata": { + "turn.id": "a1", + "turn.reply_to": "u1" + } + }, + { + "id": "r-l1", + "span_id": "l1", + "root_span_id": "call", + "span_parents": ["a1"], + "span_attributes": { + "name": "llm_response", + "type": "llm" + }, + "metrics": { + "start": 1791830003.45, + "end": 1791830004.1, + "time_to_first_token": 0.45 + }, + "input": [ + { + "role": "system", + "content": "You are Molar, the front desk for a dental clinic." + }, + { + "role": "user", + "content": "Hi, can I move my Thursday cleaning to Friday morning?" + } + ], + "output": [ + { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_1", + "type": "function", + "function": { + "name": "find_slots", + "arguments": "{\"date\": \"2026-10-16\", \"window\": \"morning\"}" + } + } + ] + } + ], + "metadata": { + "model": "gpt-4.1-nano" + } + }, + { + "id": "r-t1", + "span_id": "t1", + "root_span_id": "call", + "span_parents": ["a1"], + "span_attributes": { + "name": "find_slots", + "type": "tool" + }, + "metrics": { + "start": 1791830004.1, + "end": 1791830004.6 + }, + "input": { + "date": "2026-10-16", + "window": "morning" + }, + "output": ["09:30", "11:00"], + "metadata": { + "tool_call_id": "call_1", + "gen_ai.tool.call.id": "call_1", + "gen_ai.tool.name": "find_slots", + "contrib.livekit.request_span_id": "l1" + } + }, + { + "id": "r-l2", + "span_id": "l2", + "root_span_id": "call", + "span_parents": ["a1"], + "span_attributes": { + "name": "llm_response", + "type": "llm" + }, + "metrics": { + "start": 1791830004.6, + "end": 1791830005.1, + "time_to_first_token": 0.2 + }, + "input": [ + { + "role": "system", + "content": "You are Molar, the front desk for a dental clinic." + }, + { + "role": "user", + "content": "Hi, can I move my Thursday cleaning to Friday morning?" + }, + { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_1", + "type": "function", + "function": { + "name": "find_slots", + "arguments": "{\"date\": \"2026-10-16\", \"window\": \"morning\"}" + } + } + ] + }, + { + "role": "tool", + "tool_call_id": "call_1", + "content": "[\"09:30\", \"11:00\"]" + } + ], + "output": [ + { + "role": "assistant", + "content": "I have nine thirty or eleven on Friday. Which would you like?" + } + ], + "metadata": { + "model": "gpt-4.1-nano" + } + }, + { + "id": "r-tts1", + "span_id": "tts1", + "root_span_id": "call", + "span_parents": ["a1"], + "span_attributes": { + "name": "tts", + "type": "task" + }, + "metrics": { + "start": 1791830005.1, + "end": 1791830007.9 + }, + "input": { + "text": "I have nine thirty or eleven on Friday. Which would you like?" + }, + "metadata": { + "model": "gpt-4o-mini-tts", + "contrib.livekit.response.ttfb": 0.2 + } + }, + { + "id": "r-s2", + "span_id": "s2", + "root_span_id": "call", + "span_parents": ["a1"], + "span_attributes": { + "name": "speech", + "type": "task" + }, + "metrics": { + "start": 1791830005.3, + "end": 1791830007.9 + }, + "output": { + "text": "I have nine thirty or eleven on Fri", + "planned_text": "I have nine thirty or eleven on Friday. Which would you like?" + }, + "metadata": { + "voice.speech": { + "speaker": "agent", + "turn_id": "a1", + "state": "interrupted", + "timing_source": "playout", + "produced_by": "l2", + "interrupted_by": "s3", + "played_ms": 2600, + "transcript": { + "source": "playback_sync", + "final": true + } + }, + "audio.selections": [ + { + "recording_span_id": "call", + "recording_id": "call-0000", + "start_offset_ms": 5300, + "end_offset_ms": 7900, + "channel_index": 1 + } + ] + } + }, + { + "id": "r-u2", + "span_id": "u2", + "root_span_id": "call", + "span_parents": ["call"], + "span_attributes": { + "name": "user_turn", + "type": "task" + }, + "metrics": { + "start": 1791830007.6, + "end": 1791830008.82 + }, + "input": [ + { + "role": "user", + "content": "Nine thirty works." + } + ], + "metadata": { + "turn.id": "u2" + } + }, + { + "id": "r-s3", + "span_id": "s3", + "root_span_id": "call", + "span_parents": ["u2"], + "span_attributes": { + "name": "speech", + "type": "task" + }, + "metrics": { + "start": 1791830007.6, + "end": 1791830008.5 + }, + "output": { + "text": "Nine thirty works." + }, + "metadata": { + "voice.speech": { + "speaker": "user", + "turn_id": "u2", + "state": "complete", + "timing_source": "vad", + "heard_by": ["l3"], + "end_of_turn_ms": 8800, + "transcript": { + "source": "stt", + "final": true, + "available_ms": 8820, + "confidence": 0.97 + } + }, + "audio.selections": [ + { + "recording_span_id": "call", + "recording_id": "call-0000", + "start_offset_ms": 7600, + "end_offset_ms": 8500, + "channel_index": 0 + } + ] + } + }, + { + "id": "r-a2", + "span_id": "a2", + "root_span_id": "call", + "span_parents": ["call"], + "span_attributes": { + "name": "assistant_turn", + "type": "task" + }, + "metrics": { + "start": 1791830008.82, + "end": 1791830012.6 + }, + "output": [ + { + "role": "assistant", + "content": "Done, you're booked Friday at nine thirty." + } + ], + "metadata": { + "turn.id": "a2", + "turn.reply_to": "u2" + } + }, + { + "id": "r-l3", + "span_id": "l3", + "root_span_id": "call", + "span_parents": ["a2"], + "span_attributes": { + "name": "llm_response", + "type": "llm" + }, + "metrics": { + "start": 1791830008.82, + "end": 1791830009.0, + "time_to_first_token": 0.15 + }, + "input": [ + { + "role": "system", + "content": "You are Molar, the front desk for a dental clinic." + }, + { + "role": "user", + "content": "Hi, can I move my Thursday cleaning to Friday morning?" + }, + { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_1", + "type": "function", + "function": { + "name": "find_slots", + "arguments": "{\"date\": \"2026-10-16\", \"window\": \"morning\"}" + } + } + ] + }, + { + "role": "tool", + "tool_call_id": "call_1", + "content": "[\"09:30\", \"11:00\"]" + }, + { + "role": "assistant", + "content": "I have nine thirty or eleven on Fri" + }, + { + "role": "user", + "content": "Nine thirty works." + } + ], + "output": [ + { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_2", + "type": "function", + "function": { + "name": "reschedule", + "arguments": "{\"appointment_id\": \"apt_481\", \"slot\": \"2026-10-16T09:30\"}" + } + } + ] + } + ], + "metadata": { + "model": "gpt-4.1-nano" + } + }, + { + "id": "r-t2", + "span_id": "t2", + "root_span_id": "call", + "span_parents": ["a2"], + "span_attributes": { + "name": "reschedule", + "type": "tool" + }, + "metrics": { + "start": 1791830009.0, + "end": 1791830009.4 + }, + "input": { + "appointment_id": "apt_481", + "slot": "2026-10-16T09:30" + }, + "output": { + "ok": true + }, + "metadata": { + "tool_call_id": "call_2", + "gen_ai.tool.call.id": "call_2", + "gen_ai.tool.name": "reschedule", + "contrib.livekit.request_span_id": "l3" + } + }, + { + "id": "r-l4", + "span_id": "l4", + "root_span_id": "call", + "span_parents": ["a2"], + "span_attributes": { + "name": "llm_response", + "type": "llm" + }, + "metrics": { + "start": 1791830009.4, + "end": 1791830009.7, + "time_to_first_token": 0.12 + }, + "input": [ + { + "role": "system", + "content": "You are Molar, the front desk for a dental clinic." + }, + { + "role": "user", + "content": "Hi, can I move my Thursday cleaning to Friday morning?" + }, + { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_1", + "type": "function", + "function": { + "name": "find_slots", + "arguments": "{\"date\": \"2026-10-16\", \"window\": \"morning\"}" + } + } + ] + }, + { + "role": "tool", + "tool_call_id": "call_1", + "content": "[\"09:30\", \"11:00\"]" + }, + { + "role": "assistant", + "content": "I have nine thirty or eleven on Fri" + }, + { + "role": "user", + "content": "Nine thirty works." + }, + { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_2", + "type": "function", + "function": { + "name": "reschedule", + "arguments": "{\"appointment_id\": \"apt_481\", \"slot\": \"2026-10-16T09:30\"}" + } + } + ] + }, + { + "role": "tool", + "tool_call_id": "call_2", + "content": "{\"ok\": true}" + } + ], + "output": [ + { + "role": "assistant", + "content": "Done, you're booked Friday at nine thirty." + } + ], + "metadata": { + "model": "gpt-4.1-nano" + } + }, + { + "id": "r-tts2", + "span_id": "tts2", + "root_span_id": "call", + "span_parents": ["a2"], + "span_attributes": { + "name": "tts", + "type": "task" + }, + "metrics": { + "start": 1791830009.7, + "end": 1791830012.6 + }, + "input": { + "text": "Done, you're booked Friday at nine thirty." + }, + "metadata": { + "model": "gpt-4o-mini-tts", + "contrib.livekit.response.ttfb": 0.2 + } + }, + { + "id": "r-s4", + "span_id": "s4", + "root_span_id": "call", + "span_parents": ["a2"], + "span_attributes": { + "name": "speech", + "type": "task" + }, + "metrics": { + "start": 1791830009.9, + "end": 1791830012.6 + }, + "output": { + "text": "Done, you're booked Friday at nine thirty." + }, + "metadata": { + "voice.speech": { + "speaker": "agent", + "turn_id": "a2", + "state": "complete", + "timing_source": "playout", + "produced_by": "l4", + "played_ms": 2700, + "transcript": { + "source": "playback_sync", + "final": true + } + }, + "audio.selections": [ + { + "recording_span_id": "call", + "recording_id": "call-0000", + "start_offset_ms": 9900, + "end_offset_ms": 12600, + "channel_index": 1 + } + ] + } + } +] diff --git a/docs/voice-schema/speech-spans/examples/realtime.json b/docs/voice-schema/speech-spans/examples/realtime.json new file mode 100644 index 000000000..937185e03 --- /dev/null +++ b/docs/voice-schema/speech-spans/examples/realtime.json @@ -0,0 +1,608 @@ +[ + { + "id": "r-call", + "span_id": "call", + "root_span_id": "call", + "span_parents": null, + "span_attributes": { + "name": "livekit.agent_session", + "type": "task" + }, + "metrics": { + "start": 1791830000.0, + "end": 1791830013.0 + }, + "input": { + "audio": { + "call-0000": { + "type": "braintrust_attachment", + "filename": "call-0000.ogg", + "content_type": "audio/ogg", + "key": "fictional/call-0000.ogg" + } + } + }, + "metadata": { + "voice.call": { + "framework": "livekit", + "pipeline": "realtime" + }, + "audio.recordings": [ + { + "id": "call-0000", + "recording_group_id": "call", + "state": "ready", + "attachment": { + "span_id": "call", + "ref": "/input/audio/call-0000" + }, + "mime_type": "audio/ogg", + "duration_ms": 13000, + "channel_count": 2, + "sample_rate_hz": 48000, + "sources": [ + { + "channel_index": 0, + "boundary": "session_input" + }, + { + "channel_index": 1, + "boundary": "local_audio_output" + } + ], + "timeline": { + "origin_unix_ms": 1791830000000, + "recording_start_offset_ms": 0, + "basis": "local_input_arrival_and_output_playout" + } + } + ] + } + }, + { + "id": "r-u1", + "span_id": "u1", + "root_span_id": "call", + "span_parents": ["call"], + "span_attributes": { + "name": "user_turn", + "type": "task" + }, + "metrics": { + "start": 1791830000.3, + "end": 1791830004.3 + }, + "input": [ + { + "role": "user", + "content": "Hi, can I move my Thursday cleaning to Friday morning?" + } + ], + "metadata": { + "turn.id": "u1" + } + }, + { + "id": "r-s1", + "span_id": "s1", + "root_span_id": "call", + "span_parents": ["u1"], + "span_attributes": { + "name": "speech", + "type": "task" + }, + "metrics": { + "start": 1791830000.3, + "end": 1791830002.9 + }, + "output": { + "text": "Hi, can I move my Thursday cleaning to Friday morning?" + }, + "metadata": { + "voice.speech": { + "speaker": "user", + "turn_id": "u1", + "state": "complete", + "timing_source": "provider_vad", + "heard_by": ["r1"], + "end_of_turn_ms": 3400, + "transcript": { + "source": "provider_transcription", + "final": true, + "available_ms": 4300 + } + }, + "audio.selections": [ + { + "recording_span_id": "call", + "recording_id": "call-0000", + "start_offset_ms": 300, + "end_offset_ms": 2900, + "channel_index": 0 + } + ] + } + }, + { + "id": "r-a1", + "span_id": "a1", + "root_span_id": "call", + "span_parents": ["call"], + "span_attributes": { + "name": "assistant_turn", + "type": "task" + }, + "metrics": { + "start": 1791830003.6, + "end": 1791830007.9 + }, + "output": [ + { + "role": "assistant", + "content": "I have nine thirty or eleven on Fri" + } + ], + "metadata": { + "turn.id": "a1", + "turn.reply_to": "u1" + } + }, + { + "id": "r-r1", + "span_id": "r1", + "root_span_id": "call", + "span_parents": ["a1"], + "span_attributes": { + "name": "llm_response", + "type": "llm" + }, + "metrics": { + "start": 1791830003.6, + "end": 1791830004.1, + "time_to_first_token": 0.5 + }, + "input": [ + { + "role": "system", + "content": "You are Molar, the front desk for a dental clinic." + }, + { + "role": "user", + "content": "Hi, can I move my Thursday cleaning to Friday morning?" + } + ], + "output": [ + { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_1", + "type": "function", + "function": { + "name": "find_slots", + "arguments": "{\"date\": \"2026-10-16\", \"window\": \"morning\"}" + } + } + ] + } + ], + "metadata": { + "model": "gpt-realtime-mini", + "contrib.livekit.input_scope": "realtime_session_context", + "contrib.livekit.response_id": "resp_1" + } + }, + { + "id": "r-t1", + "span_id": "t1", + "root_span_id": "call", + "span_parents": ["a1"], + "span_attributes": { + "name": "find_slots", + "type": "tool" + }, + "metrics": { + "start": 1791830004.1, + "end": 1791830004.6 + }, + "input": { + "date": "2026-10-16", + "window": "morning" + }, + "output": ["09:30", "11:00"], + "metadata": { + "tool_call_id": "call_1", + "gen_ai.tool.call.id": "call_1", + "gen_ai.tool.name": "find_slots", + "contrib.livekit.request_span_id": "r1" + } + }, + { + "id": "r-r2", + "span_id": "r2", + "root_span_id": "call", + "span_parents": ["a1"], + "span_attributes": { + "name": "llm_response", + "type": "llm" + }, + "metrics": { + "start": 1791830004.6, + "end": 1791830007.9, + "time_to_first_token": 0.4 + }, + "input": [ + { + "role": "system", + "content": "You are Molar, the front desk for a dental clinic." + }, + { + "role": "user", + "content": "Hi, can I move my Thursday cleaning to Friday morning?" + }, + { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_1", + "type": "function", + "function": { + "name": "find_slots", + "arguments": "{\"date\": \"2026-10-16\", \"window\": \"morning\"}" + } + } + ] + }, + { + "role": "tool", + "tool_call_id": "call_1", + "content": "[\"09:30\", \"11:00\"]" + } + ], + "output": [ + { + "role": "assistant", + "content": "I have nine thirty or eleven on Fri" + } + ], + "metadata": { + "model": "gpt-realtime-mini", + "contrib.livekit.input_scope": "realtime_session_context", + "contrib.livekit.response_id": "resp_2" + } + }, + { + "id": "r-s2", + "span_id": "s2", + "root_span_id": "call", + "span_parents": ["a1"], + "span_attributes": { + "name": "speech", + "type": "task" + }, + "metrics": { + "start": 1791830005.0, + "end": 1791830007.9 + }, + "output": { + "text": "I have nine thirty or eleven on Fri", + "planned_text": "I have nine thirty or eleven on Friday. Which would you like?" + }, + "metadata": { + "voice.speech": { + "speaker": "agent", + "turn_id": "a1", + "state": "interrupted", + "timing_source": "playout", + "produced_by": "r2", + "interrupted_by": "s3", + "played_ms": 2900, + "provider_played_ms": 2600, + "transcript": { + "source": "model_output", + "final": true + } + }, + "audio.selections": [ + { + "recording_span_id": "call", + "recording_id": "call-0000", + "start_offset_ms": 5000, + "end_offset_ms": 7900, + "channel_index": 1 + } + ] + } + }, + { + "id": "r-u2", + "span_id": "u2", + "root_span_id": "call", + "span_parents": ["call"], + "span_attributes": { + "name": "user_turn", + "type": "task" + }, + "metrics": { + "start": 1791830007.6, + "end": 1791830008.5 + }, + "input": [], + "metadata": { + "turn.id": "u2" + } + }, + { + "id": "r-s3", + "span_id": "s3", + "root_span_id": "call", + "span_parents": ["u2"], + "span_attributes": { + "name": "speech", + "type": "task" + }, + "metrics": { + "start": 1791830007.6, + "end": 1791830008.5 + }, + "metadata": { + "voice.speech": { + "speaker": "user", + "turn_id": "u2", + "state": "complete", + "timing_source": "provider_vad", + "heard_by": ["r3"], + "end_of_turn_ms": 8600 + }, + "audio.selections": [ + { + "recording_span_id": "call", + "recording_id": "call-0000", + "start_offset_ms": 7600, + "end_offset_ms": 8500, + "channel_index": 0 + } + ] + } + }, + { + "id": "r-a2", + "span_id": "a2", + "root_span_id": "call", + "span_parents": ["call"], + "span_attributes": { + "name": "assistant_turn", + "type": "task" + }, + "metrics": { + "start": 1791830008.6, + "end": 1791830012.5 + }, + "output": [ + { + "role": "assistant", + "content": "Done, you're booked Friday at nine thirty." + } + ], + "metadata": { + "turn.id": "a2", + "turn.reply_to": "u2" + } + }, + { + "id": "r-r3", + "span_id": "r3", + "root_span_id": "call", + "span_parents": ["a2"], + "span_attributes": { + "name": "llm_response", + "type": "llm" + }, + "metrics": { + "start": 1791830008.6, + "end": 1791830008.9, + "time_to_first_token": 0.3 + }, + "input": [ + { + "role": "system", + "content": "You are Molar, the front desk for a dental clinic." + }, + { + "role": "user", + "content": "Hi, can I move my Thursday cleaning to Friday morning?" + }, + { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_1", + "type": "function", + "function": { + "name": "find_slots", + "arguments": "{\"date\": \"2026-10-16\", \"window\": \"morning\"}" + } + } + ] + }, + { + "role": "tool", + "tool_call_id": "call_1", + "content": "[\"09:30\", \"11:00\"]" + }, + { + "role": "assistant", + "content": "I have nine thirty or eleven on Fri" + } + ], + "output": [ + { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_2", + "type": "function", + "function": { + "name": "reschedule", + "arguments": "{\"appointment_id\": \"apt_481\", \"slot\": \"2026-10-16T09:30\"}" + } + } + ] + } + ], + "metadata": { + "model": "gpt-realtime-mini", + "contrib.livekit.input_scope": "realtime_session_context", + "contrib.livekit.response_id": "resp_3" + } + }, + { + "id": "r-t2", + "span_id": "t2", + "root_span_id": "call", + "span_parents": ["a2"], + "span_attributes": { + "name": "reschedule", + "type": "tool" + }, + "metrics": { + "start": 1791830008.9, + "end": 1791830009.3 + }, + "input": { + "appointment_id": "apt_481", + "slot": "2026-10-16T09:30" + }, + "output": { + "ok": true + }, + "metadata": { + "tool_call_id": "call_2", + "gen_ai.tool.call.id": "call_2", + "gen_ai.tool.name": "reschedule", + "contrib.livekit.request_span_id": "r3" + } + }, + { + "id": "r-r4", + "span_id": "r4", + "root_span_id": "call", + "span_parents": ["a2"], + "span_attributes": { + "name": "llm_response", + "type": "llm" + }, + "metrics": { + "start": 1791830009.3, + "end": 1791830012.5, + "time_to_first_token": 0.4 + }, + "input": [ + { + "role": "system", + "content": "You are Molar, the front desk for a dental clinic." + }, + { + "role": "user", + "content": "Hi, can I move my Thursday cleaning to Friday morning?" + }, + { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_1", + "type": "function", + "function": { + "name": "find_slots", + "arguments": "{\"date\": \"2026-10-16\", \"window\": \"morning\"}" + } + } + ] + }, + { + "role": "tool", + "tool_call_id": "call_1", + "content": "[\"09:30\", \"11:00\"]" + }, + { + "role": "assistant", + "content": "I have nine thirty or eleven on Fri" + }, + { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_2", + "type": "function", + "function": { + "name": "reschedule", + "arguments": "{\"appointment_id\": \"apt_481\", \"slot\": \"2026-10-16T09:30\"}" + } + } + ] + }, + { + "role": "tool", + "tool_call_id": "call_2", + "content": "{\"ok\": true}" + } + ], + "output": [ + { + "role": "assistant", + "content": "Done, you're booked Friday at nine thirty." + } + ], + "metadata": { + "model": "gpt-realtime-mini", + "contrib.livekit.input_scope": "realtime_session_context", + "contrib.livekit.response_id": "resp_4" + } + }, + { + "id": "r-s4", + "span_id": "s4", + "root_span_id": "call", + "span_parents": ["a2"], + "span_attributes": { + "name": "speech", + "type": "task" + }, + "metrics": { + "start": 1791830009.7, + "end": 1791830012.5 + }, + "output": { + "text": "Done, you're booked Friday at nine thirty." + }, + "metadata": { + "voice.speech": { + "speaker": "agent", + "turn_id": "a2", + "state": "complete", + "timing_source": "playout", + "produced_by": "r4", + "played_ms": 2800, + "transcript": { + "source": "model_output", + "final": true + } + }, + "audio.selections": [ + { + "recording_span_id": "call", + "recording_id": "call-0000", + "start_offset_ms": 9700, + "end_offset_ms": 12500, + "channel_index": 1 + } + ] + } + } +] diff --git a/js/src/instrumentation/livekit/voice-schema/speech-spans.test.ts b/js/src/instrumentation/livekit/voice-schema/speech-spans.test.ts new file mode 100644 index 000000000..05199f36a --- /dev/null +++ b/js/src/instrumentation/livekit/voice-schema/speech-spans.test.ts @@ -0,0 +1,74 @@ +import { readFileSync } from "node:fs"; +import path from "node:path"; +import { describe, expect, test } from "vitest"; +import { z } from "zod/v3"; +import { speechSpan, voiceCall } from "./speech-spans"; + +type Row = { + span_id: string; + span_parents: string[] | null; + span_attributes: { name: string; type: string }; + metadata?: Record; +}; + +const examples = path.resolve( + __dirname, + "../../../../../docs/voice-schema/speech-spans/examples", +); + +describe.each(["cascaded", "realtime"])("%s example", (name) => { + const rows: Row[] = JSON.parse( + readFileSync(path.join(examples, `${name}.json`), "utf8"), + ); + const byId = new Map(rows.map((row) => [row.span_id, row])); + const speech = rows + .filter((row) => row.metadata?.["voice.speech"]) + .map((row) => speechSpan.parse(row)); + + test("the call span declares its pipeline", () => { + const call = rows.find((row) => row.span_parents === null); + expect(voiceCall.parse(call?.metadata?.["voice.call"]).pipeline).toBe(name); + }); + + test("speech links resolve to spans of the right kind", () => { + expect(speech).toHaveLength(4); + for (const span of speech) { + const voice = span.metadata["voice.speech"]; + if (voice.turn_id) expect(span.span_parents).toEqual([voice.turn_id]); + for (const id of voice.heard_by ?? []) + expect(byId.get(id)?.span_attributes.type).toBe("llm"); + if (voice.produced_by) + expect(byId.get(voice.produced_by)?.span_attributes.type).toBe("llm"); + if (voice.interrupted_by) + expect( + speechSpan.parse(byId.get(voice.interrupted_by)).metadata[ + "voice.speech" + ].speaker, + ).toBe("user"); + for (const selection of span.metadata["audio.selections"] ?? []) { + expect(byId.has(selection.recording_span_id)).toBe(true); + expect(selection.channel_index).toBe(voice.speaker === "user" ? 0 : 1); + } + } + }); + + test("speech span times match their audio selections", () => { + const [recording] = z + .array(z.object({ timeline: z.object({ origin_unix_ms: z.number() }) })) + .parse( + rows.find((row) => row.span_parents === null)?.metadata?.[ + "audio.recordings" + ], + ); + const origin = recording.timeline.origin_unix_ms; + for (const span of speech) { + const [selection] = span.metadata["audio.selections"] ?? []; + expect(span.metrics.start * 1000 - origin).toBeCloseTo( + selection.start_offset_ms, + ); + expect(span.metrics.end * 1000 - origin).toBeCloseTo( + selection.end_offset_ms, + ); + } + }); +}); diff --git a/js/src/instrumentation/livekit/voice-schema/speech-spans.ts b/js/src/instrumentation/livekit/voice-schema/speech-spans.ts new file mode 100644 index 000000000..d7464012a --- /dev/null +++ b/js/src/instrumentation/livekit/voice-schema/speech-spans.ts @@ -0,0 +1,92 @@ +import { z } from "zod/v3"; + +const callMs = z.number().nonnegative(); + +const selection = z.object({ + recording_span_id: z.string(), + recording_id: z.string(), + start_offset_ms: callMs, + end_offset_ms: callMs, + channel_index: z.union([z.literal(0), z.literal(1)]), +}); + +export const voiceCall = z.object({ + framework: z.enum(["livekit", "pipecat"]), + pipeline: z.enum(["cascaded", "realtime", "half_cascade"]), +}); + +export const voiceSpeech = z + .object({ + speaker: z.enum(["user", "agent"]), + turn_id: z.string().optional(), + state: z.enum(["complete", "interrupted", "incomplete", "ignored"]), + timing_source: z.enum([ + "audio_samples", + "vad", + "provider_vad", + "playout", + "span_clock", + ]), + heard_by: z.array(z.string()).optional(), + produced_by: z.string().optional(), + interrupted_by: z.string().optional(), + played_ms: callMs.optional(), + provider_played_ms: callMs.optional(), + end_of_turn_ms: callMs.optional(), + transcript: z + .object({ + source: z.enum([ + "stt", + "provider_transcription", + "tts_input", + "model_output", + "playback_sync", + ]), + final: z.boolean(), + available_ms: callMs.optional(), + confidence: z.number().min(0).max(1).optional(), + }) + .optional(), + }) + .superRefine((speech, ctx) => { + const agentOnly = [ + "produced_by", + "interrupted_by", + "played_ms", + "provider_played_ms", + ] as const; + const userOnly = ["heard_by", "end_of_turn_ms"] as const; + for (const key of speech.speaker === "user" ? agentOnly : userOnly) + if (speech[key] !== undefined) + ctx.addIssue({ + code: z.ZodIssueCode.custom, + message: `${key} is not allowed on ${speech.speaker} speech`, + path: [key], + }); + if (speech.interrupted_by && speech.state !== "interrupted") + ctx.addIssue({ + code: z.ZodIssueCode.custom, + message: "interrupted_by requires state interrupted", + path: ["interrupted_by"], + }); + }); + +export const speechSpan = z.object({ + span_id: z.string(), + span_parents: z.array(z.string()).length(1), + span_attributes: z.object({ + name: z.literal("speech"), + type: z.literal("task"), + }), + metrics: z.object({ start: z.number(), end: z.number() }), + output: z + .object({ text: z.string(), planned_text: z.string().optional() }) + .optional(), + metadata: z.object({ + "voice.speech": voiceSpeech, + "audio.selections": z.array(selection).optional(), + }), +}); + +export type VoiceSpeech = z.infer; +export type SpeechSpan = z.infer; From 16dd784b03217f80b5e8ff42d8a214ffcd96f29c Mon Sep 17 00:00:00 2001 From: kevchoi <16812845+kevchoi@users.noreply.github.com> Date: Mon, 12 Oct 2026 17:19:36 +0000 Subject: [PATCH 2/8] feat: resolve speech span semantics gaps Co-Authored-By: Claude Opus 5.5 --- docs/voice-schema/speech-spans/README.md | 242 +++++++++++------- .../speech-spans/examples/cascaded.json | 49 +++- .../speech-spans/examples/realtime.json | 40 ++- .../livekit/voice-schema/speech-spans.test.ts | 74 ++++-- .../livekit/voice-schema/speech-spans.ts | 50 ++-- 5 files changed, 329 insertions(+), 126 deletions(-) diff --git a/docs/voice-schema/speech-spans/README.md b/docs/voice-schema/speech-spans/README.md index 9e6acc112..de9af2e3b 100644 --- a/docs/voice-schema/speech-spans/README.md +++ b/docs/voice-schema/speech-spans/README.md @@ -2,7 +2,7 @@ > AI-created or modified and not human-reviewed in its current form; treat this artifact as provisional. Updated 2026-10-12. -This option logs each utterance as its own `speech` span with framework-neutral metadata. It keeps the shared whole-call recording, `audio.recordings` and `audio.selections`. It changes how speech is identified: Lingua reads `metadata["voice.speech"]` instead of LiveKit span names and `contrib.livekit.*` keys. Turn spans, LLM spans and tool spans stay as they are. +This option logs each utterance as its own `speech` span with framework-neutral metadata. It keeps the shared whole-call recording, `audio.recordings` and `audio.selections`. It changes how speech is identified: Lingua reads `metadata["voice.speech"]` and `metadata["voice.call"]` instead of LiveKit span names and `contrib.livekit.*` keys. Turn spans, LLM spans and tool spans stay as they are. **Status:** @@ -25,18 +25,23 @@ They also need to see where the trace lacks evidence. Today Lingua finds speech ## Contract -Every time is in **call ms**: milliseconds since the recording's `timeline.origin_unix_ms`, on the recording clock. A span's `metrics.start` and `metrics.end` are Unix seconds on the same clock: `(origin_unix_ms + call_ms) / 1000`. +### Clock + +- **Call ms** are milliseconds since `voice.call.origin_unix_ms`. +- **Recording alignment:** when audio is captured, `origin_unix_ms` equals the recording's `timeline.origin_unix_ms`, so call ms are also recording-timeline ms. +- **Span times:** a span's `metrics.start` and `metrics.end` are Unix seconds on the same clock: `(origin_unix_ms + call_ms) / 1000`. ### Call span -This is the root span: LiveKit `livekit.agent_session`, Pipecat `pipecat.pipeline`. +This is the root span: LiveKit `livekit.agent_session`, Pipecat `pipecat.pipeline`. Lingua finds the call through `voice.call`, so a call without audio still has speech. -| Field | Type | Required | Meaning | -| ---------------------------------- | -------------------------------------------- | ---------------------- | --------------------------------------------------------------------------------------------------------------------- | -| `input.audio[]` | attachment | When audio is captured | Unchanged (`@braintrust/audio`). | -| `metadata["audio.recordings"]` | descriptor[] | When audio is captured | Unchanged. Channel 0 is the caller, channel 1 the agent. | -| `metadata["voice.call"].framework` | `"livekit" \| "pipecat"` | Yes | The integration that emitted the call. | -| `metadata["voice.call"].pipeline` | `"cascaded" \| "realtime" \| "half_cascade"` | Yes | Tells readers which speech evidence to expect. `half_cascade` is a realtime model with text output plus separate TTS. | +| Field | Type | Required | Meaning | +| ------------------------------ | -------------------------------------------- | ---------------------- | --------------------------------------------------------------------------------------------------------------------- | +| `input.audio[]` | attachment | When audio is captured | Unchanged (`@braintrust/audio`). | +| `metadata["audio.recordings"]` | descriptor[] | When audio is captured | Unchanged. Channel 0 is the caller, channel 1 the agent. | +| `voice.call.framework` | `"livekit" \| "pipecat"` | Yes | The integration that emitted the call. | +| `voice.call.pipeline` | `"cascaded" \| "realtime" \| "half_cascade"` | Yes | Tells readers which speech evidence to expect. `half_cascade` is a realtime model with text output plus separate TTS. | +| `voice.call.origin_unix_ms` | Unix ms | Yes | The clock for every call-ms field. Logged at session start, whether or not audio is captured. | ### Turn spans @@ -45,67 +50,110 @@ This is the root span: LiveKit `livekit.agent_session`, Pipecat `pipecat.pipelin - `turn.id` is the turn span's own `span_id`. - On an assistant turn, `turn.reply_to` is the user turn it answers. -A speech span is a direct child of its turn. In this option turns group speech and steps; they don't carry speech semantics themselves. +Both integrations emit these names; Pipecat's turn spans already use them on `delner/pipecat-voice-recording`. In this option turns group speech and steps; they don't carry speech semantics themselves. ### Speech span Each utterance gets one span with `span_attributes: {type: "task", name: "speech"}`. The name stays fixed; Lingua recognizes the span by `voice.speech`, not by name. -| Field | Type | Required | Meaning | -| -------------------------------------- | --------------------------------------------------------------------------------------- | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `metrics.start` / `metrics.end` | Unix s | Yes | When the utterance's audio started and ended on the recording clock. | -| `metadata["audio.selections"]` | Selection[] | When audio is captured | Unchanged shape. One channel: 0 for user speech, 1 for agent speech. | -| `voice.speech.speaker` | `"user" \| "agent"` | Yes | Who spoke. | -| `voice.speech.turn_id` | span id | No | The owning turn span, which is also the parent. | -| `voice.speech.state` | `"complete" \| "interrupted" \| "incomplete" \| "ignored"` | Yes | `interrupted` means agent playout was cut short. `incomplete` means the call or process ended mid-utterance. `ignored` means user speech the agent heard but did not answer (backchannel, false interruption). | -| `voice.speech.timing_source` | `"audio_samples" \| "vad" \| "provider_vad" \| "playout" \| "span_clock"` | Yes | The evidence behind `metrics.start` and `metrics.end`. `span_clock` means a guess from local spans. | -| `voice.speech.heard_by` | span id[] | User only, optional | The LLM spans whose input first contained this utterance. | -| `voice.speech.produced_by` | span id | Agent only, optional | The LLM span that generated the spoken text. | -| `voice.speech.interrupted_by` | span id | Agent only, optional | The user speech span that cut this one off. Requires `state: "interrupted"`. | -| `voice.speech.played_ms` | ms | Agent only, optional | Audio actually played locally before the end. | -| `voice.speech.provider_played_ms` | ms | Agent realtime only, optional | What the provider was told had played (`conversation.item.truncate` `audio_end_ms`). | -| `voice.speech.end_of_turn_ms` | call ms | User only, optional | When the agent decided the user had finished. | -| `voice.speech.transcript.source` | `"stt" \| "provider_transcription" \| "tts_input" \| "model_output" \| "playback_sync"` | With `transcript` | Where the text came from. | -| `voice.speech.transcript.final` | boolean | With `transcript` | `false` when only partial text arrived before the span ended. | -| `voice.speech.transcript.available_ms` | call ms | No | When the text became available. In realtime this is often after the response started. | -| `voice.speech.transcript.confidence` | 0–1 | No | STT confidence, if the provider reports one. | -| `output.text` | string | Content capture only | The best transcript of what was actually said. | -| `output.planned_text` | string | Content capture only | Agent text that was generated but not fully spoken. | +**Utterance grain:** + +| Speaker | One speech span per… | How fragments are kept | +| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | +| User | Turn: everything the user said that the turn detector grouped into one user turn | Each VAD fragment becomes one `audio.selections` range on that span. | +| User, not part of a turn (backchannel, false interruption) | Ignored utterance | One span with `state: "ignored"`. | +| Agent | Playout of one speech handle: LiveKit `SpeechHandle`, Pipecat `BotStartedSpeakingFrame` → `BotStoppedSpeakingFrame` | Pause and resume after a false interruption stays one span, with one selection per played stretch. | + +The grain matters because a cascaded call can produce several speaking fragments per user turn. Integrations must not emit one span per fragment. + +An interrupted speech handle that played no audio produces no speech span; its text remains on the LLM span. So `played_ms` is always positive when present. + +**Fields:** + +| Field | Type | Required | Meaning | +| -------------------------------------- | --------------------------------------------------------------------------------------- | ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `span_parents` | `[span id]` | Yes | The owning turn span. Ignored speech has the call span as its parent. | +| `metrics.start` / `metrics.end` | Unix s | Yes | The utterance bounds. With selections: first selection start to last selection end. | +| `metadata["audio.selections"]` | Selection[] | When audio is captured for that utterance | Unchanged shape. One channel: 0 for user speech, 1 for agent speech. One range per fragment or played stretch, in order. | +| `voice.speech.speaker` | `"user" \| "agent"` | Yes | Who spoke. | +| `voice.speech.turn_id` | span id | When the parent is a turn | The owning turn span. Omitted for ignored speech. | +| `voice.speech.state` | `"complete" \| "interrupted" \| "incomplete" \| "ignored"` | Yes | `interrupted` means agent playout was cut short after some audio played. `incomplete` means the call or process ended mid-utterance. `ignored` means user speech the agent heard but did not answer. | +| `voice.speech.timing_source` | See below | Yes | The evidence behind `metrics.start`/`metrics.end`. | +| `voice.speech.heard_by` | span id[] | User only, optional | The LLM spans whose input first contained this utterance. | +| `voice.speech.produced_by` | span id | Agent only, optional | The LLM span that generated the spoken text. | +| `voice.speech.interrupted_by` | span id | Agent only, optional | The user speech span that cut this one off. Requires `state: "interrupted"`. | +| `voice.speech.played_ms` | ms > 0 | Agent only, optional | Audio actually played locally: bytes written to the output transport divided by its byte rate, or the framework's playback position. | +| `voice.speech.provider_played_ms` | ms | Agent realtime only, optional | What the provider was told had played (`conversation.item.truncate` `audio_end_ms`). | +| `voice.speech.end_of_turn_ms` | call ms | User only, optional | When the agent decided the user had finished. | +| `voice.speech.transcript.source` | `"stt" \| "provider_transcription" \| "tts_input" \| "model_output" \| "playback_sync"` | With `transcript` | Where the text came from. | +| `voice.speech.transcript.final` | boolean | With `transcript` | `false` when only partial text arrived before the span was written. | +| `voice.speech.transcript.available_ms` | call ms | No | When the text became available. In realtime this is often after the response started. | +| `voice.speech.transcript.confidence` | 0–1 | No | STT confidence, if the provider reports one. | +| `output.text` | string | No | The best transcript of what was actually said. Absent without a transcript or with content capture off. | +| `output.planned_text` | string | No | Agent text that was generated but not fully spoken. | + +**`timing_source` values:** + +| Value | Meaning | +| --------------- | ------------------------------------------------------------------------------------------------------------------------------ | +| `audio_samples` | Bounds measured on captured audio samples. Requires `audio.selections`. | +| `vad` | Local VAD or turn-detector events (LiveKit `user_speaking`, Pipecat `UserStarted/StoppedSpeakingFrame`). | +| `provider_vad` | Realtime provider speech events, converted to call ms (below). | +| `playout` | Local output-transport playout: LiveKit playback events, Pipecat output audio write timestamps. | +| `span_clock` | Local span times that could not be aligned to any of the above, including provider offsets with an unknown input-buffer start. | + +When audio is not captured, `audio.selections` is absent and `timing_source` must not be `audio_samples`. Selections never extend outside the span bounds. + +**Converting realtime provider offsets to call ms.** Providers report `audio_start_ms`/`audio_end_ms` relative to the start of the audio the integration streamed to them. The conversion is: + +- call ms = `input_start_call_ms` + provider ms, where `input_start_call_ms` is the call ms at which the integration began appending input audio, on the recording clock. +- If that start is unknown (for example after a reconnect), use local span times and `timing_source: span_clock`. **Absence rules:** - An omitted field means "not observed". Integrations never write `null` or placeholder text. -- With no transcript, both `transcript` and `output` are absent. +- With no transcript, both `transcript` and `output.text` are absent. - When content capture is off, `output` is absent but `transcript.source` stays, so readers know a transcript existed. +### Creating and updating speech spans + +- **Creation:** integrations buffer an utterance and create its span when it ends. They pass explicit start and end times and the parent turn (or the call span for ignored speech). This avoids open spans for speech that turns out to be ignored or to merge into a turn. +- **Late links:** `heard_by` and `interrupted_by` may be known only after the end. + - Recommended: delay the end write until the first LLM span of the reply starts, up to a timeout (for example 2 s). Then most utterances need a single write. + - Otherwise: write again later. +- **Every write carries the complete current `voice.speech` object.** Braintrust merges metadata objects but replaces arrays, so a partial write would drop earlier links. + ### IDs and relationships ```text -call span ──audio.recordings──► recording segments +call span (voice.call) ──audio.recordings──► recording segments ├─ user_turn (turn.id = own span_id) - │ └─ speech {speaker:user, turn_id, heard_by:[llm span ids]} ──audio.selections──► channel 0 - └─ assistant_turn (turn.reply_to = user_turn) - ├─ llm_response ◄──produced_by── speech {speaker:agent, interrupted_by:user speech} - ├─ (tool_call_id) - └─ speech ──audio.selections──► channel 1 + │ └─ speech {speaker:user, turn_id, heard_by:[llm span ids]} ──audio.selections (1+ fragments)──► channel 0 + ├─ assistant_turn (turn.reply_to = user_turn) + │ ├─ llm_response ◄──produced_by── speech {speaker:agent, interrupted_by:user speech} + │ ├─ (tool_call_id) + │ └─ speech ──audio.selections──► channel 1 + └─ speech {speaker:user, state:ignored} (no turn_id) ``` -- All links are `span_id`s in the same trace; Lingua maps them to row ids. -- A link may name a span that ends later than the speech span. Integrations update the earlier span with a merge write; see "late links" below. +- All links are Braintrust `span_id`s in the same trace; Lingua maps them to row ids. +- Framework ids (LiveKit `speech_id`, Pipecat TTS `context_id`) are only used inside the integration to find those spans. ## What each integration captures and when -| Value | LiveKit cascaded (JS) | LiveKit realtime (JS) | Pipecat cascaded (Python) | Pipecat realtime (Python) | -| ---------------------- | -------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | ----------------------------------------------- | -| User speech start/end | Native `user_speaking` span, clipped to captured input samples at its end (`processor.ts` `onEnd`) | Realtime `input_speech_started`/`stopped` (`realtime.ts` turn tracker) | `UserStartedSpeakingFrame`/`UserStoppedSpeakingFrame` | Same frames, or the service's server-VAD events | -| `end_of_turn_ms` | `AgentActivity.onEndOfTurn` (`EndOfTurnInfo`) | Commit of the input buffer (server VAD) | Turn analyzer stop (`UserStoppedSpeakingFrame` after the end-of-turn decision) | Server VAD commit | -| User transcript | `user_turn` transcript plus `transcription_delay` → `available_ms` | `InputTranscriptionCompleted` → `provider_transcription` | `TranscriptionFrame` (`finalized`) | The service's input transcription frames | -| `heard_by` | The `llm_node` started under the `agent_turn` whose `turn.reply_to` is this turn | The `realtime_inference` started after the commit | The `llm_response` opened by the next `LLMContextFrame` that contains the transcript | The response span after the commit | -| Agent speech start/end | Native `agent_speaking`, placed by `playbackProgressed`/`playbackFinished` | Same | `BotStartedSpeakingFrame`/`BotStoppedSpeakingFrame` at the output transport | Same | -| `produced_by` | The `llm_node` with the same `contrib.livekit.speech_id` | The `realtime_inference` with the same `speech_id`/`response_id` | The `llm_response` whose TTS `context_id` matches | The response that emitted the audio | -| `state`, `played_ms` | `playbackFinished.interrupted`, `playbackPosition` | Same, plus `provider_played_ms` from the truncate call | `InterruptionFrame` while the bot is speaking; position from bot-speaking frames | Same, plus truncation from the service | -| `interrupted_by` | The user speech span active when playout stopped | Same | Same | Same | -| Agent text | `synchronizedTranscript` → `playback_sync`; `tts` input → `planned_text` | Response transcript → `model_output` | `TTSTextFrame`s observed while speaking → `playback_sync` | The service's text frames → `model_output` | +| Value | LiveKit cascaded (JS) | LiveKit realtime (JS) | Pipecat cascaded (Python) | Pipecat realtime (Python) | +| ----------------------------------- | -------------------------------------------------------------------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | +| Hook | `LiveKitSpanProcessor` (`processor.ts`) plus playback hooks | Same plus `realtime.ts` session events | A `BaseObserver` added to the `PipelineTask` | Same | +| User fragments (`audio.selections`) | Each native `user_speaking` span, clipped to captured input samples | Realtime `input_speech_started`/`stopped` | `UserStartedSpeakingFrame` → `UserStoppedSpeakingFrame` (or VAD frames) | Service speech events converted to call ms, else the same frames | +| User span created | When the `user_turn` closes: first fragment start to last fragment end | When the SDK-built turn closes | When the turn analyzer ends the user turn | Same | +| `end_of_turn_ms` | `AgentActivity.onEndOfTurn` (`EndOfTurnInfo`) | Commit of the input buffer (server VAD) | Turn analyzer stop | Server VAD commit | +| User transcript | `user_turn` transcript; `transcription_delay` → `available_ms` | `InputTranscriptionCompleted` → `provider_transcription` | `TranscriptionFrame` (`finalized`) | The service's input transcription frames | +| `heard_by` | The `llm_node` started under the `agent_turn` whose `turn.reply_to` is this turn | The `realtime_inference` started after the commit | The `llm_response` opened by the next `LLMContextFrame` containing the transcript | The response span after the commit | +| Agent playout (`audio.selections`) | `playbackStarted`/`playbackProgressed`/`playbackFinished` per `SpeechHandle` | Same | Output-transport audio-write timestamps between `BotStartedSpeakingFrame` and `BotStoppedSpeakingFrame` | Same | +| `produced_by` | The `llm_node` with the same `speech_id` | The `realtime_inference` with the same `speech_id`/`response_id` | The `llm_response` whose TTS `context_id` matches | The response that emitted the audio | +| `state`, `played_ms` | `playbackFinished.interrupted`, `playbackPosition` | Same, plus `provider_played_ms` from the truncate call | `InterruptionFrame` while speaking; bytes written to the output transport | Same, plus the service's truncation | +| `interrupted_by` | The user speech whose fragment was active when playout stopped | Same | Same | Same | +| Agent text | `synchronizedTranscript` → `playback_sync`; `tts` input → `planned_text` | Response transcript → `model_output` | `TTSTextFrame`s written while speaking → `playback_sync` | The service's text frames → `model_output` | ## What Lingua records and what it infers @@ -113,37 +161,48 @@ call span ──audio.recordings──► recording segments - the speaker, state, timing source and transcript source; - `interrupted_by`; -- `end_of_turn_time` (origin + `end_of_turn_ms`); +- `end_of_turn_time` (`voice.call.origin_unix_ms` + `end_of_turn_ms`); +- `start_time` and `end_time` (the span bounds); - the selections; - `step_ids`, taken from `produced_by` for agent speech and `heard_by` for user speech. **Inferred:** - `Speech.interrupted` (true when `state` is `interrupted`, false for other agent speech); -- the call each speech belongs to (nearest ancestor with recordings); -- speech order (by audio start), and overlap from the selections; +- the call each speech belongs to (the nearest ancestor with `voice.call` or recordings; the parent may be a turn or the call); +- speech order (by audio start, else span start), and overlap from the selections; - `step_ids` falling back to the turn span and its step children when no link was logged. Latency is not stored. The app computes it from `end_of_turn_time`, step times and the next agent speech start. -**Compatibility:** traces with no `voice.speech` keep #486's name-based behaviour unchanged. +**Rollout precedence:** + +- `voice.speech` replaces the LiveKit name rules per trace and speaker. If a trace has any user speech span, LiveKit `user_turn` speech is skipped for that trace, and the same holds for agents. +- So an integration can move one speaker at a time. +- Traces with no `voice.speech` keep #486's behaviour unchanged. ## Trajectory and app mapping -`Trajectory.voice_calls[].speech[]` gains five optional fields: +`Trajectory.voice_calls[].speech[]` gains seven optional fields: - `state` - `interrupted_by` - `transcript_source` - `timing_source` - `end_of_turn_time` +- `start_time` +- `end_time` + +A `VoiceCall` may have no recordings when audio capture was off. -The existing fields keep their meaning, so #22034 and #22036 work unchanged: +The existing fields keep their meaning, so #22034 and #22036 work unchanged when there is audio: - **Voice bar:** windows come from `selections`. - **Thread view:** clicking a window or "Play from this message" resolves `step_ids` to a turn's `request_id`, `work[].id` or `response_id`. -The new fields allow, without further schema work: +Without audio, the voice bar has nothing to play. A timeline would use `start_time`/`end_time`. + +The new fields allow: - an "interrupted by …" marker; - a "no transcript" badge when `transcript_source` is absent; @@ -152,57 +211,66 @@ The new fields allow, without further schema work: ## Worked example -`examples/cascaded.json` and `examples/realtime.json` encode the same fictional call: a booking change with `find_slots`, then a barge-in, then `reschedule`. Times are call ms. +`examples/cascaded.json` and `examples/realtime.json` encode the same fictional call: a booking change with `find_slots`, then a barge-in, then `reschedule`, with a backchannel near the end. Times are call ms. -| | Cascaded | Realtime | -| ------------------------------- | ------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | -| Steps under `assistant_turn` 1 | `l1` llm (tool call) → `find_slots` → `l2` llm → `tts` | `r1` realtime response (tool call) → `find_slots` → `r2` response with audio | -| User speech `s1` | `timing_source: vad`; `transcript {stt, available_ms: 3450, confidence: 0.94}`; `heard_by: [l1]` | `timing_source: provider_vad`; `transcript {provider_transcription, available_ms: 4300}`, which arrived after the tool call at 4100; `heard_by: [r1]` | -| Agent speech `s2` | 5300–7900; `played_ms: 2600`; `produced_by: l2`; `output.text` is what was spoken, `planned_text` the full sentence | 5000–7900; `played_ms: 2900` but `provider_played_ms: 2600`, because the local output buffer kept playing for 300 ms after the truncate | -| Barge-in `s3` | `stt` transcript "Nine thirty works." | No transcript: `transcript` and `output` are absent, which the voice view can show as missing | -| End of turn → first agent audio | 3400 → 5300 (1.9 s), split across STT, LLM, tool and TTS spans | 3400 → 5000 (1.6 s), with no STT or TTS spans to split it | +| | Cascaded | Realtime | +| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Steps under `assistant_turn` 1 | `l1` llm (tool call) → `find_slots` → `l2` llm → `tts` | `r1` realtime response (tool call) → `find_slots` → `r2` response with audio | +| User speech `s1` | Two fragments (300–700 "Hi,", 1000–2900); `timing_source: vad`; `transcript {stt, available_ms: 3450, confidence: 0.94}`; `heard_by: [l1]` | Same fragments; `timing_source: provider_vad`; `transcript {provider_transcription, available_ms: 4300}`, which arrived after the tool call at 4100; `heard_by: [r1]` | +| Agent speech `s2` | 5300–7900; `played_ms: 2600`; `produced_by: l2`; `output.text` is what was spoken, `planned_text` the full sentence | 5000–7900; `played_ms: 2900` but `provider_played_ms: 2600`, because the local output buffer kept playing for 300 ms after the truncate | +| Barge-in `s3` | `stt` transcript "Nine thirty works." | No transcript: `transcript` and `output` are absent, which the voice view can show as missing | +| Backchannel `s5` (10600–10900) | `state: ignored`, parent is the call span, transcript "Mm-hm." | `state: ignored`, no transcript | +| End of turn → first agent audio | 3400 → 5300 (1.9 s), split across STT, LLM, tool and TTS spans | 3400 → 5000 (1.6 s), with no STT or TTS spans to split it | -Lingua output for both files is checked in the companion PR's fixtures. +The companion PR checks Lingua's output for both files, plus a cascaded copy with no audio, in its fixtures. The realtime fixture shows one weakness. Because user turn 2 has no transcript, Lingua's turn builder merges both exchanges into one Trajectory turn. `voice_calls` is still correct, but the thread view shows one long turn. ## Customer usefulness -| Question | How well this option answers it | -| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | -| What was said | Good. One span per utterance, with text and its provenance. Planned and spoken agent text are kept apart. | -| Play matching audio | Good. Each speech span has exact selections. | -| Diagnose delays | Partial. End of turn and audio start are explicit, but the pipeline breakdown still comes from LLM, TTS and STT spans, and realtime has no breakdown. | -| Interruptions and overlap | Good. `state`, `interrupted_by`, `played_ms` and `provider_played_ms` are explicit; overlap shows in the selections. | -| Follow tools | Indirect. Speech links to LLM steps, and tools sit beside them in the turn. Nothing says that a tool ran while the user was waiting in silence. | -| Missing or uncertain evidence | Good per utterance: `timing_source`, transcript `final`/`source`, and absent fields. There is no call-level summary of gaps. | +| Question | How well this option answers it | +| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| What was said | Good. One span per utterance, with text and its provenance. Planned and spoken agent text are kept apart. | +| Play matching audio | Good. Each speech span has exact selections, one per fragment. | +| Diagnose delays | Partial. End of turn and audio start are explicit, but the pipeline breakdown still comes from LLM, TTS and STT spans, and realtime has no breakdown. | +| Interruptions and overlap | Good. `state`, `interrupted_by`, `played_ms` and `provider_played_ms` are explicit; overlap shows in the selections; backchannels are kept as ignored speech. | +| Follow tools | Indirect. Speech links to LLM steps, and tools sit beside them in the turn. Nothing says that a tool ran while the user was waiting in silence. | +| Missing or uncertain evidence | Good per utterance: `timing_source`, transcript `final`/`source`, and absent fields. There is no call-level summary of gaps. | ## Edge cases and weaknesses - **Barge-in truncation.** Covered by `played_ms` and `planned_text`. The thread still shows the LLM's full planned answer as the turn response, because Lingua's `response_text` comes from the LLM span. +- **Interrupted before any audio.** No speech span is written. The voice view then cannot show that the agent tried to speak; only the LLM span shows it. - **Overlap.** Two speech spans overlap in time; the app already prefers the later-starting window (#22036). -- **Late or missing realtime transcripts.** The integration must update `output` and `transcript` after the span ends, or leave them absent. A missing transcript splits turns badly (above). A fix would let a user speech with `heard_by` start a new turn without text; that is sketched, not implemented. -- **Failed or omitted segments.** Selections only point at ready segments, so the speech keeps its timing but has no playable audio. Lingua still lists it, with empty `selections`. -- **Unknown clock offset.** If the recording started late or the origin is wrong, every speech span is wrong by the same amount, and nothing in this option detects it. `timing_source: span_clock` marks only spans that were never aligned to audio. +- **Late or missing realtime transcripts.** Covered by the full-object rewrite above, or the transcript stays absent. + - A missing transcript splits turns badly (above). + - A fix would let user speech with `heard_by` start a new turn without text. That is sketched, not implemented. +- **Failed or omitted segments.** Selections only point at ready segments. The speech keeps its span bounds (`start_time`/`end_time`) but has no playable audio. +- **Audio capture off.** The call still exists through `voice.call`, and speech has bounds but no selections. Pipecat and LiveKit must still log `voice.call.origin_unix_ms`. +- **Unknown clock offset.** If the recording started late or the origin is wrong, every speech span is wrong by the same amount, and nothing in this option detects it. `timing_source: span_clock` marks only spans that were never aligned. - **Call in progress.** Speech whose call row isn't logged yet is skipped (#486 behaviour). `incomplete` is set only at shutdown. - **Content capture off.** Text disappears but every link and timing field stays. - **Tool cancelled by interruption.** The tool span has the error. The speech has no direct link to it; readers infer it from the turn. - **Agent handoff.** Speech stays under the turn of the agent that spoke. `voice.call` has no field for the active agent; it would have to come from turn or agent metadata. -- **Late links.** `heard_by` and `interrupted_by` are known only after the speech span ends. Integrations need a second, merge `log()` on an already-ended span, which Braintrust allows but which costs one extra row write per utterance. - **Strict enums.** Lingua rejects unknown `state`, `timing_source` and `transcript.source` values (the whole `voice.speech` key fails). New values need a Lingua release first. ## Implementation starting point 1. **LiveKit (`js/src/instrumentation/livekit/processor.ts`):** - - Rename the exported `livekit.user_speaking`/`livekit.agent_speaking` operations to `speech`, and log `voice.speech` at `onEnd` from the data already tracked (`speakingByTurn`, playback events, `EndOfTurnInfo`). - - Log `voice.call` on `agent_session` start. - - Add `heard_by` when an `llm_node`/`realtime_inference` span starts under a turn with `turn.reply_to`. -2. **Realtime (`realtime.ts`):** create user speech spans for SDK-built turns. Today only the turn is created and `user_speaking` is re-parented under it. -3. **Pipecat (`sdk-python/py/src/braintrust/integrations/pipecat`, `delner/pipecat-voice-recording`):** emit `speech` spans from the speaking and bot-speaking frames listed above. -4. **Lingua:** done in the companion PR. Then add `voice.speech` to `TRAJECTORY_METADATA_FIELDS` after #486 is rebased onto main. + - Log `voice.call` (including `origin_unix_ms`) on `agent_session` start, even with audio off. + - Stop exporting `livekit.user_speaking`/`livekit.agent_speaking` as rows. Instead collect their intervals per turn (`speakingByTurn`) and per `SpeechHandle`. + - When the user turn closes or playout finishes, create one `speech` span with explicit times, merged selections and `voice.speech`. + - Add `heard_by` by waiting for the first `llm_node`/`realtime_inference` under the replying turn. +2. **Realtime (`realtime.ts`):** build the user speech span from the SDK-built turn's fragments. Convert provider offsets with the input-start call ms. +3. **Pipecat (`sdk-python/py/src/braintrust/integrations/pipecat` on `delner/pipecat-voice-recording`):** + - Add a `BaseObserver` to the `PipelineTask`. + - Keep `user_turn`/`assistant_turn` names. + - Timestamp output-transport audio writes for agent playout and `played_ms`. + - Emit `speech` spans and the links as Braintrust span ids. +4. **Lingua:** done in the companion PR. Then add `voice.speech` and `voice.call` to `TRAJECTORY_METADATA_FIELDS` after #486 is rebased onto main. ## Open questions -- Should user speech that LiveKit merges into one turn (pause ≤ `maxPauseMs`) be one speech span or several? This draft logs one per VAD segment. -- Should `ignored` speech create a Trajectory turn? -- Is a per-utterance late write acceptable for the ingestion volume of long calls? +- Should `ignored` speech create a Trajectory turn, or only appear in `voice_calls`? +- Is the timeout for delaying the end write (2 s here) right for slow LLM starts, given the second write it otherwise costs? +- Should an interrupted speech handle that played nothing leave a zero-length marker so the voice view can show the attempt? diff --git a/docs/voice-schema/speech-spans/examples/cascaded.json b/docs/voice-schema/speech-spans/examples/cascaded.json index 5715b8a6a..a2194bfcb 100644 --- a/docs/voice-schema/speech-spans/examples/cascaded.json +++ b/docs/voice-schema/speech-spans/examples/cascaded.json @@ -25,7 +25,8 @@ "metadata": { "voice.call": { "framework": "livekit", - "pipeline": "cascaded" + "pipeline": "cascaded", + "origin_unix_ms": 1791830000000 }, "audio.recordings": [ { @@ -118,6 +119,13 @@ "recording_span_id": "call", "recording_id": "call-0000", "start_offset_ms": 300, + "end_offset_ms": 700, + "channel_index": 0 + }, + { + "recording_span_id": "call", + "recording_id": "call-0000", + "start_offset_ms": 1000, "end_offset_ms": 2900, "channel_index": 0 } @@ -660,5 +668,44 @@ } ] } + }, + { + "id": "r-s5", + "span_id": "s5", + "root_span_id": "call", + "span_parents": ["call"], + "span_attributes": { + "name": "speech", + "type": "task" + }, + "metrics": { + "start": 1791830010.6, + "end": 1791830010.9 + }, + "output": { + "text": "Mm-hm." + }, + "metadata": { + "voice.speech": { + "speaker": "user", + "state": "ignored", + "timing_source": "vad", + "transcript": { + "source": "stt", + "final": true, + "available_ms": 11000, + "confidence": 0.71 + } + }, + "audio.selections": [ + { + "recording_span_id": "call", + "recording_id": "call-0000", + "start_offset_ms": 10600, + "end_offset_ms": 10900, + "channel_index": 0 + } + ] + } } ] diff --git a/docs/voice-schema/speech-spans/examples/realtime.json b/docs/voice-schema/speech-spans/examples/realtime.json index 937185e03..d268a712e 100644 --- a/docs/voice-schema/speech-spans/examples/realtime.json +++ b/docs/voice-schema/speech-spans/examples/realtime.json @@ -25,7 +25,8 @@ "metadata": { "voice.call": { "framework": "livekit", - "pipeline": "realtime" + "pipeline": "realtime", + "origin_unix_ms": 1791830000000 }, "audio.recordings": [ { @@ -117,6 +118,13 @@ "recording_span_id": "call", "recording_id": "call-0000", "start_offset_ms": 300, + "end_offset_ms": 700, + "channel_index": 0 + }, + { + "recording_span_id": "call", + "recording_id": "call-0000", + "start_offset_ms": 1000, "end_offset_ms": 2900, "channel_index": 0 } @@ -604,5 +612,35 @@ } ] } + }, + { + "id": "r-s5", + "span_id": "s5", + "root_span_id": "call", + "span_parents": ["call"], + "span_attributes": { + "name": "speech", + "type": "task" + }, + "metrics": { + "start": 1791830010.6, + "end": 1791830010.9 + }, + "metadata": { + "voice.speech": { + "speaker": "user", + "state": "ignored", + "timing_source": "provider_vad" + }, + "audio.selections": [ + { + "recording_span_id": "call", + "recording_id": "call-0000", + "start_offset_ms": 10600, + "end_offset_ms": 10900, + "channel_index": 0 + } + ] + } } ] diff --git a/js/src/instrumentation/livekit/voice-schema/speech-spans.test.ts b/js/src/instrumentation/livekit/voice-schema/speech-spans.test.ts index 05199f36a..5042ad020 100644 --- a/js/src/instrumentation/livekit/voice-schema/speech-spans.test.ts +++ b/js/src/instrumentation/livekit/voice-schema/speech-spans.test.ts @@ -2,7 +2,7 @@ import { readFileSync } from "node:fs"; import path from "node:path"; import { describe, expect, test } from "vitest"; import { z } from "zod/v3"; -import { speechSpan, voiceCall } from "./speech-spans"; +import { speechSpan, voiceCall, voiceSpeech } from "./speech-spans"; type Row = { span_id: string; @@ -25,16 +25,23 @@ describe.each(["cascaded", "realtime"])("%s example", (name) => { .filter((row) => row.metadata?.["voice.speech"]) .map((row) => speechSpan.parse(row)); - test("the call span declares its pipeline", () => { - const call = rows.find((row) => row.span_parents === null); - expect(voiceCall.parse(call?.metadata?.["voice.call"]).pipeline).toBe(name); + const call = rows.find((row) => row.span_parents === null); + const callInfo = voiceCall.parse(call?.metadata?.["voice.call"]); + + test("the call span declares its pipeline and clock", () => { + expect(callInfo.pipeline).toBe(name); + const [recording] = z + .array(z.object({ timeline: z.object({ origin_unix_ms: z.number() }) })) + .parse(call?.metadata?.["audio.recordings"]); + expect(recording.timeline.origin_unix_ms).toBe(callInfo.origin_unix_ms); }); test("speech links resolve to spans of the right kind", () => { - expect(speech).toHaveLength(4); + expect(speech).toHaveLength(5); for (const span of speech) { const voice = span.metadata["voice.speech"]; - if (voice.turn_id) expect(span.span_parents).toEqual([voice.turn_id]); + expect(span.span_parents).toEqual([voice.turn_id ?? call?.span_id]); + if (voice.state === "ignored") expect(voice.turn_id).toBeUndefined(); for (const id of voice.heard_by ?? []) expect(byId.get(id)?.span_attributes.type).toBe("llm"); if (voice.produced_by) @@ -52,23 +59,50 @@ describe.each(["cascaded", "realtime"])("%s example", (name) => { } }); - test("speech span times match their audio selections", () => { - const [recording] = z - .array(z.object({ timeline: z.object({ origin_unix_ms: z.number() }) })) - .parse( - rows.find((row) => row.span_parents === null)?.metadata?.[ - "audio.recordings" - ], - ); - const origin = recording.timeline.origin_unix_ms; + test("speech spans run from the first to the last audio fragment", () => { for (const span of speech) { - const [selection] = span.metadata["audio.selections"] ?? []; - expect(span.metrics.start * 1000 - origin).toBeCloseTo( - selection.start_offset_ms, + const selections = span.metadata["audio.selections"] ?? []; + expect(span.metrics.start * 1000 - callInfo.origin_unix_ms).toBeCloseTo( + selections[0].start_offset_ms, ); - expect(span.metrics.end * 1000 - origin).toBeCloseTo( - selection.end_offset_ms, + expect(span.metrics.end * 1000 - callInfo.origin_unix_ms).toBeCloseTo( + selections[selections.length - 1].end_offset_ms, ); } + expect( + speech.find((span) => span.span_id === "s1")?.metadata[ + "audio.selections" + ], + ).toHaveLength(2); }); }); + +test("speech without audio cannot claim audio sample timing", () => { + const span = { + span_id: "s", + span_parents: ["call"], + span_attributes: { name: "speech", type: "task" }, + metrics: { start: 1, end: 2 }, + metadata: { + "voice.speech": { + speaker: "user", + state: "ignored", + timing_source: "audio_samples", + }, + }, + }; + expect(speechSpan.safeParse(span).success).toBe(false); + span.metadata["voice.speech"].timing_source = "vad"; + expect(speechSpan.safeParse(span).success).toBe(true); +}); + +test("agent speech that played nothing is not a speech span", () => { + expect( + voiceSpeech.safeParse({ + speaker: "agent", + state: "interrupted", + timing_source: "playout", + played_ms: 0, + }).success, + ).toBe(false); +}); diff --git a/js/src/instrumentation/livekit/voice-schema/speech-spans.ts b/js/src/instrumentation/livekit/voice-schema/speech-spans.ts index d7464012a..a0e3ba789 100644 --- a/js/src/instrumentation/livekit/voice-schema/speech-spans.ts +++ b/js/src/instrumentation/livekit/voice-schema/speech-spans.ts @@ -13,6 +13,7 @@ const selection = z.object({ export const voiceCall = z.object({ framework: z.enum(["livekit", "pipecat"]), pipeline: z.enum(["cascaded", "realtime", "half_cascade"]), + origin_unix_ms: z.number().nonnegative(), }); export const voiceSpeech = z @@ -30,7 +31,7 @@ export const voiceSpeech = z heard_by: z.array(z.string()).optional(), produced_by: z.string().optional(), interrupted_by: z.string().optional(), - played_ms: callMs.optional(), + played_ms: z.number().positive().optional(), provider_played_ms: callMs.optional(), end_of_turn_ms: callMs.optional(), transcript: z @@ -71,22 +72,37 @@ export const voiceSpeech = z }); }); -export const speechSpan = z.object({ - span_id: z.string(), - span_parents: z.array(z.string()).length(1), - span_attributes: z.object({ - name: z.literal("speech"), - type: z.literal("task"), - }), - metrics: z.object({ start: z.number(), end: z.number() }), - output: z - .object({ text: z.string(), planned_text: z.string().optional() }) - .optional(), - metadata: z.object({ - "voice.speech": voiceSpeech, - "audio.selections": z.array(selection).optional(), - }), -}); +export const speechSpan = z + .object({ + span_id: z.string(), + span_parents: z.array(z.string()).length(1), + span_attributes: z.object({ + name: z.literal("speech"), + type: z.literal("task"), + }), + metrics: z.object({ start: z.number(), end: z.number() }), + output: z + .object({ + text: z.string().optional(), + planned_text: z.string().optional(), + }) + .optional(), + metadata: z.object({ + "voice.speech": voiceSpeech, + "audio.selections": z.array(selection).min(1).optional(), + }), + }) + .superRefine((span, ctx) => { + if ( + span.metadata["voice.speech"].timing_source === "audio_samples" && + !span.metadata["audio.selections"] + ) + ctx.addIssue({ + code: z.ZodIssueCode.custom, + message: "audio_samples timing requires audio.selections", + path: ["metadata", "voice.speech", "timing_source"], + }); + }); export type VoiceSpeech = z.infer; export type SpeechSpan = z.infer; From b3c971dd9e938e935894d7da0f43eb41387bb60a Mon Sep 17 00:00:00 2001 From: kevchoi <16812845+kevchoi@users.noreply.github.com> Date: Mon, 12 Oct 2026 17:28:18 +0000 Subject: [PATCH 3/8] chore: specify realtime spoken text for speech spans Co-Authored-By: Claude Opus 5.5 --- docs/voice-schema/speech-spans/README.md | 84 ++++++++++--------- .../speech-spans/examples/realtime.json | 6 +- .../livekit/voice-schema/speech-spans.test.ts | 23 +++++ .../livekit/voice-schema/speech-spans.ts | 11 +++ 4 files changed, 82 insertions(+), 42 deletions(-) diff --git a/docs/voice-schema/speech-spans/README.md b/docs/voice-schema/speech-spans/README.md index de9af2e3b..8ff788234 100644 --- a/docs/voice-schema/speech-spans/README.md +++ b/docs/voice-schema/speech-spans/README.md @@ -35,13 +35,13 @@ They also need to see where the trace lacks evidence. Today Lingua finds speech This is the root span: LiveKit `livekit.agent_session`, Pipecat `pipecat.pipeline`. Lingua finds the call through `voice.call`, so a call without audio still has speech. -| Field | Type | Required | Meaning | -| ------------------------------ | -------------------------------------------- | ---------------------- | --------------------------------------------------------------------------------------------------------------------- | -| `input.audio[]` | attachment | When audio is captured | Unchanged (`@braintrust/audio`). | -| `metadata["audio.recordings"]` | descriptor[] | When audio is captured | Unchanged. Channel 0 is the caller, channel 1 the agent. | -| `voice.call.framework` | `"livekit" \| "pipecat"` | Yes | The integration that emitted the call. | -| `voice.call.pipeline` | `"cascaded" \| "realtime" \| "half_cascade"` | Yes | Tells readers which speech evidence to expect. `half_cascade` is a realtime model with text output plus separate TTS. | -| `voice.call.origin_unix_ms` | Unix ms | Yes | The clock for every call-ms field. Logged at session start, whether or not audio is captured. | +| Field | Type | Required | Meaning | +| ------------------------------ | -------------------------------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `input.audio[]` | attachment | When audio is captured | Unchanged (`@braintrust/audio`). | +| `metadata["audio.recordings"]` | descriptor[] | When audio is captured | Unchanged. Channel 0 is the caller, channel 1 the agent. | +| `voice.call.framework` | `"livekit" \| "pipecat"` | Yes | The integration that emitted the call. | +| `voice.call.pipeline` | `"cascaded" \| "realtime" \| "half_cascade"` | Yes | Tells readers which speech evidence to expect. `half_cascade` is a realtime model with text output plus separate TTS: user speech follows the realtime column below, agent speech the cascaded column. | +| `voice.call.origin_unix_ms` | Unix ms | Yes | The clock for every call-ms field. Logged at session start, whether or not audio is captured. | ### Turn spans @@ -58,11 +58,11 @@ Each utterance gets one span with `span_attributes: {type: "task", name: "speech **Utterance grain:** -| Speaker | One speech span per… | How fragments are kept | -| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | -| User | Turn: everything the user said that the turn detector grouped into one user turn | Each VAD fragment becomes one `audio.selections` range on that span. | -| User, not part of a turn (backchannel, false interruption) | Ignored utterance | One span with `state: "ignored"`. | -| Agent | Playout of one speech handle: LiveKit `SpeechHandle`, Pipecat `BotStartedSpeakingFrame` → `BotStoppedSpeakingFrame` | Pause and resume after a false interruption stays one span, with one selection per played stretch. | +| Speaker | One speech span per… | How fragments are kept | +| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| User | Turn: everything the user said that the turn detector grouped into one user turn | Each VAD fragment becomes one `audio.selections` range on that span. | +| User, not part of a turn (backchannel, false interruption) | Ignored utterance | One span with `state: "ignored"`. | +| Agent | Playout of one speech handle: LiveKit `SpeechHandle`, Pipecat `BotStartedSpeakingFrame` → `BotStoppedSpeakingFrame` | Pause and resume after a false interruption stays one span, with one selection per played stretch. Speech with no assistant turn (a greeting from `session.say` or a Pipecat `TTSSpeakFrame`) is parented to the call span without `turn_id`. | The grain matters because a cascaded call can produce several speaking fragments per user turn. Integrations must not emit one span per fragment. @@ -72,11 +72,11 @@ An interrupted speech handle that played no audio produces no speech span; its t | Field | Type | Required | Meaning | | -------------------------------------- | --------------------------------------------------------------------------------------- | ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `span_parents` | `[span id]` | Yes | The owning turn span. Ignored speech has the call span as its parent. | +| `span_parents` | `[span id]` | Yes | The owning turn span, or the call span for ignored speech and agent speech with no turn. | | `metrics.start` / `metrics.end` | Unix s | Yes | The utterance bounds. With selections: first selection start to last selection end. | | `metadata["audio.selections"]` | Selection[] | When audio is captured for that utterance | Unchanged shape. One channel: 0 for user speech, 1 for agent speech. One range per fragment or played stretch, in order. | | `voice.speech.speaker` | `"user" \| "agent"` | Yes | Who spoke. | -| `voice.speech.turn_id` | span id | When the parent is a turn | The owning turn span. Omitted for ignored speech. | +| `voice.speech.turn_id` | span id | When the parent is a turn | The owning turn span. Omitted when the parent is the call span. | | `voice.speech.state` | `"complete" \| "interrupted" \| "incomplete" \| "ignored"` | Yes | `interrupted` means agent playout was cut short after some audio played. `incomplete` means the call or process ended mid-utterance. `ignored` means user speech the agent heard but did not answer. | | `voice.speech.timing_source` | See below | Yes | The evidence behind `metrics.start`/`metrics.end`. | | `voice.speech.heard_by` | span id[] | User only, optional | The LLM spans whose input first contained this utterance. | @@ -85,12 +85,12 @@ An interrupted speech handle that played no audio produces no speech span; its t | `voice.speech.played_ms` | ms > 0 | Agent only, optional | Audio actually played locally: bytes written to the output transport divided by its byte rate, or the framework's playback position. | | `voice.speech.provider_played_ms` | ms | Agent realtime only, optional | What the provider was told had played (`conversation.item.truncate` `audio_end_ms`). | | `voice.speech.end_of_turn_ms` | call ms | User only, optional | When the agent decided the user had finished. | -| `voice.speech.transcript.source` | `"stt" \| "provider_transcription" \| "tts_input" \| "model_output" \| "playback_sync"` | With `transcript` | Where the text came from. | +| `voice.speech.transcript.source` | `"stt" \| "provider_transcription" \| "tts_input" \| "model_output" \| "playback_sync"` | With `transcript` | Where the text came from. For agent speech, `playback_sync` describes `output.text`, and `tts_input`/`model_output` describe `planned_text` only. | | `voice.speech.transcript.final` | boolean | With `transcript` | `false` when only partial text arrived before the span was written. | | `voice.speech.transcript.available_ms` | call ms | No | When the text became available. In realtime this is often after the response started. | | `voice.speech.transcript.confidence` | 0–1 | No | STT confidence, if the provider reports one. | -| `output.text` | string | No | The best transcript of what was actually said. Absent without a transcript or with content capture off. | -| `output.planned_text` | string | No | Agent text that was generated but not fully spoken. | +| `output.text` | string | No | What was actually said. User: the transcript. Agent: the text synchronized with playout (`playback_sync`), never the generated text. Absent without such a transcript or with content capture off. | +| `output.planned_text` | string | No | Agent only: the text the agent generated for this playout: the TTS input (cascaded) or the response transcript (realtime). | **`timing_source` values:** @@ -113,11 +113,15 @@ When audio is not captured, `audio.selections` is absent and `timing_source` mus - An omitted field means "not observed". Integrations never write `null` or placeholder text. - With no transcript, both `transcript` and `output.text` are absent. +- **Agent spoken text:** + - Spoken text comes only from playout-synchronized text: LiveKit `playbackFinished.synchronizedTranscript` (content mode), or Pipecat text frames observed while the bot is speaking, cut at the `InterruptionFrame`. + - This holds in both cascaded and realtime pipelines. Realtime response transcripts hold the full planned text, and an OpenAI `conversation.item.truncate` deletes the server-side transcript, so neither can stand in for what was spoken. + - Without synchronized text, `output.text` is absent, `planned_text` is kept, and `transcript.source` is `tts_input` (cascaded) or `model_output` (realtime). That source describes `planned_text` only. - When content capture is off, `output` is absent but `transcript.source` stays, so readers know a transcript existed. ### Creating and updating speech spans -- **Creation:** integrations buffer an utterance and create its span when it ends. They pass explicit start and end times and the parent turn (or the call span for ignored speech). This avoids open spans for speech that turns out to be ignored or to merge into a turn. +- **Creation:** integrations buffer an utterance and create its span when it ends. They pass explicit start and end times and the parent turn (or the call span for ignored speech and agent speech with no turn). This avoids open spans for speech that turns out to be ignored or to merge into a turn. - **Late links:** `heard_by` and `interrupted_by` may be known only after the end. - Recommended: delay the end write until the first LLM span of the reply starts, up to a timeout (for example 2 s). Then most utterances need a single write. - Otherwise: write again later. @@ -141,19 +145,21 @@ call span (voice.call) ──audio.recordings──► recording segments ## What each integration captures and when -| Value | LiveKit cascaded (JS) | LiveKit realtime (JS) | Pipecat cascaded (Python) | Pipecat realtime (Python) | -| ----------------------------------- | -------------------------------------------------------------------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | -| Hook | `LiveKitSpanProcessor` (`processor.ts`) plus playback hooks | Same plus `realtime.ts` session events | A `BaseObserver` added to the `PipelineTask` | Same | -| User fragments (`audio.selections`) | Each native `user_speaking` span, clipped to captured input samples | Realtime `input_speech_started`/`stopped` | `UserStartedSpeakingFrame` → `UserStoppedSpeakingFrame` (or VAD frames) | Service speech events converted to call ms, else the same frames | -| User span created | When the `user_turn` closes: first fragment start to last fragment end | When the SDK-built turn closes | When the turn analyzer ends the user turn | Same | -| `end_of_turn_ms` | `AgentActivity.onEndOfTurn` (`EndOfTurnInfo`) | Commit of the input buffer (server VAD) | Turn analyzer stop | Server VAD commit | -| User transcript | `user_turn` transcript; `transcription_delay` → `available_ms` | `InputTranscriptionCompleted` → `provider_transcription` | `TranscriptionFrame` (`finalized`) | The service's input transcription frames | -| `heard_by` | The `llm_node` started under the `agent_turn` whose `turn.reply_to` is this turn | The `realtime_inference` started after the commit | The `llm_response` opened by the next `LLMContextFrame` containing the transcript | The response span after the commit | -| Agent playout (`audio.selections`) | `playbackStarted`/`playbackProgressed`/`playbackFinished` per `SpeechHandle` | Same | Output-transport audio-write timestamps between `BotStartedSpeakingFrame` and `BotStoppedSpeakingFrame` | Same | -| `produced_by` | The `llm_node` with the same `speech_id` | The `realtime_inference` with the same `speech_id`/`response_id` | The `llm_response` whose TTS `context_id` matches | The response that emitted the audio | -| `state`, `played_ms` | `playbackFinished.interrupted`, `playbackPosition` | Same, plus `provider_played_ms` from the truncate call | `InterruptionFrame` while speaking; bytes written to the output transport | Same, plus the service's truncation | -| `interrupted_by` | The user speech whose fragment was active when playout stopped | Same | Same | Same | -| Agent text | `synchronizedTranscript` → `playback_sync`; `tts` input → `planned_text` | Response transcript → `model_output` | `TTSTextFrame`s written while speaking → `playback_sync` | The service's text frames → `model_output` | +| Value | LiveKit cascaded (JS) | LiveKit realtime (JS) | Pipecat cascaded (Python) | Pipecat realtime (Python) | +| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | +| Hook | `LiveKitSpanProcessor` (`processor.ts`) plus playback hooks | Same plus `realtime.ts` session events | A `BaseObserver` added to the `PipelineTask` | Same | +| User fragments (`audio.selections`) | Each native `user_speaking` span, clipped to captured input samples | Realtime `input_speech_started`/`stopped` | `UserStartedSpeakingFrame` → `UserStoppedSpeakingFrame` (or VAD frames) | Service speech events converted to call ms, else the same frames | +| User span created | When the `user_turn` closes: first fragment start to last fragment end | When the SDK-built turn closes | When the turn analyzer ends the user turn | Same | +| Ignored user speech | A `user_speaking` interval with no committed user item after end of turn, or a false interruption (`agent_false_interruption`); written once LiveKit resumes or drops the turn | Speech events with no transcript item committed to the conversation; written when the next response starts or the call ends | `UserStoppedSpeakingFrame` with no user aggregation pushed to the LLM; written when the next user turn starts or the pipeline ends | Same | +| `end_of_turn_ms` | `AgentActivity.onEndOfTurn` (`EndOfTurnInfo`) | Commit of the input buffer (server VAD) | Turn analyzer stop | Server VAD commit | +| User transcript | `user_turn` transcript; `transcription_delay` → `available_ms` | `InputTranscriptionCompleted` → `provider_transcription` | `TranscriptionFrame` (`finalized`) | The service's input transcription frames | +| `heard_by` | The `llm_node` started under the `agent_turn` whose `turn.reply_to` is this turn | The `realtime_inference` started after the commit | The `llm_response` opened by the next `LLMContextFrame` containing the transcript | The response span after the commit | +| Agent playout (`audio.selections`) | `playbackStarted`/`playbackProgressed`/`playbackFinished` per `SpeechHandle` | Same | Output-transport audio-write timestamps between `BotStartedSpeakingFrame` and `BotStoppedSpeakingFrame` | Same | +| `produced_by` | The `llm_node` with the same `speech_id` | The `realtime_inference` with the same `speech_id`/`response_id` | The `llm_response` whose TTS `context_id` matches | The response that emitted the audio | +| `state`, `played_ms` | `playbackFinished.interrupted`, `playbackPosition` | Same, plus `provider_played_ms` from the truncate call | `InterruptionFrame` while speaking; bytes written to the output transport | Same, plus the service's truncation | +| `interrupted_by` | The user speech whose fragment was active when playout stopped | Same | Same | Same | +| Agent text | `synchronizedTranscript` → `output.text` (`playback_sync`); `tts` input → `planned_text` | `synchronizedTranscript` → `output.text` (`playback_sync`); response transcript → `planned_text` | `TTSTextFrame`s written while speaking, cut at `InterruptionFrame` → `output.text` (`playback_sync`); TTS input → `planned_text` | The service's text frames observed while the bot is speaking, cut at `InterruptionFrame` → `output.text`; full response text → `planned_text` | +| Agent speech without a turn | `session.say` greeting: parent is the call span, no `turn_id`, no `produced_by` | Same | `TTSSpeakFrame`: same | Same | ## What Lingua records and what it infers @@ -213,14 +219,14 @@ The new fields allow: `examples/cascaded.json` and `examples/realtime.json` encode the same fictional call: a booking change with `find_slots`, then a barge-in, then `reschedule`, with a backchannel near the end. Times are call ms. -| | Cascaded | Realtime | -| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Steps under `assistant_turn` 1 | `l1` llm (tool call) → `find_slots` → `l2` llm → `tts` | `r1` realtime response (tool call) → `find_slots` → `r2` response with audio | -| User speech `s1` | Two fragments (300–700 "Hi,", 1000–2900); `timing_source: vad`; `transcript {stt, available_ms: 3450, confidence: 0.94}`; `heard_by: [l1]` | Same fragments; `timing_source: provider_vad`; `transcript {provider_transcription, available_ms: 4300}`, which arrived after the tool call at 4100; `heard_by: [r1]` | -| Agent speech `s2` | 5300–7900; `played_ms: 2600`; `produced_by: l2`; `output.text` is what was spoken, `planned_text` the full sentence | 5000–7900; `played_ms: 2900` but `provider_played_ms: 2600`, because the local output buffer kept playing for 300 ms after the truncate | -| Barge-in `s3` | `stt` transcript "Nine thirty works." | No transcript: `transcript` and `output` are absent, which the voice view can show as missing | -| Backchannel `s5` (10600–10900) | `state: ignored`, parent is the call span, transcript "Mm-hm." | `state: ignored`, no transcript | -| End of turn → first agent audio | 3400 → 5300 (1.9 s), split across STT, LLM, tool and TTS spans | 3400 → 5000 (1.6 s), with no STT or TTS spans to split it | +| | Cascaded | Realtime | +| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Steps under `assistant_turn` 1 | `l1` llm (tool call) → `find_slots` → `l2` llm → `tts` | `r1` realtime response (tool call) → `find_slots` → `r2` response with audio | +| User speech `s1` | Two fragments (300–700 "Hi,", 1000–2900); `timing_source: vad`; `transcript {stt, available_ms: 3450, confidence: 0.94}`; `heard_by: [l1]` | Same fragments; `timing_source: provider_vad`; `transcript {provider_transcription, available_ms: 4300}`, which arrived after the tool call at 4100; `heard_by: [r1]` | +| Agent speech `s2` | 5300–7900; `played_ms: 2600`; `produced_by: l2`; `output.text` is the synchronized spoken text, `planned_text` the TTS input | 5000–7900; `played_ms: 2900` but `provider_played_ms: 2600`, because the local output buffer kept playing for 300 ms after the truncate. `output.text` is the synchronized spoken text (`playback_sync`), and `planned_text` is the full response transcript, which the `r2` response span also logs | +| Barge-in `s3` | `stt` transcript "Nine thirty works." | No transcript: `transcript` and `output` are absent, which the voice view can show as missing | +| Backchannel `s5` (10600–10900) | `state: ignored`, parent is the call span, transcript "Mm-hm." | `state: ignored`, no transcript | +| End of turn → first agent audio | 3400 → 5300 (1.9 s), split across STT, LLM, tool and TTS spans | 3400 → 5000 (1.6 s), with no STT or TTS spans to split it | The companion PR checks Lingua's output for both files, plus a cascaded copy with no audio, in its fixtures. diff --git a/docs/voice-schema/speech-spans/examples/realtime.json b/docs/voice-schema/speech-spans/examples/realtime.json index d268a712e..fc18d544c 100644 --- a/docs/voice-schema/speech-spans/examples/realtime.json +++ b/docs/voice-schema/speech-spans/examples/realtime.json @@ -272,7 +272,7 @@ "output": [ { "role": "assistant", - "content": "I have nine thirty or eleven on Fri" + "content": "I have nine thirty or eleven on Friday. Which would you like?" } ], "metadata": { @@ -309,7 +309,7 @@ "played_ms": 2900, "provider_played_ms": 2600, "transcript": { - "source": "model_output", + "source": "playback_sync", "final": true } }, @@ -598,7 +598,7 @@ "produced_by": "r4", "played_ms": 2800, "transcript": { - "source": "model_output", + "source": "playback_sync", "final": true } }, diff --git a/js/src/instrumentation/livekit/voice-schema/speech-spans.test.ts b/js/src/instrumentation/livekit/voice-schema/speech-spans.test.ts index 5042ad020..945b39fd8 100644 --- a/js/src/instrumentation/livekit/voice-schema/speech-spans.test.ts +++ b/js/src/instrumentation/livekit/voice-schema/speech-spans.test.ts @@ -106,3 +106,26 @@ test("agent speech that played nothing is not a speech span", () => { }).success, ).toBe(false); }); + +test("agent spoken text must come from playout-synchronized text", () => { + const span = { + span_id: "s", + span_parents: ["t"], + span_attributes: { name: "speech", type: "task" }, + metrics: { start: 1, end: 2 }, + output: { text: "I have", planned_text: "I have nine thirty." }, + metadata: { + "voice.speech": { + speaker: "agent", + turn_id: "t", + state: "interrupted", + timing_source: "playout", + played_ms: 500, + transcript: { source: "model_output", final: true }, + }, + }, + }; + expect(speechSpan.safeParse(span).success).toBe(false); + const { text: _text, ...planned } = span.output; + expect(speechSpan.safeParse({ ...span, output: planned }).success).toBe(true); +}); diff --git a/js/src/instrumentation/livekit/voice-schema/speech-spans.ts b/js/src/instrumentation/livekit/voice-schema/speech-spans.ts index a0e3ba789..bf1a8e58b 100644 --- a/js/src/instrumentation/livekit/voice-schema/speech-spans.ts +++ b/js/src/instrumentation/livekit/voice-schema/speech-spans.ts @@ -93,6 +93,17 @@ export const speechSpan = z }), }) .superRefine((span, ctx) => { + const speech = span.metadata["voice.speech"]; + if ( + speech.speaker === "agent" && + span.output?.text !== undefined && + speech.transcript?.source !== "playback_sync" + ) + ctx.addIssue({ + code: z.ZodIssueCode.custom, + message: "agent output.text must be playout-synchronized text", + path: ["metadata", "voice.speech", "transcript", "source"], + }); if ( span.metadata["voice.speech"].timing_source === "audio_samples" && !span.metadata["audio.selections"] From e12b494ad6a793daed2ccca7af1a3c70c15ac158 Mon Sep 17 00:00:00 2001 From: kevchoi <16812845+kevchoi@users.noreply.github.com> Date: Mon, 12 Oct 2026 17:31:50 +0000 Subject: [PATCH 4/8] chore: tidy speech span naming Co-Authored-By: Claude Opus 5.5 --- docs/voice-schema/speech-spans/README.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/voice-schema/speech-spans/README.md b/docs/voice-schema/speech-spans/README.md index 8ff788234..18e1a412c 100644 --- a/docs/voice-schema/speech-spans/README.md +++ b/docs/voice-schema/speech-spans/README.md @@ -148,14 +148,15 @@ call span (voice.call) ──audio.recordings──► recording segments | Value | LiveKit cascaded (JS) | LiveKit realtime (JS) | Pipecat cascaded (Python) | Pipecat realtime (Python) | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | Hook | `LiveKitSpanProcessor` (`processor.ts`) plus playback hooks | Same plus `realtime.ts` session events | A `BaseObserver` added to the `PipelineTask` | Same | +| Step spans that links target | `llm_response` rows exported from native `llm_node`; links carry their Braintrust span ids | `llm_response` rows exported from native `realtime_inference` | `llm_response` | `llm_response` | | User fragments (`audio.selections`) | Each native `user_speaking` span, clipped to captured input samples | Realtime `input_speech_started`/`stopped` | `UserStartedSpeakingFrame` → `UserStoppedSpeakingFrame` (or VAD frames) | Service speech events converted to call ms, else the same frames | -| User span created | When the `user_turn` closes: first fragment start to last fragment end | When the SDK-built turn closes | When the turn analyzer ends the user turn | Same | +| User span created | When the `user_turn` closes: first fragment start to last fragment end | When the SDK-built `user_turn` closes, at the input buffer commit | When the turn analyzer ends the user turn | Same | | Ignored user speech | A `user_speaking` interval with no committed user item after end of turn, or a false interruption (`agent_false_interruption`); written once LiveKit resumes or drops the turn | Speech events with no transcript item committed to the conversation; written when the next response starts or the call ends | `UserStoppedSpeakingFrame` with no user aggregation pushed to the LLM; written when the next user turn starts or the pipeline ends | Same | | `end_of_turn_ms` | `AgentActivity.onEndOfTurn` (`EndOfTurnInfo`) | Commit of the input buffer (server VAD) | Turn analyzer stop | Server VAD commit | | User transcript | `user_turn` transcript; `transcription_delay` → `available_ms` | `InputTranscriptionCompleted` → `provider_transcription` | `TranscriptionFrame` (`finalized`) | The service's input transcription frames | -| `heard_by` | The `llm_node` started under the `agent_turn` whose `turn.reply_to` is this turn | The `realtime_inference` started after the commit | The `llm_response` opened by the next `LLMContextFrame` containing the transcript | The response span after the commit | +| `heard_by` | The `llm_response` span (exported from native `llm_node`) started under the `assistant_turn` whose `turn.reply_to` is this turn | The `llm_response` span (exported from native `realtime_inference`) started after the commit | The `llm_response` opened by the next `LLMContextFrame` containing the transcript | The response span after the commit | | Agent playout (`audio.selections`) | `playbackStarted`/`playbackProgressed`/`playbackFinished` per `SpeechHandle` | Same | Output-transport audio-write timestamps between `BotStartedSpeakingFrame` and `BotStoppedSpeakingFrame` | Same | -| `produced_by` | The `llm_node` with the same `speech_id` | The `realtime_inference` with the same `speech_id`/`response_id` | The `llm_response` whose TTS `context_id` matches | The response that emitted the audio | +| `produced_by` | The `llm_response` span (native `llm_node`) with the same `speech_id` | The `llm_response` span (native `realtime_inference`) with the same `speech_id`/`response_id` | The `llm_response` whose TTS `context_id` matches | The response that emitted the audio | | `state`, `played_ms` | `playbackFinished.interrupted`, `playbackPosition` | Same, plus `provider_played_ms` from the truncate call | `InterruptionFrame` while speaking; bytes written to the output transport | Same, plus the service's truncation | | `interrupted_by` | The user speech whose fragment was active when playout stopped | Same | Same | Same | | Agent text | `synchronizedTranscript` → `output.text` (`playback_sync`); `tts` input → `planned_text` | `synchronizedTranscript` → `output.text` (`playback_sync`); response transcript → `planned_text` | `TTSTextFrame`s written while speaking, cut at `InterruptionFrame` → `output.text` (`playback_sync`); TTS input → `planned_text` | The service's text frames observed while the bot is speaking, cut at `InterruptionFrame` → `output.text`; full response text → `planned_text` | @@ -248,6 +249,7 @@ The realtime fixture shows one weakness. Because user turn 2 has no transcript, - **Barge-in truncation.** Covered by `played_ms` and `planned_text`. The thread still shows the LLM's full planned answer as the turn response, because Lingua's `response_text` comes from the LLM span. - **Interrupted before any audio.** No speech span is written. The voice view then cannot show that the agent tried to speak; only the LLM span shows it. - **Overlap.** Two speech spans overlap in time; the app already prefers the later-starting window (#22036). +- **Pipecat realtime spoken text.** `playback_sync` counts text frames that reach the output transport. When the service sends text deltas ahead of the audio, an interrupted utterance can overstate what the user heard. - **Late or missing realtime transcripts.** Covered by the full-object rewrite above, or the transcript stays absent. - A missing transcript splits turns badly (above). - A fix would let user speech with `heard_by` start a new turn without text. That is sketched, not implemented. @@ -266,7 +268,7 @@ The realtime fixture shows one weakness. Because user turn 2 has no transcript, - Log `voice.call` (including `origin_unix_ms`) on `agent_session` start, even with audio off. - Stop exporting `livekit.user_speaking`/`livekit.agent_speaking` as rows. Instead collect their intervals per turn (`speakingByTurn`) and per `SpeechHandle`. - When the user turn closes or playout finishes, create one `speech` span with explicit times, merged selections and `voice.speech`. - - Add `heard_by` by waiting for the first `llm_node`/`realtime_inference` under the replying turn. + - Add `heard_by` by waiting for the first `llm_response` (native `llm_node`/`realtime_inference`) under the replying turn. 2. **Realtime (`realtime.ts`):** build the user speech span from the SDK-built turn's fragments. Convert provider offsets with the input-start call ms. 3. **Pipecat (`sdk-python/py/src/braintrust/integrations/pipecat` on `delner/pipecat-voice-recording`):** - Add a `BaseObserver` to the `PipelineTask`. From 330af3a7fc0c7719f9f2b86e64ee7ae163826410 Mon Sep 17 00:00:00 2001 From: kevchoi <16812845+kevchoi@users.noreply.github.com> Date: Mon, 12 Oct 2026 17:59:19 +0000 Subject: [PATCH 5/8] chore: align speech span capture with livekit 1.9 sources Co-Authored-By: Claude Opus 5.5 --- docs/voice-schema/speech-spans/README.md | 116 +++++++++--------- .../speech-spans/examples/realtime.json | 55 ++++----- .../livekit/voice-schema/speech-spans.test.ts | 2 +- 3 files changed, 86 insertions(+), 87 deletions(-) diff --git a/docs/voice-schema/speech-spans/README.md b/docs/voice-schema/speech-spans/README.md index 18e1a412c..72052819d 100644 --- a/docs/voice-schema/speech-spans/README.md +++ b/docs/voice-schema/speech-spans/README.md @@ -70,41 +70,41 @@ An interrupted speech handle that played no audio produces no speech span; its t **Fields:** -| Field | Type | Required | Meaning | -| -------------------------------------- | --------------------------------------------------------------------------------------- | ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `span_parents` | `[span id]` | Yes | The owning turn span, or the call span for ignored speech and agent speech with no turn. | -| `metrics.start` / `metrics.end` | Unix s | Yes | The utterance bounds. With selections: first selection start to last selection end. | -| `metadata["audio.selections"]` | Selection[] | When audio is captured for that utterance | Unchanged shape. One channel: 0 for user speech, 1 for agent speech. One range per fragment or played stretch, in order. | -| `voice.speech.speaker` | `"user" \| "agent"` | Yes | Who spoke. | -| `voice.speech.turn_id` | span id | When the parent is a turn | The owning turn span. Omitted when the parent is the call span. | -| `voice.speech.state` | `"complete" \| "interrupted" \| "incomplete" \| "ignored"` | Yes | `interrupted` means agent playout was cut short after some audio played. `incomplete` means the call or process ended mid-utterance. `ignored` means user speech the agent heard but did not answer. | -| `voice.speech.timing_source` | See below | Yes | The evidence behind `metrics.start`/`metrics.end`. | -| `voice.speech.heard_by` | span id[] | User only, optional | The LLM spans whose input first contained this utterance. | -| `voice.speech.produced_by` | span id | Agent only, optional | The LLM span that generated the spoken text. | -| `voice.speech.interrupted_by` | span id | Agent only, optional | The user speech span that cut this one off. Requires `state: "interrupted"`. | -| `voice.speech.played_ms` | ms > 0 | Agent only, optional | Audio actually played locally: bytes written to the output transport divided by its byte rate, or the framework's playback position. | -| `voice.speech.provider_played_ms` | ms | Agent realtime only, optional | What the provider was told had played (`conversation.item.truncate` `audio_end_ms`). | -| `voice.speech.end_of_turn_ms` | call ms | User only, optional | When the agent decided the user had finished. | -| `voice.speech.transcript.source` | `"stt" \| "provider_transcription" \| "tts_input" \| "model_output" \| "playback_sync"` | With `transcript` | Where the text came from. For agent speech, `playback_sync` describes `output.text`, and `tts_input`/`model_output` describe `planned_text` only. | -| `voice.speech.transcript.final` | boolean | With `transcript` | `false` when only partial text arrived before the span was written. | -| `voice.speech.transcript.available_ms` | call ms | No | When the text became available. In realtime this is often after the response started. | -| `voice.speech.transcript.confidence` | 0–1 | No | STT confidence, if the provider reports one. | -| `output.text` | string | No | What was actually said. User: the transcript. Agent: the text synchronized with playout (`playback_sync`), never the generated text. Absent without such a transcript or with content capture off. | -| `output.planned_text` | string | No | Agent only: the text the agent generated for this playout: the TTS input (cascaded) or the response transcript (realtime). | +| Field | Type | Required | Meaning | +| -------------------------------------- | --------------------------------------------------------------------------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `span_parents` | `[span id]` | Yes | The owning turn span, or the call span for ignored speech and agent speech with no turn. | +| `metrics.start` / `metrics.end` | Unix s | Yes | The utterance bounds. With selections: first selection start to last selection end. | +| `metadata["audio.selections"]` | Selection[] | When audio is captured for that utterance | Unchanged shape. One channel: 0 for user speech, 1 for agent speech. One range per fragment or played stretch, in order. | +| `voice.speech.speaker` | `"user" \| "agent"` | Yes | Who spoke. | +| `voice.speech.turn_id` | span id | When the parent is a turn | The owning turn span. Omitted when the parent is the call span. | +| `voice.speech.state` | `"complete" \| "interrupted" \| "incomplete" \| "ignored"` | Yes | `interrupted` means agent playout was cut short after some audio played. `incomplete` means the call or process ended mid-utterance. `ignored` means user speech the agent heard but did not answer. | +| `voice.speech.timing_source` | See below | Yes | The evidence behind `metrics.start`/`metrics.end`. | +| `voice.speech.heard_by` | span id[] | User only, optional | The LLM spans whose input first contained this utterance. | +| `voice.speech.produced_by` | span id | Agent only, optional | The LLM span that generated the spoken text. | +| `voice.speech.interrupted_by` | span id | Agent only, optional | The user speech span that cut this one off. Requires `state: "interrupted"`. | +| `voice.speech.played_ms` | ms > 0 | Agent only, optional | Audio actually played locally: bytes written to the output transport divided by its byte rate, or the framework's playback position. | +| `voice.speech.provider_played_ms` | ms | Agent realtime only, optional | The truncation sent to the provider (`conversation.item.truncate` `audio_end_ms`), only when the model supports message truncation. LiveKit derives it from the same playback position, so it equals `played_ms` there. Only integrations that observe the provider independently (Pipecat) can show a provider/playout disagreement. | +| `voice.speech.end_of_turn_ms` | call ms | User only, optional | When the agent decided the user had finished. | +| `voice.speech.transcript.source` | `"stt" \| "provider_transcription" \| "tts_input" \| "model_output" \| "playback_sync"` | With `transcript` | Where the text came from. For agent speech, `playback_sync` describes `output.text`, and `tts_input`/`model_output` describe `planned_text` only. | +| `voice.speech.transcript.final` | boolean | With `transcript` | `false` when only partial text arrived before the span was written. | +| `voice.speech.transcript.available_ms` | call ms | No | When the text became available. In realtime this is often after the response started. | +| `voice.speech.transcript.confidence` | 0–1 | No | STT confidence, if the provider reports one. | +| `output.text` | string | No | What was actually said. User: the transcript. Agent: the text synchronized with playout (`playback_sync`), never the generated text. Absent without such a transcript or with content capture off. | +| `output.planned_text` | string | No | Agent only: the text the agent generated for this playout: the TTS input (cascaded) or the response transcript (realtime). | **`timing_source` values:** -| Value | Meaning | -| --------------- | ------------------------------------------------------------------------------------------------------------------------------ | -| `audio_samples` | Bounds measured on captured audio samples. Requires `audio.selections`. | -| `vad` | Local VAD or turn-detector events (LiveKit `user_speaking`, Pipecat `UserStarted/StoppedSpeakingFrame`). | -| `provider_vad` | Realtime provider speech events, converted to call ms (below). | -| `playout` | Local output-transport playout: LiveKit playback events, Pipecat output audio write timestamps. | -| `span_clock` | Local span times that could not be aligned to any of the above, including provider offsets with an unknown input-buffer start. | +| Value | Meaning | +| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `audio_samples` | Bounds measured on captured audio samples. Requires `audio.selections`. | +| `vad` | Local VAD or turn-detector events (LiveKit `user_speaking`, Pipecat `UserStarted/StoppedSpeakingFrame`). | +| `provider_vad` | Realtime provider speech events with provider ms, converted to call ms (below). Only integrations that see raw provider events (Pipecat) have them; LiveKit realtime uses `span_clock`. | +| `playout` | Local output-transport playout: LiveKit playback events, Pipecat output audio write timestamps. | +| `span_clock` | Local span times that could not be aligned to any of the above, including provider offsets with an unknown input-buffer start. | When audio is not captured, `audio.selections` is absent and `timing_source` must not be `audio_samples`. Selections never extend outside the span bounds. -**Converting realtime provider offsets to call ms.** Providers report `audio_start_ms`/`audio_end_ms` relative to the start of the audio the integration streamed to them. The conversion is: +**Converting realtime provider offsets to call ms.** This applies to integrations that see raw provider events (Pipecat with OpenAI Realtime); LiveKit 1.9 does not expose provider ms. Providers report `audio_start_ms`/`audio_end_ms` relative to the start of the audio the integration streamed to them. - call ms = `input_start_call_ms` + provider ms, where `input_start_call_ms` is the call ms at which the integration began appending input audio, on the recording clock. - If that start is unknown (for example after a reconnect), use local span times and `timing_source: span_clock`. @@ -145,22 +145,25 @@ call span (voice.call) ──audio.recordings──► recording segments ## What each integration captures and when -| Value | LiveKit cascaded (JS) | LiveKit realtime (JS) | Pipecat cascaded (Python) | Pipecat realtime (Python) | -| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | -| Hook | `LiveKitSpanProcessor` (`processor.ts`) plus playback hooks | Same plus `realtime.ts` session events | A `BaseObserver` added to the `PipelineTask` | Same | -| Step spans that links target | `llm_response` rows exported from native `llm_node`; links carry their Braintrust span ids | `llm_response` rows exported from native `realtime_inference` | `llm_response` | `llm_response` | -| User fragments (`audio.selections`) | Each native `user_speaking` span, clipped to captured input samples | Realtime `input_speech_started`/`stopped` | `UserStartedSpeakingFrame` → `UserStoppedSpeakingFrame` (or VAD frames) | Service speech events converted to call ms, else the same frames | -| User span created | When the `user_turn` closes: first fragment start to last fragment end | When the SDK-built `user_turn` closes, at the input buffer commit | When the turn analyzer ends the user turn | Same | -| Ignored user speech | A `user_speaking` interval with no committed user item after end of turn, or a false interruption (`agent_false_interruption`); written once LiveKit resumes or drops the turn | Speech events with no transcript item committed to the conversation; written when the next response starts or the call ends | `UserStoppedSpeakingFrame` with no user aggregation pushed to the LLM; written when the next user turn starts or the pipeline ends | Same | -| `end_of_turn_ms` | `AgentActivity.onEndOfTurn` (`EndOfTurnInfo`) | Commit of the input buffer (server VAD) | Turn analyzer stop | Server VAD commit | -| User transcript | `user_turn` transcript; `transcription_delay` → `available_ms` | `InputTranscriptionCompleted` → `provider_transcription` | `TranscriptionFrame` (`finalized`) | The service's input transcription frames | -| `heard_by` | The `llm_response` span (exported from native `llm_node`) started under the `assistant_turn` whose `turn.reply_to` is this turn | The `llm_response` span (exported from native `realtime_inference`) started after the commit | The `llm_response` opened by the next `LLMContextFrame` containing the transcript | The response span after the commit | -| Agent playout (`audio.selections`) | `playbackStarted`/`playbackProgressed`/`playbackFinished` per `SpeechHandle` | Same | Output-transport audio-write timestamps between `BotStartedSpeakingFrame` and `BotStoppedSpeakingFrame` | Same | -| `produced_by` | The `llm_response` span (native `llm_node`) with the same `speech_id` | The `llm_response` span (native `realtime_inference`) with the same `speech_id`/`response_id` | The `llm_response` whose TTS `context_id` matches | The response that emitted the audio | -| `state`, `played_ms` | `playbackFinished.interrupted`, `playbackPosition` | Same, plus `provider_played_ms` from the truncate call | `InterruptionFrame` while speaking; bytes written to the output transport | Same, plus the service's truncation | -| `interrupted_by` | The user speech whose fragment was active when playout stopped | Same | Same | Same | -| Agent text | `synchronizedTranscript` → `output.text` (`playback_sync`); `tts` input → `planned_text` | `synchronizedTranscript` → `output.text` (`playback_sync`); response transcript → `planned_text` | `TTSTextFrame`s written while speaking, cut at `InterruptionFrame` → `output.text` (`playback_sync`); TTS input → `planned_text` | The service's text frames observed while the bot is speaking, cut at `InterruptionFrame` → `output.text`; full response text → `planned_text` | -| Agent speech without a turn | `session.say` greeting: parent is the call span, no `turn_id`, no `produced_by` | Same | `TTSSpeakFrame`: same | Same | +LiveKit references are to `@livekit/agents` 1.9.2 `src/voice/`; `agent_activity.ts` is abbreviated `AA`. + +| Value | LiveKit cascaded (JS) | LiveKit realtime (JS) | Pipecat cascaded (Python) | Pipecat realtime (Python) | +| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Hook | `LiveKitSpanProcessor` (`processor.ts`), plus an `AudioOutput` playback hook (`io.ts:209` `onPlaybackStarted`, `io.ts:227` `onPlaybackFinished`) installed whether or not audio is captured | Same, plus the realtime session events below | A `BaseObserver` added to the `PipelineTask` | Same; the observer also sees the service's raw provider events | +| Step spans that links target | `llm_response` rows exported from native `llm_node`; links carry their Braintrust span ids | `llm_response` rows exported from native `realtime_inference` | `llm_response` | `llm_response` | +| User fragments (`audio.selections`) | Each native `user_speaking` span, clipped to captured input samples; `timing_source: vad` | `input_speech_started`/`input_speech_stopped` (`AA:895-896`, handlers `AA:1967`, `AA:2000`), which carry no provider ms (`InputSpeechStoppedEvent = {userTranscriptionEnabled}`, `llm/realtime.ts:16`). Bounds are local receipt times, `timing_source: span_clock`; the end includes the server-VAD silence padding unless the integration is configured with that silence duration and subtracts it | `UserStartedSpeakingFrame` → `UserStoppedSpeakingFrame` (or VAD frames); `timing_source: vad` | Provider `speech_started`/`speech_stopped` `audio_start_ms`/`audio_end_ms`, converted to call ms; `timing_source: provider_vad` | +| User span created | When the `user_turn` closes: first fragment start to last fragment end | When the SDK-built `user_turn` closes, at the end of turn below | When the turn analyzer ends the user turn | Same | +| `end_of_turn_ms` | `AA:2690` `onEndOfTurn(EndOfTurnInfo)` resolving true (`audio_recognition.ts:77`) | Entry of `AA:4496` `_realtimeGenerationTaskImpl` for the generation that answers the input. With server turn detection there is no commit event: `userTurnCompleted` (`AA:3275`) returns early | Turn analyzer stop | Provider input-buffer commit | +| Ignored user speech | A `user_speaking` interval with no committed user item after end of turn, or a false interruption (`agent_false_interruption`, `AA:5917`); written once LiveKit resumes or drops the turn | Caller speech after which no generation started (no `_realtimeGenerationTaskImpl` entry) before the next caller speech start or call end; written at that point. An answered barge-in with no transcript is not ignored | `UserStoppedSpeakingFrame` with no user aggregation pushed to the LLM; written when the next user turn starts or the pipeline ends | Same as Pipecat cascaded | +| User transcript | `user_turn` transcript; `transcription_delay` → `available_ms` | Final input transcription → `provider_transcription`; it may never arrive | `TranscriptionFrame` (`finalized`) | The service's input transcription frames | +| `heard_by` | The `llm_response` started under the `assistant_turn` whose `turn.reply_to` is this turn | The `llm_response` started in the `_realtimeGenerationTaskImpl` that ended the turn | The `llm_response` opened by the next `LLMContextFrame` containing the transcript | The response span after the commit | +| Agent playout (`audio.selections`) | `playbackStarted`/`playbackFinished` (`io.ts:209`, `io.ts:227`). Playback events carry no speech id, so the playout is attributed to `activity.currentSpeech` (`AA:1168`) at the first `captureFrame` (`io.ts:164`)/`playbackStarted` | Same | Output-transport audio-write timestamps between `BotStartedSpeakingFrame` and `BotStoppedSpeakingFrame` | Same | +| `produced_by` | The `llm_response` started inside the `SpeechHandle`'s reply task (`AA:3747` `_pipelineReplyTaskImpl`; the handle is bound with `speechHandleStorage.enterWith`, `AA:3784`), the latest one before playback start. Not `lk.speech_id`: that is the turn-level `_turnSpeechId` (`AA:4516`), shared across tool continuations, each of which runs its own reply task and handle | The `llm_response` started inside the handle's `_realtimeGenerationTaskImpl` (`AA:4496`, handle bound at `AA:4523`) | The `llm_response` whose TTS `context_id` matches | The response that emitted the audio | +| `state`, `played_ms` | `PlaybackFinishedEvent.interrupted`, `playbackPosition` (`io.ts:294`) | Same | `InterruptionFrame` while speaking; bytes written to the output transport | Same | +| `provider_played_ms` | — | Only when the model has `capabilities.messageTruncation`: the `audioEndMs` sent with `realtimeSession.truncate` (`AA:4789-4797`). LiveKit computes it from the same `playbackPosition`, so it always equals `played_ms` | — | The truncation the service sent, observed independently of playout; may differ from `played_ms` | +| `interrupted_by` | The user speech whose fragment was active when playout stopped | Same | Same | Same | +| Agent text | `synchronizedTranscript` (`io.ts:301`, set at `AA:4128`/`AA:4146`) → `output.text` (`playback_sync`); `tts` input → `planned_text` | `synchronizedTranscript` (`AA:4729`/`AA:4740`) → `output.text` (`playback_sync`); response transcript → `planned_text` | `TTSTextFrame`s written while speaking, cut at `InterruptionFrame` → `output.text` (`playback_sync`); TTS input → `planned_text` | The service's text frames that reach the output transport while the bot is speaking, cut at `InterruptionFrame` → `output.text`; full response text → `planned_text` | +| Agent speech without a turn | `session.say` greeting (`agent_session.ts:1231`): parent is the call span, no `turn_id`, no `produced_by` | Same | `TTSSpeakFrame`: same | Same | ## What Lingua records and what it infers @@ -220,14 +223,14 @@ The new fields allow: `examples/cascaded.json` and `examples/realtime.json` encode the same fictional call: a booking change with `find_slots`, then a barge-in, then `reschedule`, with a backchannel near the end. Times are call ms. -| | Cascaded | Realtime | -| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Steps under `assistant_turn` 1 | `l1` llm (tool call) → `find_slots` → `l2` llm → `tts` | `r1` realtime response (tool call) → `find_slots` → `r2` response with audio | -| User speech `s1` | Two fragments (300–700 "Hi,", 1000–2900); `timing_source: vad`; `transcript {stt, available_ms: 3450, confidence: 0.94}`; `heard_by: [l1]` | Same fragments; `timing_source: provider_vad`; `transcript {provider_transcription, available_ms: 4300}`, which arrived after the tool call at 4100; `heard_by: [r1]` | -| Agent speech `s2` | 5300–7900; `played_ms: 2600`; `produced_by: l2`; `output.text` is the synchronized spoken text, `planned_text` the TTS input | 5000–7900; `played_ms: 2900` but `provider_played_ms: 2600`, because the local output buffer kept playing for 300 ms after the truncate. `output.text` is the synchronized spoken text (`playback_sync`), and `planned_text` is the full response transcript, which the `r2` response span also logs | -| Barge-in `s3` | `stt` transcript "Nine thirty works." | No transcript: `transcript` and `output` are absent, which the voice view can show as missing | -| Backchannel `s5` (10600–10900) | `state: ignored`, parent is the call span, transcript "Mm-hm." | `state: ignored`, no transcript | -| End of turn → first agent audio | 3400 → 5300 (1.9 s), split across STT, LLM, tool and TTS spans | 3400 → 5000 (1.6 s), with no STT or TTS spans to split it | +| | Cascaded | Realtime | +| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Steps under `assistant_turn` 1 | `l1` llm (tool call) → `find_slots` → `l2` llm → `tts` | `r1` realtime response (tool call) → `find_slots` → `r2` response with audio | +| User speech `s1` | Two fragments (300–700 "Hi,", 1000–2900); `timing_source: vad`; `end_of_turn_ms: 3400` (`onEndOfTurn`); `transcript {stt, available_ms: 3450, confidence: 0.94}`; `heard_by: [l1]` | 300–3400: local receipt of `input_speech_started`/`stopped`, so the 300 ms pause isn't split and the end includes 500 ms of server-VAD silence; `timing_source: span_clock`; `end_of_turn_ms: 3600` (entry of `_realtimeGenerationTaskImpl`); `transcript {provider_transcription, available_ms: 4300}`, which arrived after the tool call at 4100; `heard_by: [r1]` | +| Agent speech `s2` | 5300–7900; `played_ms: 2600`; `produced_by: l2`; `output.text` is the synchronized spoken text, `planned_text` the TTS input | 5000–7900; `played_ms: 2900` and `provider_played_ms: 2900` (truncate sent at 7900 from the same playback position); `output.text` is the synchronized spoken text, `planned_text` the full response transcript, which the `r2` response span also logs | +| Barge-in `s3` | 7600–8500, `stt` transcript "Nine thirty works." | 7600–9000 (silence padding included), `span_clock`; no transcript, so `transcript` and `output` are absent. It is not ignored, because `r3` answered it at 9000 | +| Backchannel `s5` | 10600–10900 during `s4`: `state: ignored`, parent is the call span, transcript "Mm-hm." | 12650–12950, after `s4` (speaking over a realtime agent would interrupt it): no generation started before the call ended, so `state: ignored`, no transcript | +| End of turn → first agent audio | 3400 → 5300 (1.9 s), split across STT, LLM, tool and TTS spans | 3600 → 5000 (1.4 s), with no STT or TTS spans to split it; the 500 ms server-VAD wait before 3600 is invisible | The companion PR checks Lingua's output for both files, plus a cascaded copy with no audio, in its fixtures. @@ -259,7 +262,8 @@ The realtime fixture shows one weakness. Because user turn 2 has no transcript, - **Call in progress.** Speech whose call row isn't logged yet is skipped (#486 behaviour). `incomplete` is set only at shutdown. - **Content capture off.** Text disappears but every link and timing field stays. - **Tool cancelled by interruption.** The tool span has the error. The speech has no direct link to it; readers infer it from the turn. -- **Agent handoff.** Speech stays under the turn of the agent that spoke. `voice.call` has no field for the active agent; it would have to come from turn or agent metadata. +- **Agent handoff.** Speech stays under the turn of the agent that spoke. `voice.call` has no field for the active agent; it would have to come from turn or agent metadata. `voice.call.pipeline` is the pipeline at call start; a handoff that changes the LLM type (for example cascaded to realtime) is out of MVP scope. +- **Provider truncation versus playout.** For LiveKit, `provider_played_ms` is computed from the playback position, so it cannot reveal that the provider believes more or less audio played. Only Pipecat, which observes the provider events itself, can show that disagreement. - **Strict enums.** Lingua rejects unknown `state`, `timing_source` and `transcript.source` values (the whole `voice.speech` key fails). New values need a Lingua release first. ## Implementation starting point @@ -267,9 +271,11 @@ The realtime fixture shows one weakness. Because user turn 2 has no transcript, 1. **LiveKit (`js/src/instrumentation/livekit/processor.ts`):** - Log `voice.call` (including `origin_unix_ms`) on `agent_session` start, even with audio off. - Stop exporting `livekit.user_speaking`/`livekit.agent_speaking` as rows. Instead collect their intervals per turn (`speakingByTurn`) and per `SpeechHandle`. + - Install the `AudioOutput` playback hook even when audio capture is off, and attribute each playout to `activity.currentSpeech` at its first frame. + - Record the `SpeechHandle` bound at reply-task entry (`_pipelineReplyTaskImpl`/`_realtimeGenerationTaskImpl`) and set `produced_by` to the latest `llm_response` started in that task before playback. - When the user turn closes or playout finishes, create one `speech` span with explicit times, merged selections and `voice.speech`. - Add `heard_by` by waiting for the first `llm_response` (native `llm_node`/`realtime_inference`) under the replying turn. -2. **Realtime (`realtime.ts`):** build the user speech span from the SDK-built turn's fragments. Convert provider offsets with the input-start call ms. +2. **Realtime (`realtime.ts`):** build the user speech span from the SDK-built turn's `input_speech_started`/`stopped` receipt times (`span_clock`). Close the turn and set `end_of_turn_ms` at `_realtimeGenerationTaskImpl` entry. Mark caller speech `ignored` when no generation starts before the next caller speech or call end. 3. **Pipecat (`sdk-python/py/src/braintrust/integrations/pipecat` on `delner/pipecat-voice-recording`):** - Add a `BaseObserver` to the `PipelineTask`. - Keep `user_turn`/`assistant_turn` names. diff --git a/docs/voice-schema/speech-spans/examples/realtime.json b/docs/voice-schema/speech-spans/examples/realtime.json index fc18d544c..6eac12701 100644 --- a/docs/voice-schema/speech-spans/examples/realtime.json +++ b/docs/voice-schema/speech-spans/examples/realtime.json @@ -94,7 +94,7 @@ }, "metrics": { "start": 1791830000.3, - "end": 1791830002.9 + "end": 1791830003.4 }, "output": { "text": "Hi, can I move my Thursday cleaning to Friday morning?" @@ -104,9 +104,9 @@ "speaker": "user", "turn_id": "u1", "state": "complete", - "timing_source": "provider_vad", + "timing_source": "span_clock", "heard_by": ["r1"], - "end_of_turn_ms": 3400, + "end_of_turn_ms": 3600, "transcript": { "source": "provider_transcription", "final": true, @@ -118,14 +118,7 @@ "recording_span_id": "call", "recording_id": "call-0000", "start_offset_ms": 300, - "end_offset_ms": 700, - "channel_index": 0 - }, - { - "recording_span_id": "call", - "recording_id": "call-0000", - "start_offset_ms": 1000, - "end_offset_ms": 2900, + "end_offset_ms": 3400, "channel_index": 0 } ] @@ -307,7 +300,7 @@ "produced_by": "r2", "interrupted_by": "s3", "played_ms": 2900, - "provider_played_ms": 2600, + "provider_played_ms": 2900, "transcript": { "source": "playback_sync", "final": true @@ -335,7 +328,7 @@ }, "metrics": { "start": 1791830007.6, - "end": 1791830008.5 + "end": 1791830009.0 }, "input": [], "metadata": { @@ -353,23 +346,23 @@ }, "metrics": { "start": 1791830007.6, - "end": 1791830008.5 + "end": 1791830009.0 }, "metadata": { "voice.speech": { "speaker": "user", "turn_id": "u2", "state": "complete", - "timing_source": "provider_vad", + "timing_source": "span_clock", "heard_by": ["r3"], - "end_of_turn_ms": 8600 + "end_of_turn_ms": 9000 }, "audio.selections": [ { "recording_span_id": "call", "recording_id": "call-0000", "start_offset_ms": 7600, - "end_offset_ms": 8500, + "end_offset_ms": 9000, "channel_index": 0 } ] @@ -385,7 +378,7 @@ "type": "task" }, "metrics": { - "start": 1791830008.6, + "start": 1791830009.0, "end": 1791830012.5 }, "output": [ @@ -409,8 +402,8 @@ "type": "llm" }, "metrics": { - "start": 1791830008.6, - "end": 1791830008.9, + "start": 1791830009.0, + "end": 1791830009.3, "time_to_first_token": 0.3 }, "input": [ @@ -478,8 +471,8 @@ "type": "tool" }, "metrics": { - "start": 1791830008.9, - "end": 1791830009.3 + "start": 1791830009.3, + "end": 1791830009.7 }, "input": { "appointment_id": "apt_481", @@ -505,7 +498,7 @@ "type": "llm" }, "metrics": { - "start": 1791830009.3, + "start": 1791830009.7, "end": 1791830012.5, "time_to_first_token": 0.4 }, @@ -583,7 +576,7 @@ "type": "task" }, "metrics": { - "start": 1791830009.7, + "start": 1791830010.1, "end": 1791830012.5 }, "output": { @@ -596,7 +589,7 @@ "state": "complete", "timing_source": "playout", "produced_by": "r4", - "played_ms": 2800, + "played_ms": 2400, "transcript": { "source": "playback_sync", "final": true @@ -606,7 +599,7 @@ { "recording_span_id": "call", "recording_id": "call-0000", - "start_offset_ms": 9700, + "start_offset_ms": 10100, "end_offset_ms": 12500, "channel_index": 1 } @@ -623,21 +616,21 @@ "type": "task" }, "metrics": { - "start": 1791830010.6, - "end": 1791830010.9 + "start": 1791830012.65, + "end": 1791830012.95 }, "metadata": { "voice.speech": { "speaker": "user", "state": "ignored", - "timing_source": "provider_vad" + "timing_source": "span_clock" }, "audio.selections": [ { "recording_span_id": "call", "recording_id": "call-0000", - "start_offset_ms": 10600, - "end_offset_ms": 10900, + "start_offset_ms": 12650, + "end_offset_ms": 12950, "channel_index": 0 } ] diff --git a/js/src/instrumentation/livekit/voice-schema/speech-spans.test.ts b/js/src/instrumentation/livekit/voice-schema/speech-spans.test.ts index 945b39fd8..b470a6ecf 100644 --- a/js/src/instrumentation/livekit/voice-schema/speech-spans.test.ts +++ b/js/src/instrumentation/livekit/voice-schema/speech-spans.test.ts @@ -73,7 +73,7 @@ describe.each(["cascaded", "realtime"])("%s example", (name) => { speech.find((span) => span.span_id === "s1")?.metadata[ "audio.selections" ], - ).toHaveLength(2); + ).toHaveLength(name === "cascaded" ? 2 : 1); }); }); From a6f8ce4a97c466e944767d12740201cf5b3eda60 Mon Sep 17 00:00:00 2001 From: kevchoi <16812845+kevchoi@users.noreply.github.com> Date: Mon, 12 Oct 2026 18:06:34 +0000 Subject: [PATCH 6/8] chore: tighten speech span capture rules Co-Authored-By: Claude Opus 5.5 --- docs/voice-schema/speech-spans/README.md | 38 ++++++++++++------------ 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/voice-schema/speech-spans/README.md b/docs/voice-schema/speech-spans/README.md index 72052819d..cf11fb3be 100644 --- a/docs/voice-schema/speech-spans/README.md +++ b/docs/voice-schema/speech-spans/README.md @@ -147,23 +147,23 @@ call span (voice.call) ──audio.recordings──► recording segments LiveKit references are to `@livekit/agents` 1.9.2 `src/voice/`; `agent_activity.ts` is abbreviated `AA`. -| Value | LiveKit cascaded (JS) | LiveKit realtime (JS) | Pipecat cascaded (Python) | Pipecat realtime (Python) | -| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Hook | `LiveKitSpanProcessor` (`processor.ts`), plus an `AudioOutput` playback hook (`io.ts:209` `onPlaybackStarted`, `io.ts:227` `onPlaybackFinished`) installed whether or not audio is captured | Same, plus the realtime session events below | A `BaseObserver` added to the `PipelineTask` | Same; the observer also sees the service's raw provider events | -| Step spans that links target | `llm_response` rows exported from native `llm_node`; links carry their Braintrust span ids | `llm_response` rows exported from native `realtime_inference` | `llm_response` | `llm_response` | -| User fragments (`audio.selections`) | Each native `user_speaking` span, clipped to captured input samples; `timing_source: vad` | `input_speech_started`/`input_speech_stopped` (`AA:895-896`, handlers `AA:1967`, `AA:2000`), which carry no provider ms (`InputSpeechStoppedEvent = {userTranscriptionEnabled}`, `llm/realtime.ts:16`). Bounds are local receipt times, `timing_source: span_clock`; the end includes the server-VAD silence padding unless the integration is configured with that silence duration and subtracts it | `UserStartedSpeakingFrame` → `UserStoppedSpeakingFrame` (or VAD frames); `timing_source: vad` | Provider `speech_started`/`speech_stopped` `audio_start_ms`/`audio_end_ms`, converted to call ms; `timing_source: provider_vad` | -| User span created | When the `user_turn` closes: first fragment start to last fragment end | When the SDK-built `user_turn` closes, at the end of turn below | When the turn analyzer ends the user turn | Same | -| `end_of_turn_ms` | `AA:2690` `onEndOfTurn(EndOfTurnInfo)` resolving true (`audio_recognition.ts:77`) | Entry of `AA:4496` `_realtimeGenerationTaskImpl` for the generation that answers the input. With server turn detection there is no commit event: `userTurnCompleted` (`AA:3275`) returns early | Turn analyzer stop | Provider input-buffer commit | -| Ignored user speech | A `user_speaking` interval with no committed user item after end of turn, or a false interruption (`agent_false_interruption`, `AA:5917`); written once LiveKit resumes or drops the turn | Caller speech after which no generation started (no `_realtimeGenerationTaskImpl` entry) before the next caller speech start or call end; written at that point. An answered barge-in with no transcript is not ignored | `UserStoppedSpeakingFrame` with no user aggregation pushed to the LLM; written when the next user turn starts or the pipeline ends | Same as Pipecat cascaded | -| User transcript | `user_turn` transcript; `transcription_delay` → `available_ms` | Final input transcription → `provider_transcription`; it may never arrive | `TranscriptionFrame` (`finalized`) | The service's input transcription frames | -| `heard_by` | The `llm_response` started under the `assistant_turn` whose `turn.reply_to` is this turn | The `llm_response` started in the `_realtimeGenerationTaskImpl` that ended the turn | The `llm_response` opened by the next `LLMContextFrame` containing the transcript | The response span after the commit | -| Agent playout (`audio.selections`) | `playbackStarted`/`playbackFinished` (`io.ts:209`, `io.ts:227`). Playback events carry no speech id, so the playout is attributed to `activity.currentSpeech` (`AA:1168`) at the first `captureFrame` (`io.ts:164`)/`playbackStarted` | Same | Output-transport audio-write timestamps between `BotStartedSpeakingFrame` and `BotStoppedSpeakingFrame` | Same | -| `produced_by` | The `llm_response` started inside the `SpeechHandle`'s reply task (`AA:3747` `_pipelineReplyTaskImpl`; the handle is bound with `speechHandleStorage.enterWith`, `AA:3784`), the latest one before playback start. Not `lk.speech_id`: that is the turn-level `_turnSpeechId` (`AA:4516`), shared across tool continuations, each of which runs its own reply task and handle | The `llm_response` started inside the handle's `_realtimeGenerationTaskImpl` (`AA:4496`, handle bound at `AA:4523`) | The `llm_response` whose TTS `context_id` matches | The response that emitted the audio | -| `state`, `played_ms` | `PlaybackFinishedEvent.interrupted`, `playbackPosition` (`io.ts:294`) | Same | `InterruptionFrame` while speaking; bytes written to the output transport | Same | -| `provider_played_ms` | — | Only when the model has `capabilities.messageTruncation`: the `audioEndMs` sent with `realtimeSession.truncate` (`AA:4789-4797`). LiveKit computes it from the same `playbackPosition`, so it always equals `played_ms` | — | The truncation the service sent, observed independently of playout; may differ from `played_ms` | -| `interrupted_by` | The user speech whose fragment was active when playout stopped | Same | Same | Same | -| Agent text | `synchronizedTranscript` (`io.ts:301`, set at `AA:4128`/`AA:4146`) → `output.text` (`playback_sync`); `tts` input → `planned_text` | `synchronizedTranscript` (`AA:4729`/`AA:4740`) → `output.text` (`playback_sync`); response transcript → `planned_text` | `TTSTextFrame`s written while speaking, cut at `InterruptionFrame` → `output.text` (`playback_sync`); TTS input → `planned_text` | The service's text frames that reach the output transport while the bot is speaking, cut at `InterruptionFrame` → `output.text`; full response text → `planned_text` | -| Agent speech without a turn | `session.say` greeting (`agent_session.ts:1231`): parent is the call span, no `turn_id`, no `produced_by` | Same | `TTSSpeakFrame`: same | Same | +| Value | LiveKit cascaded (JS) | LiveKit realtime (JS) | Pipecat cascaded (Python) | Pipecat realtime (Python) | +| ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Hook | `LiveKitSpanProcessor` (`processor.ts`), plus an `AudioOutput` playback hook (`io.ts:209` `onPlaybackStarted`, `io.ts:227` `onPlaybackFinished`) installed whether or not audio is captured | Same, plus the realtime session events below | A `BaseObserver` added to the `PipelineTask` | Same; the observer also sees the service's raw provider events | +| Step spans that links target | `llm_response` rows exported from native `llm_node`; links carry their Braintrust span ids | `llm_response` rows exported from native `realtime_inference` | `llm_response` | `llm_response` | +| User fragments (`audio.selections`) | Each native `user_speaking` span, clipped to captured input samples; `timing_source: vad` | `input_speech_started`/`input_speech_stopped` (`AA:895-896`, handlers `AA:1967`, `AA:2000`), which carry no provider ms (`InputSpeechStoppedEvent = {userTranscriptionEnabled}`, `llm/realtime.ts:16`). Bounds are local receipt times, `timing_source: span_clock`; the end includes the server-VAD silence padding unless the integration is configured with that silence duration and subtracts it | `UserStartedSpeakingFrame` → `UserStoppedSpeakingFrame` (or VAD frames); `timing_source: vad` | Provider `speech_started`/`speech_stopped` `audio_start_ms`/`audio_end_ms`, converted to call ms; `timing_source: provider_vad` | +| User span created | When the `user_turn` closes: first fragment start to last fragment end | When the SDK-built `user_turn` closes, at the end of turn below | When the turn analyzer ends the user turn | Same | +| `end_of_turn_ms` | `AA:2690` `onEndOfTurn(EndOfTurnInfo)` resolving true (`audio_recognition.ts:77`) | Entry of an answering generation: a `_realtimeGenerationTaskImpl` entry for a generation created by `onGenerationCreated` (`AA:2058-2089`) that is neither user-initiated (`generateReply`) nor a tool-call continuation (`autoToolReplyTurn`). With server turn detection there is no commit event: `userTurnCompleted` (`AA:3275`) returns early | Turn analyzer stop | Provider input-buffer commit | +| Ignored user speech | A `user_speaking` interval with no committed user item after end of turn, or a false interruption (`agent_false_interruption`, `AA:5917`); written once LiveKit resumes or drops the turn | Caller speech after which no answering generation started (as defined in the `end_of_turn_ms` row) before the next caller speech start or call end; written at that point. An answered barge-in with no transcript is not ignored | `UserStoppedSpeakingFrame` with no user aggregation pushed to the LLM; written when the next user turn starts or the pipeline ends | Same as Pipecat cascaded | +| User transcript | `user_turn` transcript; `available_ms` = last fragment end + `transcriptionDelay` (`audio_recognition.ts:144`) | Final input transcription → `provider_transcription`; it may never arrive | `TranscriptionFrame` (`finalized`) | The service's input transcription frames | +| `heard_by` | The `llm_response` started under the `assistant_turn` whose `turn.reply_to` is this turn | The `llm_response` started in the `_realtimeGenerationTaskImpl` that ended the turn | The `llm_response` opened by the next `LLMContextFrame` containing the transcript | The response span after the commit | +| Agent playout (`audio.selections`) | `playbackStarted`/`playbackFinished` (`io.ts:209`, `io.ts:227`). Playback events carry no speech id, so the playout is attributed to `activity.currentSpeech` (`AA:1168`) at the first `captureFrame` (`io.ts:164`)/`playbackStarted` | Same | Output-transport audio-write timestamps between `BotStartedSpeakingFrame` and `BotStoppedSpeakingFrame` | Same | +| `produced_by` | The latest `llm_node` row under the playout owner's `currentSpeech._agentTurnSpan` that started before `playbackStarted`; the SDK already looks up that owner when playout starts (`playback.ts:241`). `speechHandleStorage` (`agent.ts:70`) is not exported, so it can't be read directly. Not `lk.speech_id`: that is the turn-level `_turnSpeechId` (`AA:4516`), shared across tool continuations | The latest `realtime_inference` row under the same owner that started before `playbackStarted` | The `llm_response` whose TTS `context_id` matches | The response that emitted the audio | +| `state`, `played_ms` | `PlaybackFinishedEvent.interrupted`; `played_ms` = `playbackPosition` (seconds, `io.ts:294`) × 1000 | Same | `InterruptionFrame` while speaking; bytes written to the output transport | Same | +| `provider_played_ms` | — | Only when `output.played === "partial"` and the model supports `capabilities.messageTruncation` (`AA:4786-4789`): `Math.floor(playbackPositionInS * 1000)`, the `audioEndMs` sent with `realtimeSession.truncate` (`AA:4797`). Computed from the same playback position, so it equals `played_ms` up to flooring | — | The truncation the service sent, observed independently of playout; may differ from `played_ms` | +| `interrupted_by` | The user speech whose fragment was active when playout stopped | Same | Same | Same | +| Agent text | `synchronizedTranscript` (`io.ts:301`, set at `AA:4128`/`AA:4146`) → `output.text` (`playback_sync`); `tts` input → `planned_text` | `synchronizedTranscript` (`AA:4729`/`AA:4740`) → `output.text` (`playback_sync`); response transcript → `planned_text` | `TTSTextFrame`s written while speaking, cut at `InterruptionFrame` → `output.text` (`playback_sync`); TTS input → `planned_text` | The service's text frames that reach the output transport while the bot is speaking, cut at `InterruptionFrame` → `output.text`; full response text → `planned_text` | +| Agent speech without a turn | `session.say` greeting (`agent_session.ts:1231`): parent is the call span, no `turn_id`, no `produced_by` | Same | `TTSSpeakFrame`: same | Same | ## What Lingua records and what it infers @@ -272,10 +272,10 @@ The realtime fixture shows one weakness. Because user turn 2 has no transcript, - Log `voice.call` (including `origin_unix_ms`) on `agent_session` start, even with audio off. - Stop exporting `livekit.user_speaking`/`livekit.agent_speaking` as rows. Instead collect their intervals per turn (`speakingByTurn`) and per `SpeechHandle`. - Install the `AudioOutput` playback hook even when audio capture is off, and attribute each playout to `activity.currentSpeech` at its first frame. - - Record the `SpeechHandle` bound at reply-task entry (`_pipelineReplyTaskImpl`/`_realtimeGenerationTaskImpl`) and set `produced_by` to the latest `llm_response` started in that task before playback. + - Set `produced_by` to the latest `llm_node`/`realtime_inference` row under the playout owner's `currentSpeech._agentTurnSpan` that started before `playbackStarted`; `playback.ts` already resolves that owner. - When the user turn closes or playout finishes, create one `speech` span with explicit times, merged selections and `voice.speech`. - Add `heard_by` by waiting for the first `llm_response` (native `llm_node`/`realtime_inference`) under the replying turn. -2. **Realtime (`realtime.ts`):** build the user speech span from the SDK-built turn's `input_speech_started`/`stopped` receipt times (`span_clock`). Close the turn and set `end_of_turn_ms` at `_realtimeGenerationTaskImpl` entry. Mark caller speech `ignored` when no generation starts before the next caller speech or call end. +2. **Realtime (`realtime.ts`):** build the user speech span from the SDK-built turn's `input_speech_started`/`stopped` receipt times (`span_clock`). Close the turn and set `end_of_turn_ms` at the entry of an answering generation (not a tool continuation or `generateReply`). Mark caller speech `ignored` when no answering generation starts before the next caller speech or call end. 3. **Pipecat (`sdk-python/py/src/braintrust/integrations/pipecat` on `delner/pipecat-voice-recording`):** - Add a `BaseObserver` to the `PipelineTask`. - Keep `user_turn`/`assistant_turn` names. From 961f4cdb8806f87ebba1777b115ef85dcdbb59af Mon Sep 17 00:00:00 2001 From: kevchoi <16812845+kevchoi@users.noreply.github.com> Date: Mon, 12 Oct 2026 19:58:30 +0000 Subject: [PATCH 7/8] feat: add schema at a glance for speech-spans Co-Authored-By: Claude Opus 5.5 --- docs/voice-schema/speech-spans/README.md | 304 +++++++++++++++--- .../livekit/voice-schema/speech-spans.test.ts | 28 +- .../livekit/voice-schema/speech-spans.ts | 8 +- 3 files changed, 278 insertions(+), 62 deletions(-) diff --git a/docs/voice-schema/speech-spans/README.md b/docs/voice-schema/speech-spans/README.md index cf11fb3be..ccd4a9fb1 100644 --- a/docs/voice-schema/speech-spans/README.md +++ b/docs/voice-schema/speech-spans/README.md @@ -23,13 +23,181 @@ Customers want to replay a call and see four things together: They also need to see where the trace lacks evidence. Today Lingua finds speech by span names that only LiveKit emits (`user_turn`, `assistant_turn`, `livekit.*_speaking`), so a Pipecat trace or a renamed span drops out of the voice view. When those names are missing, Lingua guesses each utterance's LLM step from start times. Interruption appears only as a LiveKit-specific turn flag. -## Contract +## Schema at a glance + +Evidence lives on one `speech` span per utterance (`metadata["voice.speech"]` plus its audio selections), under the turn span it belongs to; the call span carries the clock and the recording. + + +```ts +// Every row: span_id, root_span_id, span_parents: SpanId[] | null, metrics: { start, end } in unix s. SpanId = a span_id in the same trace. + +// Call span (root). span_attributes = { type: "task", name: "livekit.agent_session" | "pipecat.pipeline" } +type CallRow = { + span_parents: null + input?: { audio: Record } // shared @braintrust/audio; when audio is captured + metadata: { + "voice.call": { + framework: "livekit" | "pipecat" + pipeline: "cascaded" | "realtime" | "half_cascade" // at call start + origin_unix_ms: number // unix ms; origin of every call-ms field + } + "audio.recordings"?: { // shared, abridged, one per segment; when audio is captured + id: RecordingId; state: "ready" | "pending" | "omitted" + attachment?: { span_id: SpanId; ref: string } // ready only; ref = "/input/audio/" + timeline: { origin_unix_ms: number // unix ms, = voice.call.origin_unix_ms + recording_start_offset_ms: number } // call ms at which this segment starts + }[] + } +} +type Selection = { // shared; item of metadata["audio.selections"] + recording_span_id: SpanId; recording_id: RecordingId // the call span; a ready audio.recordings[].id + start_offset_ms: number; end_offset_ms: number // ms from that segment's start, not call ms + channel_index: 0 | 1 // 0 caller, 1 agent +} + +// Turn spans, unchanged. span_attributes = { type: "task", name: "user_turn" | "assistant_turn" }; parent = call +type TurnRow = { metadata: { "turn.id": SpanId // own span_id + "turn.reply_to"?: SpanId } } // assistant_turn only: the user_turn it answers + +// Speech span, one per utterance. span_attributes = { type: "task", name: "speech" } +type SpeechRow = { + span_parents: [SpanId] // the turn span, or the call span when there is no turn + output?: { text?: string // what was said; agent: playout-synchronized text only + planned_text?: string } // agent only: TTS input or response transcript + metadata: { + "audio.selections"?: Selection[] // >= 1 when present, in order; user ch 0, agent ch 1 + "voice.speech": { + speaker: "user" | "agent" + state: "complete" | "interrupted" | "incomplete" | "ignored" + timing_source: "audio_samples" | "vad" | "provider_vad" | "playout" | "span_clock" // evidence for metrics + turn_id?: SpanId // = span_parents[0] when that is a turn + heard_by?: SpanId[] // user only: llm_response spans whose input first held it + end_of_turn_ms?: number // user only: call ms + produced_by?: SpanId // agent only: llm_response that generated the text + interrupted_by?: SpanId // agent only: user speech span; requires state "interrupted" + played_ms?: number // agent only: > 0, duration ms played locally + provider_played_ms?: number // agent realtime only: duration ms sent in truncate + transcript?: { + source: "stt" | "provider_transcription" | "tts_input" | "model_output" | "playback_sync" + final: boolean + available_ms?: number // call ms + confidence?: number // 0-1 + } + } + } +} + +// Step spans, unchanged; parent = assistant_turn +type LlmRow = { output: Message[] // { type: "llm", name: "llm_response" }; may hold tool_calls[].id + metrics: { time_to_first_token?: number } } // s +type ToolRow = { input: unknown; output: unknown // { type: "tool", name: }; arguments, result + metadata: { tool_call_id: string } } // = tool_calls[].id in the requesting llm_response output +``` -### Clock +**Time:** -- **Call ms** are milliseconds since `voice.call.origin_unix_ms`. -- **Recording alignment:** when audio is captured, `origin_unix_ms` equals the recording's `timeline.origin_unix_ms`, so call ms are also recording-timeline ms. -- **Span times:** a span's `metrics.start` and `metrics.end` are Unix seconds on the same clock: `(origin_unix_ms + call_ms) / 1000`. +- **Call ms** (`end_of_turn_ms`, `transcript.available_ms`) are ms since `voice.call.origin_unix_ms`, which equals every `audio.recordings[].timeline.origin_unix_ms` when audio is captured. +- **Selection offsets** are ms from the start of the named segment: call ms = `recording_start_offset_ms` + offset. `played_ms` and `provider_played_ms` are durations, not points in time. +- **`metrics.start`/`metrics.end`** are Unix seconds on the same clock, `(origin_unix_ms + call ms) / 1000`. On a speech span with selections they are the first selection start and last selection end in call ms; `timing_source` names the evidence. `time_to_first_token` is in seconds. + +**Relationships:** + +```text +speech.span_parents[0] == speech.turn_id -> user_turn / assistant_turn span_id (call span_id when no turn) +speech(user).heard_by[] -> llm_response span_ids that first saw the utterance +speech(agent).produced_by -> llm_response span_id that generated the spoken text +speech(agent).interrupted_by -> speech(user) span_id +assistant_turn["turn.reply_to"] -> user_turn span_id +tool.metadata.tool_call_id == llm_response.output[].tool_calls[].id (both under one assistant_turn) +selection.recording_span_id / recording_id -> call span_id / audio.recordings[].id +``` + +**Scenario excerpt, cascaded** (user turn, `find_slots` link, interrupted agent utterance and barge-in; each selection also has `"recording_span_id": "call"`): + + +```jsonc +[ + { "span_id": "s1", "span_parents": ["u1"], "metrics": { "start": 1791830000.3, "end": 1791830002.9 }, + "output": { "text": "Hi, can I move my Thursday cleaning to Friday morning?" }, + "metadata": { + "voice.speech": { "speaker": "user", "turn_id": "u1", "state": "complete", "timing_source": "vad", + "heard_by": ["l1"], "end_of_turn_ms": 3400, + "transcript": { "source": "stt", "final": true, "available_ms": 3450, "confidence": 0.94 } }, + "audio.selections": [{ "recording_id": "call-0000", "start_offset_ms": 300, "end_offset_ms": 700, "channel_index": 0 }, + { "recording_id": "call-0000", "start_offset_ms": 1000, "end_offset_ms": 2900, "channel_index": 0 }] } }, + { "span_id": "l1", "span_parents": ["a1"], "span_attributes": { "name": "llm_response", "type": "llm" }, + "output": [{ "role": "assistant", "tool_calls": [{ "id": "call_1", "function": { "name": "find_slots" } }] }] }, + { "span_id": "t1", "span_parents": ["a1"], "span_attributes": { "name": "find_slots", "type": "tool" }, + "output": ["09:30", "11:00"], "metadata": { "tool_call_id": "call_1" } }, + { "span_id": "s2", "span_parents": ["a1"], "metrics": { "start": 1791830005.3, "end": 1791830007.9 }, + "output": { "text": "I have nine thirty or eleven on Fri", + "planned_text": "I have nine thirty or eleven on Friday. Which would you like?" }, + "metadata": { + "voice.speech": { "speaker": "agent", "turn_id": "a1", "state": "interrupted", "timing_source": "playout", + "produced_by": "l2", "interrupted_by": "s3", "played_ms": 2600, "transcript": { "source": "playback_sync", "final": true } }, + "audio.selections": [{ "recording_id": "call-0000", "start_offset_ms": 5300, "end_offset_ms": 7900, "channel_index": 1 }] } }, + { "span_id": "s3", "span_parents": ["u2"], "metrics": { "start": 1791830007.6, "end": 1791830008.5 }, + "output": { "text": "Nine thirty works." }, + "metadata": { + "voice.speech": { "speaker": "user", "turn_id": "u2", "state": "complete", "timing_source": "vad", "heard_by": ["l3"], + "end_of_turn_ms": 8800, "transcript": { "source": "stt", "final": true, "available_ms": 8820, "confidence": 0.97 } }, + "audio.selections": [{ "recording_id": "call-0000", "start_offset_ms": 7600, "end_offset_ms": 8500, "channel_index": 0 }] } } +] +``` + +**Scenario excerpt, realtime** (same rows; `//` marks what differs from cascaded): + + +```jsonc +[ + { "span_id": "s1", "span_parents": ["u1"], "metrics": { "start": 1791830000.3, "end": 1791830003.4 }, // end includes 500 ms server-VAD silence + "output": { "text": "Hi, can I move my Thursday cleaning to Friday morning?" }, + "metadata": { + "voice.speech": { "speaker": "user", "turn_id": "u1", "state": "complete", + "timing_source": "span_clock", // local event receipt, not vad + "heard_by": ["r1"], "end_of_turn_ms": 3600, // answering generation start, not onEndOfTurn + "transcript": { "source": "provider_transcription", "final": true, "available_ms": 4300 } }, // after the 4100 tool call; no confidence + "audio.selections": [{ "recording_id": "call-0000", "start_offset_ms": 300, "end_offset_ms": 3400, "channel_index": 0 }] } }, // one range: pause not split + { "span_id": "r1", "span_parents": ["a1"], "span_attributes": { "name": "llm_response", "type": "llm" }, // from realtime_inference + "output": [{ "role": "assistant", "tool_calls": [{ "id": "call_1", "function": { "name": "find_slots" } }] }] }, + { "span_id": "t1", "span_parents": ["a1"], "span_attributes": { "name": "find_slots", "type": "tool" }, + "output": ["09:30", "11:00"], "metadata": { "tool_call_id": "call_1" } }, + { "span_id": "s2", "span_parents": ["a1"], "metrics": { "start": 1791830005.0, "end": 1791830007.9 }, // first audio 5000, not 5300 + "output": { "text": "I have nine thirty or eleven on Fri", + "planned_text": "I have nine thirty or eleven on Friday. Which would you like?" }, // response transcript, not TTS input + "metadata": { + "voice.speech": { "speaker": "agent", "turn_id": "a1", "state": "interrupted", "timing_source": "playout", + "produced_by": "r2", "interrupted_by": "s3", "played_ms": 2900, + "provider_played_ms": 2900, // realtime only; equals played_ms in LiveKit + "transcript": { "source": "playback_sync", "final": true } }, + "audio.selections": [{ "recording_id": "call-0000", "start_offset_ms": 5000, "end_offset_ms": 7900, "channel_index": 1 }] } }, + { "span_id": "s3", "span_parents": ["u2"], "metrics": { "start": 1791830007.6, "end": 1791830009.0 }, // ends at generation start + // no "output": the provider never transcribed the barge-in + "metadata": { + "voice.speech": { "speaker": "user", "turn_id": "u2", "state": "complete", "timing_source": "span_clock", + "heard_by": ["r3"], "end_of_turn_ms": 9000 }, // no "transcript"; answered, so not ignored + "audio.selections": [{ "recording_id": "call-0000", "start_offset_ms": 7600, "end_offset_ms": 9000, "channel_index": 0 }] } } +] +``` + +**Cascaded vs realtime: what changes in the payload** + +| Field | Cascaded | Realtime (LiveKit) | +| ---------------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | +| User `timing_source` and selections | `vad`; one selection per VAD fragment | `span_clock`; one range from local event receipt, silence padding included | +| User `end_of_turn_ms` | `onEndOfTurn` resolving true (3400) | Entry of the answering generation (3600) | +| User `transcript` | `stt` with `confidence`, available before the LLM starts (3450) | `provider_transcription`, may arrive after the tool call (4300) or never (`s3`: no `transcript`, no `output`) | +| Targets of `heard_by` / `produced_by` | `llm_response` rows from `llm_node`, beside `stt`/`tts` spans | `llm_response` rows from `realtime_inference`; no STT or TTS spans | +| Agent `played_ms` / `provider_played_ms` | 2600 / absent | 2900 / 2900 (truncate sent from the same playback position) | +| Agent `planned_text` | TTS input | Response transcript | + +Full rows: [`examples/cascaded.json`](examples/cascaded.json) and [`examples/realtime.json`](examples/realtime.json). + +Implemented: zod types (`js/src/instrumentation/livekit/voice-schema/speech-spans.ts`), examples, test. Sketch: LiveKit/Pipecat emission. Lingua mapping: implemented in the companion Lingua PR (branch `kevchoi/voice-schema-speech-spans`; link TBD). + +## Contract + +_Implemented: zod schema, examples and test. Emission is a sketch._ All `voice.*`, `audio.*` and `turn.*` keys live under `metadata`. ### Call span @@ -38,7 +206,7 @@ This is the root span: LiveKit `livekit.agent_session`, Pipecat `pipecat.pipelin | Field | Type | Required | Meaning | | ------------------------------ | -------------------------------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `input.audio[]` | attachment | When audio is captured | Unchanged (`@braintrust/audio`). | -| `metadata["audio.recordings"]` | descriptor[] | When audio is captured | Unchanged. Channel 0 is the caller, channel 1 the agent. | +| `metadata["audio.recordings"]` | descriptor[] | When audio is captured | Unchanged. One descriptor per segment; `timeline.recording_start_offset_ms` is the segment start in call ms. Channel 0 is the caller, channel 1 the agent. | | `voice.call.framework` | `"livekit" \| "pipecat"` | Yes | The integration that emitted the call. | | `voice.call.pipeline` | `"cascaded" \| "realtime" \| "half_cascade"` | Yes | Tells readers which speech evidence to expect. `half_cascade` is a realtime model with text output plus separate TTS: user speech follows the realtime column below, agent speech the cascaded column. | | `voice.call.origin_unix_ms` | Unix ms | Yes | The clock for every call-ms field. Logged at session start, whether or not audio is captured. | @@ -48,10 +216,19 @@ This is the root span: LiveKit `livekit.agent_session`, Pipecat `pipecat.pipelin `user_turn` and `assistant_turn` task spans are unchanged: - `turn.id` is the turn span's own `span_id`. -- On an assistant turn, `turn.reply_to` is the user turn it answers. +- On an assistant turn, `turn.reply_to` is the `span_id` of the user turn it answers. Both integrations emit these names; Pipecat's turn spans already use them on `delner/pipecat-voice-recording`. In this option turns group speech and steps; they don't carry speech semantics themselves. +### LLM and tool spans + +These are unchanged and are parented to the `assistant_turn`: + +- `llm_response` (`type: "llm"`) spans are the targets of `heard_by` and `produced_by`. In LiveKit they are the rows exported from native `llm_node` (cascaded) or `realtime_inference` (realtime). +- A tool span (`type: "tool"`, named after the tool) carries `metadata.tool_call_id`, equal to `tool_calls[].id` in the output of the `llm_response` that requested it. That is the only link Lingua reads between a tool and its LLM step. LiveKit also logs `gen_ai.tool.call.id` (same value) and `contrib.livekit.request_span_id` (the requesting span's id); neither is part of this contract. + +Speech never links to a tool span directly. + ### Speech span Each utterance gets one span with `span_attributes: {type: "task", name: "speech"}`. The name stays fixed; Lingua recognizes the span by `voice.speech`, not by name. @@ -70,44 +247,27 @@ An interrupted speech handle that played no audio produces no speech span; its t **Fields:** -| Field | Type | Required | Meaning | -| -------------------------------------- | --------------------------------------------------------------------------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `span_parents` | `[span id]` | Yes | The owning turn span, or the call span for ignored speech and agent speech with no turn. | -| `metrics.start` / `metrics.end` | Unix s | Yes | The utterance bounds. With selections: first selection start to last selection end. | -| `metadata["audio.selections"]` | Selection[] | When audio is captured for that utterance | Unchanged shape. One channel: 0 for user speech, 1 for agent speech. One range per fragment or played stretch, in order. | -| `voice.speech.speaker` | `"user" \| "agent"` | Yes | Who spoke. | -| `voice.speech.turn_id` | span id | When the parent is a turn | The owning turn span. Omitted when the parent is the call span. | -| `voice.speech.state` | `"complete" \| "interrupted" \| "incomplete" \| "ignored"` | Yes | `interrupted` means agent playout was cut short after some audio played. `incomplete` means the call or process ended mid-utterance. `ignored` means user speech the agent heard but did not answer. | -| `voice.speech.timing_source` | See below | Yes | The evidence behind `metrics.start`/`metrics.end`. | -| `voice.speech.heard_by` | span id[] | User only, optional | The LLM spans whose input first contained this utterance. | -| `voice.speech.produced_by` | span id | Agent only, optional | The LLM span that generated the spoken text. | -| `voice.speech.interrupted_by` | span id | Agent only, optional | The user speech span that cut this one off. Requires `state: "interrupted"`. | -| `voice.speech.played_ms` | ms > 0 | Agent only, optional | Audio actually played locally: bytes written to the output transport divided by its byte rate, or the framework's playback position. | -| `voice.speech.provider_played_ms` | ms | Agent realtime only, optional | The truncation sent to the provider (`conversation.item.truncate` `audio_end_ms`), only when the model supports message truncation. LiveKit derives it from the same playback position, so it equals `played_ms` there. Only integrations that observe the provider independently (Pipecat) can show a provider/playout disagreement. | -| `voice.speech.end_of_turn_ms` | call ms | User only, optional | When the agent decided the user had finished. | -| `voice.speech.transcript.source` | `"stt" \| "provider_transcription" \| "tts_input" \| "model_output" \| "playback_sync"` | With `transcript` | Where the text came from. For agent speech, `playback_sync` describes `output.text`, and `tts_input`/`model_output` describe `planned_text` only. | -| `voice.speech.transcript.final` | boolean | With `transcript` | `false` when only partial text arrived before the span was written. | -| `voice.speech.transcript.available_ms` | call ms | No | When the text became available. In realtime this is often after the response started. | -| `voice.speech.transcript.confidence` | 0–1 | No | STT confidence, if the provider reports one. | -| `output.text` | string | No | What was actually said. User: the transcript. Agent: the text synchronized with playout (`playback_sync`), never the generated text. Absent without such a transcript or with content capture off. | -| `output.planned_text` | string | No | Agent only: the text the agent generated for this playout: the TTS input (cascaded) or the response transcript (realtime). | - -**`timing_source` values:** - -| Value | Meaning | -| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `audio_samples` | Bounds measured on captured audio samples. Requires `audio.selections`. | -| `vad` | Local VAD or turn-detector events (LiveKit `user_speaking`, Pipecat `UserStarted/StoppedSpeakingFrame`). | -| `provider_vad` | Realtime provider speech events with provider ms, converted to call ms (below). Only integrations that see raw provider events (Pipecat) have them; LiveKit realtime uses `span_clock`. | -| `playout` | Local output-transport playout: LiveKit playback events, Pipecat output audio write timestamps. | -| `span_clock` | Local span times that could not be aligned to any of the above, including provider offsets with an unknown input-buffer start. | - -When audio is not captured, `audio.selections` is absent and `timing_source` must not be `audio_samples`. Selections never extend outside the span bounds. - -**Converting realtime provider offsets to call ms.** This applies to integrations that see raw provider events (Pipecat with OpenAI Realtime); LiveKit 1.9 does not expose provider ms. Providers report `audio_start_ms`/`audio_end_ms` relative to the start of the audio the integration streamed to them. - -- call ms = `input_start_call_ms` + provider ms, where `input_start_call_ms` is the call ms at which the integration began appending input audio, on the recording clock. -- If that start is unknown (for example after a reconnect), use local span times and `timing_source: span_clock`. +| Field | Type | Required | Meaning | +| -------------------------------------- | --------------------------------------------------------------------------------------- | ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `span_parents` | `[span id]` | Yes | The owning turn span, or the call span for ignored speech and agent speech with no turn. | +| `metrics.start` / `metrics.end` | Unix s | Yes | The utterance bounds. With selections: first selection start to last selection end, converted to call ms (see [Timing semantics](#timing-semantics)). | +| `metadata["audio.selections"]` | Selection[] | When audio is captured for that utterance | Unchanged shape; at least one range when present. One channel: 0 for user speech, 1 for agent speech. One range per fragment or played stretch, in order; the recorder splits a range that crosses a segment boundary. Offsets are ms from the start of the segment named by `recording_id`. | +| `voice.speech.speaker` | `"user" \| "agent"` | Yes | Who spoke. | +| `voice.speech.turn_id` | span id | When the parent is a turn | The owning turn span, equal to `span_parents[0]`. Omitted when the parent is the call span. | +| `voice.speech.state` | `"complete" \| "interrupted" \| "incomplete" \| "ignored"` | Yes | `interrupted` means agent playout was cut short after some audio played. `incomplete` means the call or process ended mid-utterance. `ignored` means user speech the agent heard but did not answer. | +| `voice.speech.timing_source` | See [Timing semantics](#timing-semantics) | Yes | The evidence behind `metrics.start`/`metrics.end`. | +| `voice.speech.heard_by` | span id[] | User only, optional | The `llm_response` spans whose input first contained this utterance. | +| `voice.speech.produced_by` | span id | Agent only, optional | The `llm_response` span that generated the spoken text. | +| `voice.speech.interrupted_by` | span id | Agent only, optional | The user speech span that cut this one off. Requires `state: "interrupted"`. | +| `voice.speech.played_ms` | duration ms, > 0 | Agent only, optional | Audio actually played locally: bytes written to the output transport divided by its byte rate, or the framework's playback position. | +| `voice.speech.provider_played_ms` | duration ms | Agent realtime only, optional | The truncation sent to the provider (`conversation.item.truncate` `audio_end_ms`, ms from the start of that item's audio), only when the model supports message truncation. LiveKit derives it from the same playback position, so it equals `played_ms` there. Only integrations that observe the provider independently (Pipecat) can show a provider/playout disagreement. | +| `voice.speech.end_of_turn_ms` | call ms | User only, optional | When the agent decided the user had finished. | +| `voice.speech.transcript.source` | `"stt" \| "provider_transcription" \| "tts_input" \| "model_output" \| "playback_sync"` | With `transcript` | Where the text came from. For agent speech, `playback_sync` describes `output.text`, and `tts_input`/`model_output` describe `planned_text` only. | +| `voice.speech.transcript.final` | boolean | With `transcript` | `false` when only partial text arrived before the span was written. | +| `voice.speech.transcript.available_ms` | call ms | No | When the text became available. In realtime this is often after the response started. | +| `voice.speech.transcript.confidence` | 0–1 | No | STT confidence, if the provider reports one. | +| `output.text` | string | No | What was actually said. User: the transcript. Agent: the text synchronized with playout (`playback_sync`), never the generated text. Absent without such a transcript or with content capture off. | +| `output.planned_text` | string | No | Agent only: the text the agent generated for this playout: the TTS input (cascaded) or the response transcript (realtime). | **Absence rules:** @@ -121,13 +281,17 @@ When audio is not captured, `audio.selections` is absent and `timing_source` mus ### Creating and updating speech spans +_Sketch: emission rule for integrations._ + - **Creation:** integrations buffer an utterance and create its span when it ends. They pass explicit start and end times and the parent turn (or the call span for ignored speech and agent speech with no turn). This avoids open spans for speech that turns out to be ignored or to merge into a turn. - **Late links:** `heard_by` and `interrupted_by` may be known only after the end. - Recommended: delay the end write until the first LLM span of the reply starts, up to a timeout (for example 2 s). Then most utterances need a single write. - Otherwise: write again later. - **Every write carries the complete current `voice.speech` object.** Braintrust merges metadata objects but replaces arrays, so a partial write would drop earlier links. -### IDs and relationships +## IDs and relationships + +_Implemented in the examples and checked by the test._ ```text call span (voice.call) ──audio.recordings──► recording segments @@ -135,17 +299,43 @@ call span (voice.call) ──audio.recordings──► recording segments │ └─ speech {speaker:user, turn_id, heard_by:[llm span ids]} ──audio.selections (1+ fragments)──► channel 0 ├─ assistant_turn (turn.reply_to = user_turn) │ ├─ llm_response ◄──produced_by── speech {speaker:agent, interrupted_by:user speech} - │ ├─ (tool_call_id) + │ ├─ (tool_call_id = llm_response tool_calls[].id) │ └─ speech ──audio.selections──► channel 1 └─ speech {speaker:user, state:ignored} (no turn_id) ``` -- All links are Braintrust `span_id`s in the same trace; Lingua maps them to row ids. +- All links are Braintrust `span_id`s in the same trace; Lingua maps them to row ids. The only non-span-id link is `tool_call_id`, which matches the provider's tool call id. - Framework ids (LiveKit `speech_id`, Pipecat TTS `context_id`) are only used inside the integration to find those spans. -## What each integration captures and when +## Timing semantics -LiveKit references are to `@livekit/agents` 1.9.2 `src/voice/`; `agent_activity.ts` is abbreviated `AA`. +_Implemented in the zod schema and examples; capture rules are a sketch._ + +- **Call ms** are milliseconds since `voice.call.origin_unix_ms`. When audio is captured, `origin_unix_ms` equals each recording's `timeline.origin_unix_ms`, so call ms are also recording-timeline ms. +- **Selection offsets** are ms from the start of the segment named by `recording_id`. Call ms = that segment's `timeline.recording_start_offset_ms` + offset. In the examples there is one segment starting at 0, so the two coincide. +- **Span times:** a span's `metrics.start` and `metrics.end` are Unix seconds on the same clock: `(origin_unix_ms + call_ms) / 1000`. +- **Durations:** `played_ms` and `provider_played_ms` are lengths of agent audio, not points on the call clock. + +**`timing_source` values:** + +| Value | Meaning | +| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `audio_samples` | Bounds measured on captured audio samples. Requires `audio.selections`. | +| `vad` | Local VAD or turn-detector events (LiveKit `user_speaking`, Pipecat `UserStarted/StoppedSpeakingFrame`). | +| `provider_vad` | Realtime provider speech events with provider ms, converted to call ms (below). Only integrations that see raw provider events (Pipecat) have them; LiveKit realtime uses `span_clock`. | +| `playout` | Local output-transport playout: LiveKit playback events, Pipecat output audio write timestamps. | +| `span_clock` | Local span times that could not be aligned to any of the above, including provider offsets with an unknown input-buffer start. | + +When audio is not captured, `audio.selections` is absent and `timing_source` must not be `audio_samples`. Selections never extend outside the span bounds. + +**Converting realtime provider offsets to call ms.** This applies to integrations that see raw provider events (Pipecat with OpenAI Realtime); LiveKit 1.9 does not expose provider ms. Providers report `audio_start_ms`/`audio_end_ms` relative to the start of the audio the integration streamed to them. + +- call ms = `input_start_call_ms` + provider ms, where `input_start_call_ms` is the call ms at which the integration began appending input audio, on the recording clock. +- If that start is unknown (for example after a reconnect), use local span times and `timing_source: span_clock`. + +## What integrations capture, and when + +_Sketch: no integration emits speech spans yet._ LiveKit references are to `@livekit/agents` 1.9.2 `src/voice/`; `agent_activity.ts` is abbreviated `AA`. | Value | LiveKit cascaded (JS) | LiveKit realtime (JS) | Pipecat cascaded (Python) | Pipecat realtime (Python) | | ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -165,7 +355,9 @@ LiveKit references are to `@livekit/agents` 1.9.2 `src/voice/`; `agent_activity. | Agent text | `synchronizedTranscript` (`io.ts:301`, set at `AA:4128`/`AA:4146`) → `output.text` (`playback_sync`); `tts` input → `planned_text` | `synchronizedTranscript` (`AA:4729`/`AA:4740`) → `output.text` (`playback_sync`); response transcript → `planned_text` | `TTSTextFrame`s written while speaking, cut at `InterruptionFrame` → `output.text` (`playback_sync`); TTS input → `planned_text` | The service's text frames that reach the output transport while the bot is speaking, cut at `InterruptionFrame` → `output.text`; full response text → `planned_text` | | Agent speech without a turn | `session.say` greeting (`agent_session.ts:1231`): parent is the call span, no `turn_id`, no `produced_by` | Same | `TTSSpeakFrame`: same | Same | -## What Lingua records and what it infers +## What Lingua records versus infers + +_Implemented in the companion Lingua PR._ **Recorded as logged:** @@ -183,6 +375,8 @@ LiveKit references are to `@livekit/agents` 1.9.2 `src/voice/`; `agent_activity. - speech order (by audio start, else span start), and overlap from the selections; - `step_ids` falling back to the turn span and its step children when no link was logged. +**Not read:** `played_ms`, `provider_played_ms`, `transcript.final`, `transcript.available_ms`, `confidence`, `voice.call.framework` and `voice.call.pipeline` stay on the spans. + Latency is not stored. The app computes it from `end_of_turn_time`, step times and the next agent speech start. **Rollout precedence:** @@ -193,6 +387,8 @@ Latency is not stored. The app computes it from `end_of_turn_time`, step times a ## Trajectory and app mapping +_Trajectory types are implemented in the companion Lingua PR; app changes are a sketch._ + `Trajectory.voice_calls[].speech[]` gains seven optional fields: - `state` @@ -219,9 +415,9 @@ The new fields allow: - a "timing estimated" badge for `span_clock`; - an end-of-turn → first-audio latency bar. -## Worked example +## Worked example (cascaded vs realtime) -`examples/cascaded.json` and `examples/realtime.json` encode the same fictional call: a booking change with `find_slots`, then a barge-in, then `reschedule`, with a backchannel near the end. Times are call ms. +_Implemented: both files parse with the zod schema in the test._ `examples/cascaded.json` and `examples/realtime.json` encode the same fictional call: a booking change with `find_slots`, then a barge-in, then `reschedule`, with a backchannel near the end. Times are call ms. | | Cascaded | Realtime | | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -268,6 +464,8 @@ The realtime fixture shows one weakness. Because user turn 2 has no transcript, ## Implementation starting point +_Sketch, except step 4._ + 1. **LiveKit (`js/src/instrumentation/livekit/processor.ts`):** - Log `voice.call` (including `origin_unix_ms`) on `agent_session` start, even with audio off. - Stop exporting `livekit.user_speaking`/`livekit.agent_speaking` as rows. Instead collect their intervals per turn (`speakingByTurn`) and per `SpeechHandle`. diff --git a/js/src/instrumentation/livekit/voice-schema/speech-spans.test.ts b/js/src/instrumentation/livekit/voice-schema/speech-spans.test.ts index b470a6ecf..12ad7bd8a 100644 --- a/js/src/instrumentation/livekit/voice-schema/speech-spans.test.ts +++ b/js/src/instrumentation/livekit/voice-schema/speech-spans.test.ts @@ -27,13 +27,27 @@ describe.each(["cascaded", "realtime"])("%s example", (name) => { const call = rows.find((row) => row.span_parents === null); const callInfo = voiceCall.parse(call?.metadata?.["voice.call"]); + const recordings = z + .array( + z.object({ + id: z.string(), + timeline: z.object({ + origin_unix_ms: z.number(), + recording_start_offset_ms: z.number(), + }), + }), + ) + .parse(call?.metadata?.["audio.recordings"]); + const segmentStart = new Map( + recordings.map((r) => [r.id, r.timeline.recording_start_offset_ms]), + ); + const toCallMs = (recordingId: string, offset: number) => + (segmentStart.get(recordingId) ?? NaN) + offset; test("the call span declares its pipeline and clock", () => { expect(callInfo.pipeline).toBe(name); - const [recording] = z - .array(z.object({ timeline: z.object({ origin_unix_ms: z.number() }) })) - .parse(call?.metadata?.["audio.recordings"]); - expect(recording.timeline.origin_unix_ms).toBe(callInfo.origin_unix_ms); + for (const recording of recordings) + expect(recording.timeline.origin_unix_ms).toBe(callInfo.origin_unix_ms); }); test("speech links resolve to spans of the right kind", () => { @@ -62,11 +76,13 @@ describe.each(["cascaded", "realtime"])("%s example", (name) => { test("speech spans run from the first to the last audio fragment", () => { for (const span of speech) { const selections = span.metadata["audio.selections"] ?? []; + const first = selections[0]; + const last = selections[selections.length - 1]; expect(span.metrics.start * 1000 - callInfo.origin_unix_ms).toBeCloseTo( - selections[0].start_offset_ms, + toCallMs(first.recording_id, first.start_offset_ms), ); expect(span.metrics.end * 1000 - callInfo.origin_unix_ms).toBeCloseTo( - selections[selections.length - 1].end_offset_ms, + toCallMs(last.recording_id, last.end_offset_ms), ); } expect( diff --git a/js/src/instrumentation/livekit/voice-schema/speech-spans.ts b/js/src/instrumentation/livekit/voice-schema/speech-spans.ts index bf1a8e58b..9c6019e0f 100644 --- a/js/src/instrumentation/livekit/voice-schema/speech-spans.ts +++ b/js/src/instrumentation/livekit/voice-schema/speech-spans.ts @@ -1,12 +1,14 @@ import { z } from "zod/v3"; const callMs = z.number().nonnegative(); +const durationMs = z.number().nonnegative(); +const segmentMs = z.number().nonnegative(); const selection = z.object({ recording_span_id: z.string(), recording_id: z.string(), - start_offset_ms: callMs, - end_offset_ms: callMs, + start_offset_ms: segmentMs, + end_offset_ms: segmentMs, channel_index: z.union([z.literal(0), z.literal(1)]), }); @@ -32,7 +34,7 @@ export const voiceSpeech = z produced_by: z.string().optional(), interrupted_by: z.string().optional(), played_ms: z.number().positive().optional(), - provider_played_ms: callMs.optional(), + provider_played_ms: durationMs.optional(), end_of_turn_ms: callMs.optional(), transcript: z .object({ From 279da4923a64b1117b96cdd75191fd2fb0c9e016 Mon Sep 17 00:00:00 2001 From: kevchoi <16812845+kevchoi@users.noreply.github.com> Date: Mon, 12 Oct 2026 19:59:31 +0000 Subject: [PATCH 8/8] chore: link companion lingua pr Co-Authored-By: Claude Opus 5.5 --- docs/voice-schema/speech-spans/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/voice-schema/speech-spans/README.md b/docs/voice-schema/speech-spans/README.md index ccd4a9fb1..9076fb308 100644 --- a/docs/voice-schema/speech-spans/README.md +++ b/docs/voice-schema/speech-spans/README.md @@ -193,7 +193,7 @@ selection.recording_span_id / recording_id -> call span_id / audio.recordings[]. Full rows: [`examples/cascaded.json`](examples/cascaded.json) and [`examples/realtime.json`](examples/realtime.json). -Implemented: zod types (`js/src/instrumentation/livekit/voice-schema/speech-spans.ts`), examples, test. Sketch: LiveKit/Pipecat emission. Lingua mapping: implemented in the companion Lingua PR (branch `kevchoi/voice-schema-speech-spans`; link TBD). +Implemented: zod types (`js/src/instrumentation/livekit/voice-schema/speech-spans.ts`), examples, test. Sketch: LiveKit/Pipecat emission. Lingua mapping: implemented in the companion Lingua PR [braintrustdata/lingua#489](https://github.com/braintrustdata/lingua/pull/489). ## Contract