Voice schema option: speech spans

[kev (kevchoi)]

AI-created or modified and not human-reviewed in its current form; treat this artifact as provisional. Updated 2026-10-12.

Exploration draft. This is one of several alternative voice-trace schemas; it is not meant to merge as-is.

Customer problem

Customers replaying a voice call need to know what each side said, hear the matching audio, see where the delay was, see who interrupted whom, and see which tools ran. They also need to see when evidence is missing. Today speech is recognized only by LiveKit span names and contrib.livekit.* keys, so other integrations don't show up. Links from an utterance to its LLM step are guessed from start times.

Proposed contract

• Each utterance is a speech task span under its turn, carrying audio.selections and a neutral metadata["voice.speech"]. That key holds:
  • the speaker and its state (complete, interrupted, incomplete or ignored) and timing source;
  • the links heard_by (user), produced_by (agent) and interrupted_by;
  • the played durations, local and provider-reported;
  • end_of_turn_ms;
  • the transcript's source, finality, availability time and confidence.
• output.text and planned_text hold the content.
• The call span adds voice.call {framework, pipeline}. The recording layer (audio.recordings) is unchanged.
• The full contract, a capture table for LiveKit and Pipecat in cascaded and realtime modes, edge cases and an implementation plan are in docs/voice-schema/speech-spans/README.md.

Main tradeoff

Per-utterance evidence is explicit and framework-neutral, at the cost of extra spans and late merge writes for links known only after the utterance ends. Delay diagnosis still depends on the pipeline step spans, which realtime pipelines don't have.

Implemented vs sketched

• Implemented:
  • zod schema and inferred types (js/src/instrumentation/livekit/voice-schema/speech-spans.ts, not exported);
  • example rows for one fictional call, cascaded and realtime (docs/voice-schema/speech-spans/examples/);
  • a test that validates the rows, link targets, channels and timing.
• Sketched (README only): LiveKit and Pipecat emission. The LiveKit processor is unchanged.

Checks

• pnpm exec vitest run src/instrumentation/livekit/voice-schema (in js/): 6 passed. It failed as expected when produced_by pointed at a non-LLM span.
• pnpm exec tsc --noEmit -p tsconfig.json (in js/): passed.
• pnpm exec eslint src/instrumentation/livekit/voice-schema: passed.
• pnpm exec prettier --check on the changed files: passed.
• Skipped: e2e scenarios and changeset. There is no runtime change.

Update 2026-10-12: semantics gaps resolved

A review against "can an SDK engineer start without inventing essential semantics" found gaps in this option; they are resolved below. This section supersedes the Checks section above where they differ.

SDK 16dd784b and Lingua 265a4a46 resolve gaps that an implementer would otherwise have had to invent:

• Utterance grain:
  • User speech is one span per user turn, with each VAD fragment as an audio.selections range. Span bounds run from the first fragment to the last.
  • Agent speech is one span per playout of a speech handle (LiveKit SpeechHandle, Pipecat BotStarted → BotStopped). Pause and resume stays one span.
  • The example s1 now has two fragments.
• Creation and parenting:
  • Spans are buffered and created at their end, with explicit times and parent.
  • Ignored speech and backchannels are parented to the call span without turn_id. The examples add a backchannel s5, and Lingua accepts either parent.
• Audio off:
  • Required voice.call.origin_unix_ms is the clock for every call-ms field, and Lingua finds calls via voice.call.
  • Without audio there are no selections, and timing_source must not be audio_samples.
  • Speech gains start_time/end_time. A new no-audio Lingua fixture covers this.
• Late links: every write carries the complete voice.speech object, because arrays are replaced on merge. The end write waits for the first LLM span, with a timeout, so most utterances need one write.
• Timing:
  • Defined the relation between metrics and selections, and each timing_source value.
  • Defined the conversion of realtime provider offsets to call ms (input-start offset on the recording clock, else span_clock).
• Precedence: voice.speech now replaces LiveKit name rules per trace and speaker, not per trace (new test).
• Nothing played:
  • An agent speech handle that played no audio gets no speech span, so played_ms is always > 0.
  • output.text is optional everywhere.
• Pipecat:
  • Keeps the user_turn/assistant_turn names.
  • Captures through a BaseObserver on the PipelineTask, using output-transport audio-write timestamps.
  • played_ms comes from bytes written, and links use Braintrust span ids.

Checks:

Repo	Check	Result
SDK	vitest run src/instrumentation/livekit/voice-schema	8 passed (adds the audio_samples-without-audio and played_ms: 0 rejections)
SDK	tsc --noEmit -p tsconfig.json, eslint, prettier --check	Clean
Lingua	cargo test -p lingua (private target dir)	1451 lib tests plus integration and doc tests passed
Lingua	clippy --all-targets, cargo fmt --check	Clean
Lingua	Regenerated bindings	Only Speech.ts changed; git status clean after commit

Still skipped: wasm/TS binding tests, e2e, app.

Independent re-run after these commits: SDK vitest run src/instrumentation/livekit/voice-schema 8 passed and tsc --noEmit passed; full cargo test -p lingua in the Lingua worktree passed (1475 tests, 0 failed).

Confirmation review 2026-10-12

After the gap fixes above, fresh reviewers re-read this option as an engineer about to implement it and gave a pass/fail verdict on whether any essential semantics were still left to invent. Result: PASS on the second round. The follow-up changes below closed the remaining items.

Final independent re-run: SDK vitest run src/instrumentation/livekit/voice-schema 9 passed, tsc --noEmit and prettier --check passed; full cargo test -p lingua 1475 passed, 0 failed.

Follow-up 2026-10-12: realtime spoken text

SDK b3c971dd and Lingua 23513037:

• Agent spoken text:
  • output.text comes only from playout-synchronized text, transcript.source: "playback_sync", in both cascaded and realtime. On LiveKit that is playbackFinished.synchronizedTranscript in content mode. On Pipecat it is the text frames observed while the bot speaks, cut at the InterruptionFrame.
  • planned_text is the TTS input (cascaded) or the response transcript (realtime). Realtime response transcripts hold the full planned text, and OpenAI truncate deletes the server transcript.
  • Without synchronized text, output.text is absent and transcript.source (tts_input/model_output) describes planned_text only.
  • The zod schema enforces this, with a new test.
• Realtime example: s2 and s4 now use playback_sync. The r2 response span logs the full planned text, and the next request's history holds the truncated text. The Lingua realtime fixture is updated to match.
• Agent speech with no turn: a session.say or TTSSpeakFrame greeting is parented to the call span without turn_id.
• Ignored speech: new capture rows say how it is detected (LiveKit: no committed user item, or a false interruption; Pipecat: UserStopped with no aggregation pushed) and when it is written.
• half_cascade: documented as the realtime user path plus the cascaded agent path.
• Lingua spec: a voice.speech key that fails to parse is reported on its span and does not count toward per-speaker precedence. This was already the code's behaviour; it is now documented.

Checks:

Repo	Check	Result
SDK	vitest	9 passed
SDK	tsc, eslint, prettier --check	Clean
Lingua	cargo test -p lingua (private target dir)	1451 lib tests plus integration and doc tests passed
Lingua	clippy --all-targets, fmt --check	Clean
Lingua	Bindings	Types unchanged, so no regeneration

Follow-up 2026-10-12: naming tidy

SDK e12b494a (README only):

• Capture table naming: uses assistant_turn throughout. Links target exported llm_response span ids (native llm_node/realtime_inference); a new "step spans" row says so.
• Realtime user_turn: an SDK-built realtime user_turn closes at the input buffer commit.
• Edge case added: Pipecat realtime playback_sync counts text frames at the output transport, so it can overstate what was heard when text deltas lead the audio.

Checks: SDK vitest 9 passed; tsc, eslint and prettier --check clean. Lingua is unchanged.

LiveKit source alignment 2026-10-12

Engineers who hadn't written this contract did an emission dry run against the real @livekit/agents 1.9.2 source, plus @livekit/agents-plugin-openai 1.9.2 for realtime items. They mapped every field to the hook that supplies it. The first round found realtime signals the examples had assumed that LiveKit doesn't expose: provider timestamps on realtime speech events, a server-VAD commit event, and a provider truncation point independent of local playout. The changes below fix that, and every capture point is now cited by file:line. Result: PASS on the second round.

Final independent re-run: SDK vitest run src/instrumentation/livekit/voice-schema 9 passed, tsc --noEmit and prettier --check passed; full cargo test -p lingua 1475 passed, 0 failed.

LiveKit source alignment (2026-10-12)

SDK 330af3a7 and Lingua f42171ce align the contract with @livekit/agents 1.9.2. The README capture table now cites src/voice/…:line for each hook.

• A. Caller timing in LiveKit realtime:
  • input_speech_started/stopped carry no provider ms (llm/realtime.ts:16), so caller bounds are local receipt times with timing_source: span_clock.
  • The end includes the server-VAD silence unless the integration is configured with that silence duration.
  • provider_vad and provider-ms conversion now apply only to integrations that see raw provider events (Pipecat).
• B. End of turn: in LiveKit realtime, end_of_turn_ms is the entry of _realtimeGenerationTaskImpl (agent_activity.ts:4496), because there is no commit event and userTurnCompleted returns early.
• C. Provider truncation:
  • provider_played_ms is the truncate audioEndMs, sent only with capabilities.messageTruncation (agent_activity.ts:4789-4797) and computed from the same playback position. So it equals played_ms for LiveKit.
  • Example s2 is now 2900/2900.
  • A provider/playout disagreement is observable only by Pipecat.
• D. Agent playout:
  • produced_by is the llm_response started in the SpeechHandle's reply task (_pipelineReplyTaskImpl :3747, _realtimeGenerationTaskImpl :4496), not lk.speech_id. That id is the turn-level _turnSpeechId (:4516), shared across tool continuations.
  • Playout is attributed through activity.currentSpeech (:1168).
  • The playback hook (io.ts:209/227) is installed even with audio capture off.
• G. Realtime ignored speech: caller speech is ignored when no generation starts before the next caller speech or call end. An answered barge-in without a transcript is not ignored.
• I. Handoff: voice.call.pipeline is the pipeline at call start; an LLM-type change in a handoff is out of MVP scope.
• Realtime example changes:
  • s1 is now 300–3400 with span_clock; the end of turn moves from 3400 to 3600.
  • s3 is 7600–9000; r3 and the later spans shift by 400 ms.
  • Backchannel s5 moves after s4 (12650–12950), because speaking over a realtime agent would interrupt it.
  • The Lingua realtime fixture's expected output is regenerated to match.

Checks:

Repo	Check	Result
SDK	vitest	9 passed (fragment-count assertion is now pipeline-aware)
SDK	tsc, eslint, prettier --check	Clean
Lingua	cargo test -p lingua (private target dir)	1451 lib tests plus integration and doc tests passed
Lingua	clippy --all-targets, fmt --check	Clean
Lingua	Bindings	Types unchanged, so not regenerated

Precision notes (SDK a6f8ce4a, README only)

• produced_by: speechHandleStorage (agent.ts:70) is not exported. The implementable rule is the latest llm_node/realtime_inference row under the playout owner's currentSpeech._agentTurnSpan that started before playbackStarted; playback.ts:241 already resolves that owner.
• Played durations:
  • provider_played_ms exists only when output.played === "partial" and messageTruncation is supported (agent_activity.ts:4786-4789). Its value is Math.floor(playbackPositionInS * 1000).
  • played_ms is playbackPosition (seconds, io.ts:294) × 1000.
• Answering generation: for the realtime end of turn and the ignored rule, this excludes tool-continuation and generateReply generations (agent_activity.ts:2058-2089).
• Cascaded transcript timing: transcript.available_ms = last fragment end + transcriptionDelay (audio_recognition.ts:144).

Checks:

• SDK vitest: 9 passed.
• tsc, eslint, prettier --check: clean.
• Lingua: untouched, not re-run.

Base: delner/livekit-voice-recording (2f184fa).

Schema at a glance 2026-10-12

• What changed: SDK 961f4cdb reorders the README into the shared 14-section skeleton and adds "Schema at a glance": the logged row shapes as one TS block, time rules, ID links, and trimmed cascaded and realtime JSONC excerpts (checked value by value against the example files), plus a cascaded vs realtime table. Lingua 61040980 adds a glance to the spec covering the input keys read and the Speech output fields, with what is recorded and what is inferred for each.
• Gap fixed, selection clock: audio.selections offsets are ms from the start of their segment (@braintrust/audio segmentSelections; Lingua AudioSelection), not call ms. The README, zod (segmentMs) and the test now map offsets through recording_start_offset_ms. The examples are unchanged: they have one segment starting at 0.
• Gap fixed, tool link: the README now states the link: tool metadata.tool_call_id equals tool_calls[].id on the requesting llm_response. contrib.livekit.request_span_id is not part of the contract.
• Gap fixed, units: played_ms and provider_played_ms are labelled as durations (zod durationMs), and turn.reply_to and turn_id as span ids. The README also lists the fields Lingua does not read.
• Spec fix: Lingua reads voice.call.framework and pipeline as plain strings, so an unknown value is not rejected. The spec said it was.
• Checks: vitest passed (9 tests). tsc --noEmit and prettier --check on the changed files: clean. The leak grep matched only call-0000, the SDK's own segment id (already in the base branch's e2e snapshots).
• Skipped: Lingua cargo checks and make generate-types (docs-only change), and eslint (the test file is eslint-ignored; the zod file is clean).
• Fourth option: a separate "message ids + timeline events" option was assessed and not added. See the comparison note.

Companion PR: braintrustdata/lingua#489
Other options: call timeline (#2612, braintrustdata/lingua#490); message audio (#2613, braintrustdata/lingua#491)
Comparison note: shared with Kev in the originating Slack thread; it links all six drafts.

🤖 Generated with Claude Code