Voice schema option: message audio

[kev (kevchoi)]

AI-created or modified and not human-reviewed in its current form; treat this artifact as provisional. Updated 2026-10-12.

Draft exploration. This is one of three alternative voice-trace schemas, kept for comparison; it is not meant to merge as is.

Customer problem

A customer reading a voice call wants to see each message, play exactly that message's audio, and know whether its text is what was actually said or heard. Today speech lives on span names and vendor keys (user_turn, livekit.agent_speaking, contrib.livekit.interrupted), so Lingua links audio to LLM steps by start time. That linking is weakest at barge-in, split user turns, and late realtime transcripts.

Proposed contract

• Call span (livekit.agent_session / pipecat.pipeline): metadata["voice.origin_unix_ms"] plus metadata["voice.messages"], a map of message id to VoiceMessage. Each VoiceMessage records:
  • role, seq, committed, and state (complete / interrupted / unplayed / open);
  • spoken text and generated_text;
  • text_status (final / partial / missing / not_captured) and text_source;
  • call-ms bounds and end-of-turn, transcript-final and interruption times;
  • the provider truncation point, unconverted;
  • @braintrust/audio selections, produced_by_span_id, and native source_ids.
• LLM spans: voice.context_message_ids, aligned 1:1 with input, and voice.output_message_id.
• Recording: the existing @braintrust/audio whole-call recording is unchanged.
• No speech spans. Turn spans become optional.
• Why a metadata map rather than message content parts: Lingua universal messages round-trip to providers and are hashed for turn alignment, transcripts and audio arrive after the message is logged, and Lingua builds voice data from headers only.

Main tradeoff

Audio, text and evidence quality are keyed by the same id the thread view renders, and the LLM step link is explicit, so nothing is matched by time. The cost:

• Capture has to track message ids across chat context, playout and transcription.
• One call-span map grows with call length.
• Delay causes and interruption causes are inferred from timing, not recorded.

Implemented vs sketched

• Implemented:
  • zod schemas and types (js/src/instrumentation/livekit/voice-schema/message-audio.ts, not exported);
  • cascaded and realtime example traces of the shared fictional call, in docs/voice-schema/message-audio/examples/;
  • a vitest suite that validates them and their cross-links.
• Sketched: the README at docs/voice-schema/message-audio/README.md covers fields, IDs, timing, capture points for LiveKit and Pipecat (cascaded and realtime), Lingua mapping, usefulness, edge cases, and an implementation order. No integration code changes.

Checks

• pnpm exec vitest run src/instrumentation/livekit/voice-schema/message-audio.test.ts (in js/): 8 passed.
• pnpm exec tsc --noEmit -p tsconfig.json and -p tsconfig.test.json (in js/): passed.
• pnpm exec eslint src/instrumentation/livekit/voice-schema/ (in js/) and pnpm exec prettier --write on the changed files: clean.
• Skipped: e2e scenarios and recordings, since there are no integration changes; Pipecat (Python), which is only sketched; and staging trace compatibility checks.

Update 2026-10-12: semantics gaps resolved

A review against "can an SDK engineer start without inventing essential semantics" found gaps in this option; they are resolved below. This section supersedes the Checks section above where they differ.

• Merge writes. There is one accumulator per message, and every write logs the whole entry; selections is always the full array. Fields only move forward:
  • state goes from open to one terminal value;
  • text_status may go partial → final/missing and missing → final, but never partial back from missing;
  • Lingua validates only the merged row.
• Assistant writer. voice.messages is no longer gated on content capture.
  • A record is keyed by speech id when the SpeechHandle first appears at _pipelineReplyTaskImpl / _realtimeGenerationTaskImpl, and seq is assigned then.
  • Playback events fill it, and it moves to item.id at _conversationItemAdded.
  • Speech with no item is keyed by its LiveKit speech_id, which already carries the speech_ prefix (unplayed, or interrupted if anything played).
• Message grain.
  • A user item takes every VAD interval between the previous commit and its own, so it can have several ranges, and its bounds are the outer ones.
  • An assistant item takes only its own speech's playout.
  • The cascaded example's item_u1 now has two fragments.
• Context ids. These are context items in order, with null for inputs that are not items. They are read from chatCtx.items outside the PII path. Lingua checks length only when input is present.
• Uncommitted speech.
  • It is written only after VAD stop plus UNCOMMITTED_SPEECH_TIMEOUT_MS with no item and no end-of-turn in progress, as committed: false, complete, with no text.
  • The LiveKit and Pipecat cascaded sources are named from hooks already instrumented.
  • The MVP drops uncommitted speech for Pipecat realtime.
  • No "verify" cells remain.
• Contradictions fixed.
  • voice.context_message_ids stays optional, and Lingua warns when a call has messages but no LLM span carries ids.
  • Neither Lingua nor the SDK stores derived flags; the app computes them.
  • text_source is allowed without text, as the expected source.
  • timing_source was added to the Lingua VoiceMessage (new VoiceTimingSource).
  • turn_span_id and source_ids are intentionally not stored.
• Realtime.
  • Items are paired to the response span by responseId/messageId, the provider's response.output_item.added.
  • generated_text is that item's transcript only.
  • A transcript is marked missing when the next response after the one that heard it is created, or at session close, and a late arrival can still upgrade it.
• Pipecat ids. A list of ids parallel to context indices, synced on appends. On rewrites it is re-mapped by an in-memory content hash that is never logged, and unmatched messages get new ids.

Checks

• SDK (js/):
  • vitest run …/message-audio.test.ts: 13 passed;
  • tsc --noEmit on tsconfig.json and tsconfig.test.json: passed;
  • eslint: clean;
  • prettier --check: clean.
• Lingua:
  • make generate-types: passed;
  • cargo test -p lingua --lib processing::trajectory: 96 passed;
  • cargo test -p lingua --test trajectory: 2 passed;
  • cargo clippy -p lingua --all-targets -- -D warnings: clean;
  • cargo fmt: applied.
• git status is clean in both worktrees.
• Still skipped: e2e, wasm/TS package tests, Pipecat code, and staging traces.

Independent re-run after these commits: SDK vitest run src/instrumentation/livekit/voice-schema 13 passed and tsc --noEmit passed; full cargo test -p lingua in the Lingua worktree passed (1471 tests, 0 failed).

Confirmation review 2026-10-12

After the gap fixes above, fresh reviewers re-read this option as an engineer about to implement it and gave a pass/fail verdict on whether any essential semantics were still left to invent. Result: PASS on the third round. The follow-up changes below closed the remaining items.

Final independent re-run: SDK vitest run src/instrumentation/livekit/voice-schema 16 passed, tsc --noEmit and prettier --check passed; full cargo test -p lingua 1471 passed, 0 failed.

Update 2026-10-12: tool-only speech and realtime turn timing

• Tool-only speech. LiveKit creates a SpeechHandle for each reply, including tool-call-only steps. A speech record with no generated text and no played audio is never written, which leaves a gap in seq; the examples now show gaps at 1 and 4.
• unplayed. Used only for speech that has generated text but played no audio: text is the generated text (tts_input), and selections is empty.
• committed. For assistant records, it means the record entered the agent's chat context. Cancelled speech that never entered the context is committed: false, unplayed or interrupted. Zod enforces this and the new unplayed constraints.
• Realtime end_ms. It is the input_speech_stopped receipt time minus the configured server-VAD silence_duration_ms when known (provider_events), otherwise the receipt time.
• Realtime end_of_turn_ms. It is the entry of _realtimeGenerationTaskImpl for the generation that first includes the item. Without a known silence setting the two values can coincide.
• Realtime example. User turn 1 now has end_of_turn_ms 3600. Turn 2 uses the same 500 ms silence, so the realtime reschedule call happens at 9150 instead of the scenario's 8900; this deviation is noted in the README.
• transcript_final_ms. Defined as the arrival of the final STT result: user_input_transcribed with isFinal, realtime input_audio_transcription_completed, or a finalized Pipecat TranscriptionFrame.
• Cascaded example. item_u2 now has end_of_turn_ms 8900 and transcript_final_ms 8950, with c-llm3 starting at 8960, consistent with item_u1.
• Other notes. The README now says that realtime LLM steps are the native realtime_inference span exported as llm_response. The uncommitted-speech open question notes that Pipecat has no end-of-turn-in-progress signal.
• Lingua spec. It now has the matching validation rules and the expected seq gaps.

Checks

• SDK (js/):
  • vitest run …/message-audio.test.ts: 15 passed.
  • tsc --noEmit on tsconfig.json and tsconfig.test.json: passed.
  • eslint: clean.
  • prettier --check: clean.
• Lingua: only specs/voice-schema-message-audio.md changed, so I skipped generate-types, cargo test and clippy; they last passed at 243922a0.
• git status is clean in both worktrees.

Update 2026-10-12: assistant spoken text sources

• Assistant spoken text, one rule for cascaded and realtime:
  • text is the text aligned to played audio, with text_source: "playout_aligned". LiveKit takes it from playbackFinished.synchronizedTranscript. Pipecat takes it from the TTSTextFrames that passed the output transport before the interruption, which counts as aligned only with word-timestamped TTS.
  • generated_text is set when it differs: the TTS input (cascaded) or the item's output transcript (realtime).
  • Fallback: text is the generated text, with text_source tts_input or realtime_output_transcript, which may overstate what was heard.
  • The realtime example's assistant messages now use playout_aligned; no example shows the fallback.
• Pipecat assistant records:
  • At TTS start the integration records context_id → the LLM span opened at LLMFullResponseStartFrame, which gives produced_by_span_id and voice.output_message_id.
  • The record gets its msg_<uuid> id, and committed: true, when the assistant aggregator appends the message.
  • A context without an append follows the tool-only and cancelled-speech rule.
  • Realtime works the same way, using the provider's text frames and the response span.
• Minor fixes:
  • The Lingua spec's expected item_a1 now has seq 2 and includes committed, text_status and text_source.
  • A late transcript upgrade now sets transcript_final_ms.
  • The seq rule is stated as intended: user records are numbered at VAD start, assistant records at SpeechHandle creation (Pipecat: TTSStartedFrame).

Checks

• SDK (js/):
  • vitest: 15 passed;
  • tsc --noEmit on both configs: passed;
  • eslint: clean;
  • prettier: clean.
• Lingua: only the spec Markdown changed, so cargo checks were skipped; they last passed at 243922a0.
• git status is clean in both worktrees.

Update 2026-10-12: validation notes

• Lingua spec, failures and warnings, aligned with zod:
  • Lingua now only warns, keeping the entry as logged, for role-mismatched fields, user messages that are interrupted or unplayed, interrupted messages without interrupted_at_ms, and end_ms earlier than start_ms. Zod still rejects all of these at the SDK.
  • "Unplayed with no generated text" is defined as no text while text_status is not not_captured.
• Pipecat pairing: a TTS context is paired with the most recent LLM span whose output feeds it. That span may already have ended.
• Open assistant entries:
  • text is omitted until playout ends, and partial spoken text is never written.
  • text_status stays missing (or not_captured) until playout ends, when its final value is set.
  • Zod rejects text on an open assistant entry.

Checks

• SDK (js/):
  • vitest: 16 passed;
  • tsc --noEmit on both configs: passed;
  • eslint: clean;
  • prettier: clean.
• Lingua: only the spec Markdown changed, so cargo checks were skipped.
• git status is clean in both worktrees.

Lingua builder 2026-10-12

The Lingua mapping for this option is now implemented in braintrustdata/lingua#491. It was built from the spec by a second engineer, and the open points they settled are written into that spec. This README's status line now says so. Integration capture is still a sketch.

LiveKit source alignment 2026-10-12

Engineers who hadn't written this contract did an emission dry run against the real @livekit/agents 1.9.2 source, plus @livekit/agents-plugin-openai 1.9.2 for realtime items. They mapped every field to the hook that supplies it. The first round found realtime signals the examples had assumed that LiveKit doesn't expose: provider timestamps on realtime speech events, a server-VAD commit event, and a provider truncation point independent of local playout. The changes below fix that, and every capture point is now cited by file:line. Result: PASS on the third round.

Final independent re-run: SDK vitest run src/instrumentation/livekit/voice-schema 16 passed, tsc --noEmit and prettier --check passed; full cargo test -p lingua 1474 passed, 0 failed.

Update 2026-10-12: LiveKit 1.9.2 capture alignment

The LiveKit capture section now cites @livekit/agents 1.9.2 src/ file:line for each hook and event.

• Realtime user bounds (A).
  • input_speech_started/stopped carry no provider times (llm/realtime.ts:16), so bounds are local receipt times, with the new timing_source: "local_receipt".
  • The server-VAD silence is subtracted only when the integration is configured with it.
  • Provider ms (provider_events) remain only for integrations that see raw provider events, such as Pipecat.
  • The realtime example now has item_rt_u1 at 300–3400 and item_rt_u2 at 7600–9000.
• Realtime end of turn (B). There is no commit event: userTurnCompleted returns early when server turn detection is on (agent_activity.ts:3302). end_of_turn_ms is the entry of _realtimeGenerationTaskImpl.
• Truncation (C).
  • provider_audio_end_ms now means "truncate sent to the provider".
  • LiveKit computes it from the local playback position (agent_activity.ts:4789), and only when messageTruncation is on, so it equals the played position: the example now has 2900 with playout stopping at 7900.
  • The fake 300 ms gap is gone. A disagreement is observable only through Pipecat, which sees the provider independently.
• Speech ↔ step ↔ item (D).
  • Records are keyed by speechHandle.id, captured at reply-task entry, not by the turn-level lk.speech_id.
  • produced_by_span_id is the llm_response span started in that handle's reply task.
  • Playback is attributed through activity.currentSpeech at first frame, with the playback hook installed independently of audio capture.
  • The item id comes from wrapping that handle's _itemAdded (speech_handle.ts:741), which LiveKit calls just before _conversationItemAdded (agent_activity.ts:4272/4312/4826).
• Realtime user id and commit (E). At _realtimeGenerationTaskImpl entry the integration diffs realtimeSession.chatCtx.items and pairs the oldest new user item with the oldest open user record. This gives the id and committed: true even without a transcript.
• Context ids (F).
  • Cascaded lists the real instructions item lk.agent_task.instructions.
  • Realtime instructions are not listed.
  • input is logged one message per chat item. This is a customer-visible change for realtime: no appended transcripts and no system message.
  • The manual wrap path reads llmNode args[0].
• Handoff (I). The capture rules follow the pipeline at call start. A handoff between cascaded and realtime is out of MVP scope.
• Lingua.
  • VoiceTimingSource gains LocalReceipt, and the bindings are regenerated.
  • The cascaded and realtime fixtures were rebuilt from the updated examples, with expected output updated (local-receipt bounds, provider_audio_end_ms 2900).
  • The spec gains a "LiveKit 1.9.2 alignment" section.

Checks

• SDK (js/):
  • vitest: 16 passed;
  • tsc --noEmit on both configs: passed;
  • eslint: clean;
  • prettier: clean.
• Lingua (private target dir):
  • make generate-types: passed;
  • cargo test -p lingua: all passed, including 1450 library tests and the 3 voice_message_audio_* fixtures;
  • cargo clippy -p lingua --all-targets -- -D warnings: clean;
  • cargo fmt: applied.
• I confirmed the fixture assertions are live: a deliberately wrong provider_audio_end_ms fails the test.
• git status is clean in both worktrees, and nothing leaked into the sibling worktrees.

Update 2026-10-12: realtime item sources pinned

• Realtime user id and commit.
  • Verified for the OpenAI RealtimeModel (@livekit/agents-plugin-openai 1.9.2): conversation.item.added/created → handleConversationItemCreated (realtime/realtime_model.ts:1409) → remoteChatCtx.insert (:1697), and the chatCtx getter returns remoteChatCtx.toChatCtx() (:589). The user audio item therefore exists before any transcript, and the chat-context diff works.
  • Fallback for plugins without such items (GPT Live duplex and Gemini, unverified): the record keeps vad_<seq>, becomes committed: true when a generation answers it, and is not listed in context ids or input. This is documented as a known limitation: there is no step link.
• Speech ↔ item link.
  • The _itemAdded wrapper accepts only assistant messages; function calls and outputs (agent_activity.ts:4168/4175/4874/4881) are ignored.
  • A realtime handle that streams several messages (:4759, :4826) gets one record per messageId, each with its own seq, and playout is attributed per forwarded message.
  • A played message with no forwarded text (:4805) gets no item and stays committed: false with state complete or interrupted. This is expected.
• Rule changes.
  • committed may go from false to true once, for assistant records written during playout.
  • Zod and the Lingua builder no longer restrict the state of uncommitted agent speech. The Lingua spec lists both changes.
• Minor fixes.
  • Citations corrected: the instructions are prepended at generation.ts:500; InputSpeechStoppedEvent is at llm/realtime.ts:17.
  • provider_audio_end_ms equals the played position except when a non-finite position is clamped to 0 (:4794).

Checks

• SDK (js/):
  • vitest: 16 passed;
  • tsc --noEmit on both configs: passed;
  • eslint: clean;
  • prettier: clean.
• Lingua:
  • cargo test -p lingua: all passed, including 1450 library tests;
  • cargo clippy -p lingua --all-targets -- -D warnings: clean;
  • cargo fmt: applied.
• No fixture values changed.
• git status is clean in both worktrees.

Base: delner/livekit-voice-recording @ 2f184fa.

Schema at a glance 2026-10-12

• What changed: SDK 73d428f6 reorders the README to the shared 14-section skeleton, with a new "Schema at a glance": the logged contract as TS row shapes (matches zod), time and ID rules, abridged cascaded/realtime excerpts checked value-for-value against the examples, and a cascaded-vs-realtime table. Implemented/Sketch is marked per section. Lingua 835d8693: the spec gains a matching "Schema at a glance" (input keys, Trajectory output, recorded vs inferred).
• Gaps fixed: voice.output_message_id back-link exists only for committed messages (README and test said always); selection offsets are segment ms, not call ms; this option writes no audio.selections on turn spans; voice.origin_unix_ms "equals" the recording origin is now "written as", with the mismatch as the clock-offset case; produced_by_span_id is a span_id. Lingua doc comments for generated_text, request_id, Speech.id and VoiceCall order now match the builder.
• No zod, example or fixture changes.
• SDK checks: vitest message-audio.test.ts 16 passed; tsc --noEmit (both configs) passed; prettier clean (repo root). eslint skipped: the file is in eslint's ignore list.
• Lingua checks (doc-comment-only Rust change): cargo fmt --check ok; make generate-types regenerated only Speech.ts, VoiceCall.ts, VoiceMessage.ts doc comments; clippy -D warnings ok; cargo test -p lingua 1474 passed, 0 failed (coordinator re-run).
• Leak grep: only call-0000, the deterministic segment-id format already in the base branches' tests.
• Fourth option: a separate "message ids + timeline events" option was assessed and not added. See the comparison note.

Companion PR: braintrustdata/lingua#491
Other options: speech spans (#2611, braintrustdata/lingua#489); call timeline (#2612, braintrustdata/lingua#490)
Comparison note: shared with Kev in the originating Slack thread; it links all six drafts.

🤖 Generated with Claude Code