Voice schema option: call timeline

[kev (kevchoi)]

AI-created or modified and not human-reviewed in its current form; treat this artifact as provisional. Updated 2026-10-12.

Exploration draft, one of several voice-trace schema options kept for review. Not for merge.

Problem

Voice-trace evidence today is scattered across span names (user_turn, livekit.agent_speaking), vendor keys (contrib.livekit.*) and spans timed on different clocks. Realtime pipelines add transcripts that arrive late or never, and a provider view of an interruption that differs from what the caller heard. Customers can't easily see where time went, who talked over whom, or what evidence is missing.

Proposed contract

Integrations log an append-only call timeline: voice.timeline child rows of the call span, each holding up to 256 voice.events with a per-call seq.

• Timestamps: every event carries at_ms on the recording's call clock and a basis (audio / local / provider).
• Event kinds: speech start/end, transcript, end of turn, response start, tool call/result, truncate, missing, call end.
• IDs: utterance_id, response_id and tool_call_id join events. span_id links into the step tree.
• What the SDK doesn't emit: turns and speech spans. Lingua derives voice_calls[].speech and step links from the log; the app's voice bar is unchanged.

Main tradeoff

This option gives the most precise view of latency, overlap and missing evidence, because everything is on one clock with provenance. The cost is that speech and turns become a Lingua derivation over a new row type. The raw trace tree no longer shows speech by itself, and the SDK must deduplicate hooks that observe the same moment.

Implemented vs sketched

• Implemented:

  • zod schema and inferred types (js/src/instrumentation/livekit/voice-schema/call-timeline.ts, not exported)
  • cascaded and realtime example traces of one fictional call with a tool round trip, a barge-in and a missing realtime transcript (docs/voice-schema/call-timeline/examples/)
  • a test that parses them and checks seq, ordering and start/end pairing

• Sketched in docs/voice-schema/call-timeline/README.md:

  • LiveKit capture (hook → event table, cascaded and realtime)
  • Pipecat capture (frame → event table)
  • Lingua mapping
  • edge cases
  • implementation order

  No instrumentation emits the timeline yet.

Checks

• pnpm exec vitest run src/instrumentation/livekit/voice-schema (in js/): 4 passed.
• pnpm exec tsc --noEmit -p tsconfig.json (in js/): passed.
• pnpm exec eslint src/instrumentation/livekit/voice-schema/: passed.
• prettier --write on the changed files.
• Skipped:
  • e2e scenarios, since no instrumentation changed
  • comparing against experimental reference traces

Update 2026-10-12: semantics gaps resolved

A review against "can an SDK engineer start without inventing essential semantics" found gaps in this option; they are resolved below. This section supersedes the Checks section above where they differ.

• Caller utterances: a caller utterance is one VAD speech segment. speech_end is the real end: the VAD stop time minus the silence delay. LiveKit cascaded capture now uses new onStartOfSpeech/onEndOfSpeech hooks, not the EndOfTurnInfo bounds. Lingua groups utterances into turns through end_of_turn.utterance_ids.
• Turns: a turn is the window between consecutive end_of_turn events, ordered by seq.
  • Every response_start in the window links to the turn's caller utterances. in_reply_to is only a hint, and Lingua warns when it disagrees with the window.
  • A caller utterance missing from every end_of_turn joins the window that contains its start.
  • Lingua does not compute latencies. The app computes them, and the algorithm is documented.
• Transcripts without item ids: each final transcript attaches to the open, or else most recently ended, caller utterance of the current uncommitted turn. Several finals are joined in seq order. A new message event links message_id when the chat item commits.
• Keys: the voice.timeline object is used everywhere. Chunks hold at most 256 events, flush at 64, and carry the backlog. voice.timeline_version (required, which also gates Lingua) and voice.pipeline are now on the call span. A dropped backlog is reported as missing{evidence: timeline}.
• Ordering and recovery:
  • seq is emission order and at_ms is real time; Lingua orders speech by at_ms.
  • Retries reuse chunk_index and seq.
  • Unclosed speech ends at the same speaker's next start, else at call_end, else at the last event.
  • A missing span_id is a warning, as is the absence of a final chunk.
• Selections: the channel comes from the call span's recordings[].sources. Agent speech ends at the played speech_end; truncate is only a marker.
• Realtime:
  • Provider times are anchored at the call ms of the first frame sent after each buffer clear or commit.
  • Gemini Live uses u<n> utterance ids plus a later message.
  • Agent transcripts use playout text in both pipelines, with provider only as a fallback.
• Pipecat:
  • The observer sits at the transport output.
  • Agent start and end come from bytes written, which also gives played_ms.
  • Agent ids are <context_id>/<n>.
  • span_id comes from the SDK's own llm_response span.
  • in_reply_to lists the VAD utterances since the previous end_of_turn.
• Examples: regenerated for the same scenario. Cascaded now shows u2's STT final arriving before the VAD stop is emitted, so seq and at_ms order differ.

Checks:

• SDK: vitest run src/instrumentation/livekit/voice-schema 8/8, tsc --noEmit passed, eslint passed, prettier applied.
• Lingua: cargo test -p lingua processing::trajectory 96/96, --test trajectory 3/3, cargo clippy --all-targets -D warnings passed, fmt applied, bindings regenerated.
• Skipped: e2e, Lingua TS/wasm, typed-boundary check, and comparison with the staging traces.

Independent re-run after these commits: SDK vitest run src/instrumentation/livekit/voice-schema 8 passed and tsc --noEmit passed; full cargo test -p lingua in the Lingua worktree passed (1472 tests, 0 failed).

Confirmation review 2026-10-12

After the gap fixes above, fresh reviewers re-read this option as an engineer about to implement it and gave a pass/fail verdict on whether any essential semantics were still left to invent. Result: PASS on the third round. The follow-up changes below closed the remaining items.

Final independent re-run: SDK vitest run src/instrumentation/livekit/voice-schema 8 passed, tsc --noEmit and prettier --check passed; full cargo test -p lingua 1472 passed, 0 failed.

Update 2026-10-12: realtime caller timing

• Realtime caller utterances:
  • LiveKit input_speech_started / input_speech_stopped carry no item id or audio ms, so the SDK assigns u<n> at speech_start, timed at receipt (local).
  • speech_end is dated at receipt minus the configured server-VAD silence (voice.server_vad_silence_ms on the call span) when known. Otherwise it is the receipt time, which includes provider silence padding.
  • input_audio_transcription_completed attaches by the cascaded rule and then emits message{message_id: itemId}.
  • The realtime example now uses u1/u2, and the README ID rule is fixed.
• Provider timing anchor: a new per-provider table.
  • OpenAI audio_*_ms is cumulative for the session, anchored at the call ms of the first audio frame sent in the session.
  • Gemini Live has no such fields, so it uses local.
  • When the anchor is unknown, local is used.
• Preemptive generation: in_reply_to is required on any response_start emitted before its turn's end_of_turn. Lingua uses in_reply_to whenever it is present; the seq window is only a fallback (Lingua spec §3).
• Minor:
  • LiveKit agent speech_start is emitted at playout start (the first playbackStarted / playbackProgressed).
  • Pipecat byte-count times use basis: audio, and n in <context_id>/<n> is defined as the playout-stretch counter.
  • The VoiceCall.pipeline wording is reconciled: validated, not yet carried.
  • A half_cascade note was added.

Checks:

• SDK: vitest 8/8, tsc --noEmit passed, eslint passed, prettier applied.
• Lingua: cargo test -p lingua --test trajectory 3/3 (fixture updated), clippy --all-targets -D warnings passed, cargo fmt --check passed.
• Skipped: processing::trajectory, because no Rust source changed; e2e; Lingua TS/wasm.

Late realtime transcripts

• Two attachment rules:
  • STT finals attach to the open, or most recently ended, utterance of the current uncommitted turn.
  • Provider (realtime) finals attach FIFO to the oldest committed caller utterance (listed in an end_of_turn) that has no final yet. This holds even when a later barge-in utterance is open.
• Where it's documented: the README's "Transcript to utterance" section and both capture tables. A sentence covers the late final arriving during a barge-in, and the Lingua spec states that Lingua takes the SDK's attachment as given.
• message: follows the same attachment. Utterances without a final never get a message; Lingua message matching tolerates that and relies on the missing event.
• half_cascade: agent ids follow the cascaded <speech_id>/<n> scheme.
• New Lingua warnings:
  • a bad or missing voice.pipeline: warning, field ignored;
  • voice.timeline rows whose call span lacks voice.timeline_version: warning, rows ignored.

Checks:

• SDK: vitest 8/8, tsc --noEmit passed, eslint passed, prettier --check passed.
• Lingua: spec only; no Rust or fixture changes, so cargo was not re-run.

Realtime transcript grouping

• FIFO assumption: each committed end_of_turn group becomes one provider input item. A provider final attaches to the whole oldest group that lacks a final. The text goes on the group's first utterance, and the group's other utterances are marked covered, not missing, so later turns don't shift. This is stated in both the README and the Lingua spec.
• Pipecat realtime message: emitted only for providers with input item ids (OpenAI); none for Gemini Live.
• LiveKit realtime with server turn detection off: caller speech comes from the local VAD hooks (onStartOfSpeech / onEndOfSpeech, local), and end_of_turn comes from LiveKit's own commit (onEndOfTurn).

Checks: SDK vitest 8/8, tsc --noEmit passed, eslint passed, prettier --check passed. Lingua: spec only; cargo not re-run.

Lingua builder 2026-10-12

The Lingua mapping for this option is now implemented in braintrustdata/lingua#490. It was built from the spec by a second engineer, and the open points they settled are written into that spec. This README's status line now says so. Integration capture is still a sketch.

LiveKit source alignment 2026-10-12

Engineers who hadn't written this contract did an emission dry run against the real @livekit/agents 1.9.2 source, plus @livekit/agents-plugin-openai 1.9.2 for realtime items. They mapped every field to the hook that supplies it. The first round found realtime signals the examples had assumed that LiveKit doesn't expose: provider timestamps on realtime speech events, a server-VAD commit event, and a provider truncation point independent of local playout. The changes below fix that, and every capture point is now cited by file:line. Result: PASS on the second round.

Final independent re-run: SDK vitest run src/instrumentation/livekit/voice-schema 8 passed, tsc --noEmit and prettier --check passed; full cargo test -p lingua 1475 passed, 0 failed.

Update 2026-10-12: aligned with @livekit/agents 1.9.2 sources

The LiveKit capture table now cites @livekit/agents 1.9.2 src/ file:line for every hook it relies on.

• Realtime caller speech (A):
  • input_speech_started / input_speech_stopped carry no provider ms, so caller speech is timed at local receipt.
  • The end subtracts voice.server_vad_silence_ms only when that option is passed to the integration. Otherwise the end includes the silence padding.
  • With server turn detection on, provider events win and local VAD is not used. With it off, the local VAD hooks are used, minus inferenceDuration as LiveKit does for native user_speaking.
• Realtime end of turn (B): there is no commit event, so end_of_turn is emitted at _realtimeGenerationTaskImpl entry for a generation that answers caller input. In the realtime example this moved from 3400 to 3600. Tool continuations emit none.
• Truncate (C): LiveKit computes audioEndMs from the same playbackPosition, and only when capabilities.messageTruncation is set. The realtime example now shows the truncate at 7900 with 2900, equal to played_ms. The README no longer presents a 300 ms provider/playout gap as LiveKit behavior; only integrations that observe the provider independently (Pipecat) can show one.
• Agent linking (D):
  • response_id = SpeechHandle.id, captured at reply-task entry, and agent utterance_id = <SpeechHandle.id>/<n> in both pipelines (speech_2/0, speech_4/0). lk.speech_id is not used.
  • response_start.span_id = the llm_response span started inside that handle's reply task.
  • Playback is attributed through activity.currentSpeech, and the playback hook is installed independently of audio capture.
  • The realtime output item id is linked by a later agent message.
• Cascaded end of turn and preemption (H):
  • end_of_turn is emitted only when onEndOfTurn resolves true and commits, not when the turn was skipped while scheduling is paused. Its at_ms is the resolve time.
  • in_reply_to is snapshotted in onPreemptiveGeneration and carried on that handle's response_start, including discarded preemptive responses.
• Handoff (I): voice.pipeline is the pipeline at call start. A handoff that changes the LLM type is out of MVP scope.
• Lingua:
  • Builder fixtures call-timeline-*.json now use the new example spans, and their expected voice_calls were regenerated. Only ids and event values changed: speech intervals, channels and step_ids are identical.
  • The event round-trip fixtures were updated.
  • The spec gained a "LiveKit 1.9.2 alignment" section.

Checks:

• SDK: vitest 8/8, tsc --noEmit passed, eslint passed, prettier --check passed.
• Lingua (private target dir): cargo test -p lingua 1475 passed / 0 failed, cargo clippy --all-targets -D warnings passed, cargo fmt --check passed.
• Skipped: e2e, Lingua TS/wasm, and running against a live LiveKit session.

Capture rule precision (round 2)

• message: emitted only from the AgentSession._conversationItemAdded hook.
  • Cascaded user items come from the normal commit at agent_activity.ts:3975 (only when speechHandle.scheduled), not from commitSkippedUserTurn (:3272).
  • Realtime user items come from :2054, and assistant items from :4273 / :4827.
  • Transcription events emit only transcript.
• Cascaded end_of_turn:
  • The resolve time is recorded when onEndOfTurn returns true and neither schedulingPaused (:1230) nor newTurnsBlocked (:550) was set before the call.
  • The event is emitted when the reply is scheduled (the :3975 commit). The cascaded example now has the turn-1 transcript before end_of_turn in seq.
  • Turns that userTurnCompleted drops emit no end_of_turn: AMD skipReply (:2697, :3316-3319), non-interruptible (:3328-3333) and StopResponse (:3388-3390). Their utterances fall back to the turn open at their start.
• response_start → span:
  • Cascaded: through the SDK's own async-local context set at _pipelineReplyTaskImpl entry, because LiveKit's speechHandleStorage (agent.ts:70) is not exported.
  • Realtime: through the inferenceSpan argument (realtime_inference, :4478).
  • The preemptive guard is cited at :2440.
• Realtime end_of_turn: excludes utterances still open at task entry.
• Lingua:
  • The cascaded builder fixture and the event fixture were reordered and their expected output regenerated; speech is unchanged.
  • The spec alignment section gained the two end_of_turn notes.

Checks:

• SDK: vitest 8/8, tsc --noEmit passed, eslint passed, prettier --check passed.
• Lingua: cargo test -p lingua 1475 passed / 0 failed, clippy -D warnings passed, fmt --check passed.

Base: delner/livekit-voice-recording (2f184fa).

Schema at a glance 2026-10-12

• What changed (SDK 3d5118e4, fb273cc0, a10e64cb; Lingua aca261a5, 95849123): the SDK README follows the shared 14-section skeleton and opens with "Schema at a glance": the logged row shapes as one TS block, time and relationship summaries, trimmed cascaded/realtime JSONC excerpts checked value-for-value against the examples, and a cascaded-vs-realtime payload table. The Lingua spec opens with the input keys read and the VoiceCall/Speech/VoiceEvent output shapes, each marked recorded or inferred.
• Decision E applied: realtime caller message is emitted at the answering generation's entry, so the barge-in u2 maps to item_u2 with no transcript. Provider finals carry message_id, and FIFO is only the fallback. The realtime example and both Lingua realtime fixtures were regenerated; speech and step_ids are unchanged. A new test asserts that every committed caller utterance has a message.
• Gaps fixed: voice.timeline_origin_unix_ms required iff no recording; final: false reserved; transcript.message_id defined; missing emitters, with audio/tool_result reserved; agent-only response_id/interrupted/played_ms; caller speech_end without a subtracted silence; row metrics vs call ms; tool_call_id join; truncate.audio_end_ms origin; stale Lingua spec lines (VoiceCall.pipeline, TRAJECTORY_METADATA_FIELDS).
• Realtime retimed (aligned with the other options): caller speech_end is the receipt time including the 500 ms server-VAD padding (u1 3400, u2 9000; voice.server_vad_silence_ms removed from the example). Turn 2: end_of_turn/generation/message 9000, reschedule 9300–9700, final audio 9700–12500. Barge-in 7900/2900 unchanged. Lingua realtime fixture selections now end at 3400/9000.
• Open question added: voice.context_message_ids / voice.output_message_id on LLM spans as the recommended extension for click-message-to-audio.
• Checks: vitest 10/10, tsc --noEmit passed, eslint passed, prettier --check passed (SDK). cargo test -p lingua 1475 passed / 0 failed (Lingua).
• Skipped: cargo fmt/clippy and make generate-types, because no Rust source changed (only spec and fixture JSON). Also skipped: e2e and a live LiveKit run.
• Independent page review: a separate reviewer checked a call-timeline explainer page against this README, the zod, the examples, the Lingua spec and the LiveKit 1.9.2 source. It found one README nit, fixed in a10e64cb: call span input.audio is optional when capture is off.
• Fourth option: a separate "message ids + timeline events" option was assessed and not added. See the comparison note.

Companion PR: braintrustdata/lingua#490
Other options: speech spans (#2611, braintrustdata/lingua#489); message audio (#2613, braintrustdata/lingua#491)
Comparison note: shared with Kev in the originating Slack thread; it links all six drafts.

🤖 Generated with Claude Code