Skip to content

feat(ui): voice dictation on AI prompt inputs - #97

Merged
kartikmanimuthu merged 1 commit into
master-v1from
feat/speech-to-text-prompt-inputs
Jul 25, 2026
Merged

kartikmanimuthu merged 1 commit into
master-v1from
feat/speech-to-text-prompt-inputs

Conversation

@kartikmanimuthu

Copy link
Copy Markdown
Collaborator

Adds a mic button to every surface where a natural-language prompt is typed, backed by the browser's Web Speech API. No new dependencies, no API route, no AWS cost.

What's new

Module Purpose
hooks/use-dictation.ts Binds speech recognition to an already-controlled text field
components/voice/mic-button.tsx Ghost icon button; pulses and reports aria-pressed while listening
components/voice/dictation-textarea.tsx Drop-in Textarea wrapper with the mic in the bottom-right corner

useDictation snapshots the field value when a session starts, then rewrites snapshot + finalSoFar + interim on each result event. Words appear live as you speak, and the interim tail is swapped for the final transcript rather than duplicating after it. Continuous + interim recognition; the mic is released on unmount, the session stops when the field disables, and not-allowed / audio-capture / network errors surface as toasts while no-speech / aborted stay silent. isSupported resolves in an effect so SSR and hydration agree.

Surfaces wired (9)

AI Ops composer (next to the existing Enhance-prompt wand, sharing its 2000-char cap) · Inventory Ask AI · KB chat · Deep Agent chat · Agent Ops new-run dialog · scheduled-task Objective · both HIL respond textareas · chat clarification card.

Dictation appends to existing text everywhere rather than clobbering it. On browsers without the API (Firefox) the mic renders nothing at all instead of showing a dead control.

Incidental fix

Deep Agent chat auto-resized its textarea inside the onChange handler off e.target, so programmatic writes never resized it. Moved to an effect keyed on the value — this also fixes the box not shrinking when cleared on send.

Verification

  • 21 new tests pass — append-not-replace, interim→final swap, cumulative multi-segment accumulation, maxLength clamping, unsupported browser, disabled-mid-session, silence timeout, permission errors, unmount cleanup
  • 169/169 pass across components/agent, components/voice, hooks
  • tsc --noEmit clean on every touched file; ESLint clean on all three new files
  • next build succeeds

The full suite reports 59 failures across 19 files — all pre-existing, none in files touched here. They are DB-backed tests failing on Can't reach database server at localhost:5432 plus the known mock-harness set.

Note on the engine

Chrome routes captured audio to Google's servers for recognition, worth weighing for prompts that name real account IDs and ARNs. The engine is isolated inside useDictation, so a server-side model (self-hosted Parakeet via sherpa-onnx-node, or AWS Transcribe streaming) can be swapped in later by changing one file — the nine call sites, MicButton, and DictationTextarea all stay as they are.

🤖 Generated with Claude Code

Adds a mic button to every surface where a natural-language prompt is
typed, backed by the browser's Web Speech API — no new dependencies, no
API route, no AWS cost.

New modules:
- hooks/use-dictation.ts — binds speech recognition to an already-controlled
  text field. Snapshots the value at session start, then rewrites
  `snapshot + finalSoFar + interim` on each result event, so words appear
  live while the interim tail is swapped for the final transcript instead of
  duplicating after it. Continuous + interim recognition, mic released on
  unmount, session stopped when the field disables, and permission/capture
  errors surfaced as toasts while no-speech/aborted stay silent.
- components/voice/mic-button.tsx — ghost icon button that pulses and reports
  aria-pressed while listening; renders nothing on browsers without the API
  rather than showing a dead control.
- components/voice/dictation-textarea.tsx — drop-in Textarea wrapper that
  tucks the mic into the bottom-right corner.

Wired into nine surfaces: AI Ops composer, Inventory Ask AI, KB chat, Deep
Agent chat, Agent Ops new-run dialog, scheduled-task objective, both HIL
respond textareas, and the chat clarification card. Dictation appends to
existing text everywhere rather than replacing it.

Also fixes Deep Agent chat auto-resizing its textarea inside the onChange
handler off e.target, which meant programmatic writes never resized it. Moved
to an effect keyed on the value, which also fixes the box not shrinking when
cleared on send.

The engine is isolated inside useDictation, so a server-side model (Parakeet,
Transcribe) can be swapped in later without touching any call site.

Tests: 21 new (append-not-replace, interim to final swap, cumulative
multi-segment accumulation, maxLength clamping, unsupported browser,
disabled-mid-session, silence timeout, permission errors, unmount cleanup).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kartikmanimuthu
kartikmanimuthu merged commit e7b14e0 into master-v1 Jul 25, 2026
1 check failed
@kartikmanimuthu
kartikmanimuthu deleted the feat/speech-to-text-prompt-inputs branch July 25, 2026 04:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant