The focus-in handler registers a composer as the active dictation target
but never clears it on focus-out, so it stays active until unmount. Since
AppShell dispatches the primary-D shortcut globally (outside settings),
pressing ⌘D while focus is in quick search, a create-channel dialog, or
any other mounted input could start mic capture and append transcript
into the background composer's draft. Gate the shortcut on focus still
being contained in the composer element at keydown time.
A timer-tick flushAudioBatch() whose native IPC is still awaiting could be
left un-awaited by stopRecording()/cleanup(), which only awaited the flush
they kicked off themselves. Stopping in that window let stop_dictation fire
before an earlier flush's chunks were enqueued native-side, dropping late
chunks and losing transcript words.
Serialize flushes through a flushChainRef promise tail: each flush drains the
batch synchronously (distinct FIFO payload) then chains its IPC send after any
prior in-flight flush. stopRecording()/cleanup() await the chain tail (the
empty-batch path now returns the tail), guaranteeing every started flush is
fully enqueued before stop. Chain is reset per session in startRecording.
Bump MessageComposer.tsx file-size override (+15) for the dictation wiring
that pushed it 1033->1049, matching the existing load-bearing-growth pattern.
cleanup() tore down resources but never set startAbortedRef, so a
composer unmounting while startRecording() awaited start_dictation /
listen / getUserMedia / addModule would resume the async start against a
torn-down instance — opening the mic/worklet and leaving the native
session and event listeners running after the component was gone. Set
the abort flag first in cleanup(); a fresh startRecording clears it
before its first await, so subsequent starts are unaffected.
Address two codex P2s on cross-session/late-arrival races:
1. Cancel dictation during the final transcribing window. The draftKey
cancel effect only checked isRecording/isStarting, but stopRecording()
keeps the transcript listener alive (isTranscribing=true) until the
native stopped event. A channel/thread switch during that grace window
didn't cancel, so a late transcript leaked into the newly restored
draft. Include isTranscribing in the ownership check.
2. Scope push_dictation_audio to the owning session. Late flush chunks
from a just-stopped session could be fed into a newer session's engine
and transcribed into the new draft. Prepend an 8-byte LE u64 session
header to each raw audio payload; native only feeds audio whose header
matches the active session_id and drops stale chunks.
Two P2 review findings:
1. Resume the AudioContext before wiring the worklet. The WebView can create
the context in the 'suspended' state after the async start path (mic prompt
etc.); a suspended context never pulls the worklet's process(), so no PCM
reaches push_dictation_audio while the UI shows an active recording. Mirror
the huddle capture path and resume() when suspended.
2. Chunk merged PCM under the native 100 KB IPC cap. push_dictation_audio
rejects any raw batch over MAX_AUDIO_BATCH_BYTES (100 KB = ~0.53s at 48kHz
f32 mono). If the batch timer is delayed (main-thread stall) the merged
buffer can exceed the cap, native rejects it, and the .catch silently drops
the whole chunk. Split into <=24,000-sample (96 KB) chunks sent in order.
The unmount cleanup handler (registered for every useLocalDictation
instance) called stop_dictation unconditionally, so a non-recording
composer unmounting (e.g. closing a thread reply composer) while another
composer was actively dictating would tear down the singleton engine and
drop the real recording's final transcript.
Scope every frontend stop_dictation call to a specific session:
- cleanup() passes nativeSessionRef.current (0 for an instance that never
started; native session IDs start at 1, so a never-recorded composer's
stop can never match the live session and correctly no-ops).
- startRecording bail paths pass the just-started sessionId, robust against
a concurrent restart during an await window.
- error path and cancelRecording drop their redundant unconditional stop and
rely on cleanup()'s scoped stop.
Address two P2 review comments:
- stt_engine: the worker broke out of its loop as soon as the shutdown flag
was set, skipping any audio already enqueued by the caller's final
push_dictation_audio flush. The shutdown flush then only transcribed older
speech and dropped the last words. Now drain and process everything still
in the audio channel before the final flush.
- dictation/useLocalDictation: stop_dictation now takes an optional session
id and only tears down the engine when it matches the currently-stored
session_id. stopRecording captures its session and passes it to the
deferred (post-flush) stop, so restarting during the Transcribing grace
window no longer lets the stale stop kill the newly-started session.
Unconditional callers (cancel, unmount, abort-bail, start) pass None.
Address two P2 review comments on useLocalDictation:
- Recheck startAbortedRef after audioWorklet.addModule() resolves. If
stop/cancel fired during the module load, the start path previously
continued to wire up the worklet, start the flush timer, and set
isRecording=true, leaving dictation running after it was stopped.
Now bails and tears down the mic/context it opened.
- Clear worklet.port.onmessage on stop/cleanup so PCM messages still
queued on the main thread are dropped instead of appended to the reused
audioBatchRef after the final flush. Also reset the batch on cleanup and
at the start of each new recording so a stale batch can't leak into the
next session/draft.
Each listener now compares event.payload.session against the local
`sessionId` const (captured by the closure) instead of reading from
the mutable `nativeSessionRef.current`. This prevents stale listeners
from a previous session from matching the new session's ID after the
ref is updated.
Also unregisters any lingering listeners at the start of a new session
before registering fresh ones, ensuring no duplicate handlers exist.
The native dictation forwarder now includes a monotonically increasing
session ID in every emitted event payload:
- Transcript events: { text: string, session: number }
- State events: { state: 'started'|'stopped', session: number }
start_dictation returns the session ID so the frontend stores it in
nativeSessionRef before registering listeners. Listeners compare the
event's session field against nativeSessionRef.current and ignore
mismatches, definitively preventing stale transcripts from a previous
session's forwarder from leaking into a new draft.
Also reordered the start flow: engine start (which returns the session ID)
now happens before listener registration, eliminating the timing gap where
listeners could accept events with an unknown session.
Addresses 2 review comments:
1. **Tag dictation events by session** — each startRecording increments a
session counter. Transcript and state event listeners check the session ID
and ignore events from a previous session's forwarder, preventing stale
transcripts from leaking into a new draft.
2. **Abort pending starts** — stopRecording and cancelRecording set an abort
flag. startRecording checks this flag after each async await (listener
setup, engine start, getUserMedia) and bails early if the user already
stopped/cancelled, preventing zombie worklets and timers.
Addresses 3 new review comments:
1. **Avoid cancelling another composer's dictation** — the draftKey effect
now checks isRecordingRef before calling cancelRecording(), so switching
channels in one composer won't kill a recording in another.
2. **Stop native dictation on unmount** — cleanup() now calls
invoke('stop_dictation') so the native SttEngine doesn't linger when
the hook unmounts mid-recording (navigation, composer close).
3. **Await final audio flush before stopping STT** — stopRecording() and
cleanup() now await the flushAudioBatch() promise before invoking
stop_dictation, ensuring the last batch of audio arrives at the native
engine before it shuts down and flushes its speech buffer.
Addresses all 8 unresolved review threads on PR #1511:
1. **Accumulate local transcript segments** (already fixed) — each streaming
partial resets lastTranscriptRef so segments are appended, not replaced.
2. **Flush local speech before shutdown** (already fixed) — worker flushes
remaining speech_buf on loop exit.
3. **Scope dictation shortcut to one composer** — new activeComposer module
tracks which composer instance last received focus. The ⌘D handler only
dispatches to the active instance, preventing duplicate recordings when
both channel and thread composers are mounted.
4. **Keep listening through user-stop finalization** (already fixed) —
stopRecording no longer calls cleanup(); event listeners stay alive until
the native 'stopped' event arrives with the final transcript.
5. **Refresh dictation availability after model downloads** — availability
check now polls every 5s until the model is ready, then stops. Covers
the fresh-install case where the model downloads in the background.
6. **Stop native engine after startup failures** — catch block now calls
invoke('stop_dictation') before cleanup so the native SttEngine doesn't
linger when mic permission is denied or AudioWorklet setup fails.
7. **Prevent disabled composers from shortcut-starting** — keydown handler
checks disabledRef/isSendBlockedRef before starting dictation.
8. **Batch dictation audio before IPC** — worklet frames are accumulated in
a Float32Array batch and flushed every 100ms (~10 IPC calls/s instead of
~375). Also removed the erroneous worklet→destination connection that was
playing mic audio back through speakers.
Additionally:
- Removed the entire dead OpenAI relay proxy (transcribe.rs, routes, config
fields, .env.example entries, reqwest dep, bridge.rs visibility widening).
- Scoped the keyup event to only fire when ⌘D keydown was actually dispatched
(no more spurious events on normal 'd' typing).
Three fixes for the push-to-talk dictation experience:
1. **Streaming partial transcripts**: The STT engine now flushes a partial
transcript every ~2 seconds of continuous speech (DICTATION_PARTIAL_FLUSH_SAMPLES).
Previously, text only appeared after a silence gap or when the user released
the key. Now text streams into the composer while you're still speaking.
2. **Final flush on shutdown**: When the STT worker exits (key released or
stop_dictation called), it flushes any remaining speech buffer before
shutting down. Previously, speech accumulated since the last silence gap
was silently discarded.
3. **Suppress macOS accent picker**: Holding ⌘D triggered the macOS
press-and-hold accent character popup (showing ð, đ, etc.). Fixed by
setting ApplePressAndHoldEnabled=false for the app's bundle ID at startup.
Also fixes the event ordering: the 'dictation-state: stopped' event is now
emitted by the forwarder task after all pending transcripts have been
delivered, ensuring the frontend receives the final text before the stopped
signal.
- ⌘D is now push-to-talk: hold to record, release to stop.
AppShell dispatches keydown/keyup events; useComposerDictation
starts on keydown and stops on keyup.
- Button click still toggles (click mic to start, click stop to end).
- Mic icon swaps to a filled stop square while recording.
- Shortcut description updated to reflect hold-to-talk behavior.
- Remove useRealtimeDictation, realtimeAudio, realtimeBufferWorklet,
and transcribeSession API — dictation is now local-only (Parakeet).
- useDictation simplified to use only useLocalDictation.
- Add ⌘D (Ctrl+D on Windows) global shortcut to toggle dictation.
- DictationButton tooltip shows 'Voice Dictation ⌘D' with kbd styling
matching the search bar's ⌘K pattern.
- Shortcut registered in keyboard-shortcuts.ts settings list.
- AppShell dispatches custom event; useComposerDictation listens.
Phase 1: Extract reusable SttEngine from huddle/stt.rs into stt_engine.rs.
The core STT logic (resample 48→16 kHz, earshot VAD, sherpa-onnx Parakeet
inference) is now a standalone component configurable via SttEngineConfig.
huddle/stt.rs becomes a thin wrapper that passes huddle-specific flags
(TTS barge-in, PTT gating). drain_until_shutdown moves to stt_engine and
is re-exported by huddle/mod.rs for backward compat.
Phase 2: Add Tauri dictation commands (start_dictation, stop_dictation,
push_dictation_audio, get_dictation_status) that create a standalone
SttEngine instance with dictation-tuned settings (longer silence threshold,
no TTS/PTT flags). Transcribed text is emitted to the frontend via
'dictation-transcript' Tauri events.
Phase 3: Add useLocalDictation hook that captures mic audio via AudioWorklet
and sends raw PCM to the native STT engine. useDictation now routes to
local STT when available (offline, no API key), falling back to cloud
(OpenAI Realtime via relay) when the model isn't downloaded.
Key wins:
- Works fully offline — no BUZZ_OPENAI_API_KEY needed
- Self-hosters get dictation for free
- Lower latency (no network round-trip)
- No relay billing concern
- Cloud fallback preserved for higher accuracy
OpenAI emits conversation.item.input_audio_transcription.failed when an
item cannot be transcribed (e.g. audio_unintelligible). Previously this
event was dropped by the type filter, leaving isTranscribing stuck true
with no user feedback after a preceding buffer commit.
Now the failed event is treated as terminal: clears the transcribing
indicator and surfaces the error message via toast.
1. Merge session + SDP into single POST /transcribe/connect
The two-step flow (POST /transcribe/session → POST /transcribe/sdp) stored
the OpenAI client secret in an in-memory DashMap, which breaks in HA
deployments where the two requests may land on different relay replicas.
Replaced with a single POST /transcribe/connect that accepts the SDP offer,
mints the OpenAI session, proxies the SDP exchange, and returns the SDP
answer + model — all in one request. No server-side session state is needed
between requests, so this works correctly across any number of replicas.
Removed the transcribe_sessions DashMap from AppState entirely.
2. Clear isTranscribing even when final text is unchanged
When a TRANSCRIPT_COMPLETED_EVENT produces no text change (final matches
accumulated deltas), the early-return in handleRealtimeEvent skipped the
setIsTranscribing(false) call. This left the 'Transcribing…' indicator
stuck indefinitely. Fixed by updating the transcribing flag before the
text-change early return.
1. Proxy SDP exchange through the relay (transcribe.rs)
The relay no longer returns the raw OpenAI client secret to the desktop
client. Instead, POST /transcribe/session returns an opaque session ID,
and the new POST /transcribe/sdp endpoint accepts the client's SDP offer,
looks up the cached secret, forwards it to OpenAI, and returns the SDP
answer. This prevents a compromised client from reusing the bearer token
to open non-transcription Realtime sessions under the operator account.
2. Allow send during the transcribing grace window (MessageComposer.tsx)
Previously, pressing Enter/Send while isTranscribing was true (during the
3s grace window after user-stop) blocked the send entirely. Now the send
proceeds with whatever content is already in the composer (transcript
deltas have been applied incrementally) and cancels the dictation run so
late events don't refill the cleared composer.
Two fixes:
1. (transcribe.rs) expires_after must be an object with anchor + seconds
fields per OpenAI's client-secret schema, not a bare integer. The bare
number would cause OpenAI to reject the request with a 400, making
dictation fail with a 502 from the relay.
2. (useRealtimeDictation.ts) The user-stop grace window (3s delayed
teardown with run kept valid) now applies to ALL models, not just
manual-commit ones. For server-VAD models, the final VAD completion
event may still be in-flight when the user clicks the mic to stop —
without the grace window, that event was dropped and the tail of the
dictated message disappeared.
Add expires_after: 60 (seconds) to the OpenAI Realtime client-secret
request payload. This limits the reuse window of minted secrets — a
client must establish its WebRTC connection within 60s, and cannot reuse
the secret to open additional sessions after that. Without this, the
default 600s TTL allowed a compromised or malicious client to bypass the
per-pubkey rate limiter by reusing a single minted secret for many
concurrent sessions.
Distinguish user-initiated stop (mic button) from send/navigation cleanup:
- stopRecording(): user stop — keeps the run valid during the 3s grace
window so the final commit's transcript is delivered to the composer.
- cancelRecording(): send/edit-save/navigation — immediately invalidates
the run so late transcripts cannot refill the cleared composer.
MessageComposer's stopDictationRef now uses cancelRecording (called on
send and edit-save). useComposerDictation's draftKey/disabled effects
also use cancelRecording. The user-facing toggleRecording/stopRecording
path preserves the final transcript for short recordings.
Address two review comments:
1. (useRealtimeDictation.ts) Increment activeRunIdRef immediately in the
manual-commit cleanup path so late transcript events from the 3s grace
window are rejected by handleRealtimeEvent. Previously, the run stayed
valid during the timeout, allowing transcripts to write back into the
composer after a send, edit-save, or navigation.
2. (transcribe.rs) Replace as_object_mut().unwrap() with a branch that
builds the correct JSON literal directly. Avoids introducing an unwrap
in a production path per AGENTS.md rules.
When BUZZ_TRANSCRIPTION_MODEL is a realtime-whisper variant, the relay
now omits server_vad from the session config. Without server VAD, OpenAI
buffers audio indefinitely until a manual input_audio_buffer.commit is
sent. This commit adds the client-side counterpart:
1. Export commitAudioBuffer() and requiresManualCommit() from
realtimeAudio.ts.
2. On session creation, track whether the model requires manual commit.
3. After flushing the pre-connection buffer, commit immediately and start
a 2s periodic commit interval so streaming transcripts flow during
recording.
4. On stop, send a final commit before teardown and keep the data channel
open briefly (3s) to receive the last transcript response.
5. Clear the commit interval on cleanup.
Without this, realtime-whisper sessions would stream/buffer audio but
never produce transcripts because no commit was ever sent.
Address two review comments:
1. /transcribe/status now returns configured: false for non-members on open
relays, preventing the mic button from appearing for users who would get
a 403 on session creation. Uses the same require_relay_member check.
2. The OpenAI Realtime session payload now omits turn_detection for
realtime-whisper models (which require manual audio commit per OpenAI
guidance). Other models continue to use server_vad.
Adds unit tests for the model-aware payload builder.
1. P1 — Force NIP-98 signed auth for /transcribe/* endpoints regardless of
BUZZ_REQUIRE_AUTH_TOKEN. The X-Pubkey dev fallback is spoofable, so a
billable endpoint must always require cryptographic proof of identity
before the membership check trusts the pubkey.
2. P2 — DictationButton now allows the stop action whenever isRecording is
true, even during startup (mic live but SDP exchange in progress) or
when the composer is disabled. Only blocks the button when idle and
disabled, or when startup hasn't captured the mic yet.
1. P1 — Require relay membership for billable /transcribe/session even on
open relays. Added require_relay_member() that always checks actual
membership (with NIP-OA fallback) regardless of the
BUZZ_REQUIRE_RELAY_MEMBERSHIP setting. Prevents arbitrary NIP-98 signers
from minting metered OpenAI sessions on the operator's bill.
2. P2 — Keep stop control usable while recording. DictationButton now
allows the stop action even when the composer is disabled — only
*starting* a new recording is blocked. Additionally, useComposerDictation
auto-stops the active session when the composer becomes disabled
mid-recording (e.g. channel becomes read-only).
3. P2 — .expect() already removed in prior commit (openai_client() returns
Result and propagates errors). No additional change needed.
4. P2 — Stop dictation before edit saves already addressed in prior commit
(stopDictationRef.current() at line 540). No additional change needed.
- Stop active dictation run before the edit branch clears/saves, matching
the normal-send path. Prevents late transcript events from writing back
into the restored or fresh draft after an edit save.
- Replace .expect() in openai_client() with proper error propagation via
Result, avoiding a panic if TLS backend initialization ever fails.
Addresses review feedback on PR #1511.
Addresses Wes's review feedback on PR #1511:
1. Per-pubkey rate limit on POST /transcribe/session (5/min, configurable
via BUZZ_TRANSCRIBE_SESSIONS_PER_MINUTE). Each session opens a metered
OpenAI Realtime connection on the operator's bill.
2. Auto-submit disabled by default (DEFAULT_AUTO_SUBMIT_PHRASE = '').
The infrastructure for configurable phrases remains in place and can
be wired to a user setting later.
Also:
- Move BUZZ_TRANSCRIPTION_MODEL into Config (consistent with other knobs)
- Use a shared reqwest::Client via OnceLock (connection pooling)
- Stop dictation on manual send (stopDictationRef wiring)
When the user manually sends (Enter/click) while recording, the submit
flow already calls stopRecording(). However, queued data channel messages
could still fire handleRealtimeEvent after the peer connection teardown
began, writing stale transcript text back into the now-empty composer.
Fix: pass the run ID captured at startRecording into the data channel
message handler closure. handleRealtimeEvent now checks activeRunIdRef
against the captured run ID and drops events from stale runs. This
guarantees that once stopRecording() increments the run counter, no
further transcript events from that session can mutate composer state.
Handle input_audio_buffer.committed events to register items in the
correct utterance order using previous_item_id before any transcript
events arrive. This ensures that when completions for different turns
arrive out of order (or when only completions are sent without deltas),
the composer reconstructs multi-utterance dictation in the correct
sequence rather than event-arrival order.
Added tests for committed-order preservation, out-of-order completions
with pre-registered order, and completion-only flows.
Signed-off-by: klopez4212 <klopez4212@gmail.com>
The OpenAI client_secrets endpoint expects the body as
{ session: { type, audio: { input: { transcription, turn_detection } } } }
not as top-level fields. Also moves turn_detection under audio.input per
the Realtime transcription guide.
Signed-off-by: klopez4212 <klopez4212@gmail.com>
- Relay: restructure OpenAI client-secrets payload to use the current
typed transcription schema (audio.input.transcription) instead of the
deprecated top-level input_audio_transcription field.
- realtimeAudio: insert space separators between transcript items when
neither the preceding nor following text has whitespace, preventing
multi-utterance runs from merging into unreadable text.
- useDictation: remove premature setText('') after auto-submit — the
send flow handles clearing on success, so dictated text survives if a
mention dialog opens or the send is blocked.
Signed-off-by: klopez4212 <klopez4212@gmail.com>
When the composer's draftKey changes (channel or thread switch), stop any
active dictation session so transcript events from a stale WebRTC connection
don't leak into the wrong draft.
Signed-off-by: klopez4212 <klopez4212@gmail.com>
- Use OpenAI typed transcription session format (type: "transcription")
instead of legacy realtime fields that would fail or produce no transcripts
- Sync editor content via syncContentRef before merging dictation text so
manually typed prefixes are preserved when dictation starts
- Read send-blocked state from refs at transcript time so uploads prevent
auto-submit from clearing the composer
Signed-off-by: klopez4212 <klopez4212@gmail.com>
- Switch relay from /v1/realtime/sessions to /v1/realtime/client_secrets
with the wrapped { session: { ... } } request shape per OpenAI's current
WebRTC guide. The old endpoint returns non-2xx, breaking dictation.
- Redesign TranscriptSegmentState to track per-item segments keyed by
item_id. Completed events for different turns can arrive out of order;
reconciling by item_id preserves utterance ordering and prevents text
reordering or partial-turn drops during fast consecutive speech.
Signed-off-by: klopez4212 <klopez4212@gmail.com>
- Add nonce tag to NIP-98 auth events to prevent replay rejection when
multiple components call /transcribe/status in the same second.
- Wire dictation text into both the Tiptap editor and contentRef via
setComposerContent + setEditorContentRef, so dictated text actually
appears in the composer and is serialized on submit.
- Call submitMessageRef.current() synchronously in onSend instead of via
queueMicrotask, ensuring the editor content is consumed before the
subsequent setText('') clears it.
- Replace naive append-based transcript merging with segment-aware state
tracking (TranscriptSegmentState). Delta events accumulate into
pendingDelta; completed events replace accumulated deltas with the
finalized text, preventing duplication.
Signed-off-by: klopez4212 <klopez4212@gmail.com>
Both /transcribe/status and /transcribe/session now require NIP-98
authentication and relay membership (with NIP-OA fallback), matching
the security posture of /events, /query, and /count.
Promotes verify_bridge_auth, check_nip98_replay, and nip98_expected_url
to pub(crate) so the transcribe module can reuse them without duplication.
Signed-off-by: klopez4212 <klopez4212@gmail.com>
New public API needs doc comments — clippy runs with -D missing-docs, so
TranscribeStatus and TranscribeSession were failing the Rust Lint gate.
Signed-off-by: klopez4212 <klopez4212@gmail.com>
Adds dictation support using OpenAI's Realtime API over WebRTC:
Relay:
- New /transcribe/status and /transcribe/session endpoints
- BUZZ_OPENAI_API_KEY env var gates the feature (hidden when absent)
- Proxies ephemeral client-secret minting from OpenAI
Desktop:
- New features/dictation module with:
- AudioWorklet for 24kHz PCM capture + buffering
- WebRTC peer connection to OpenAI Realtime API
- Real-time transcript merging into composer
- Auto-submit on trigger phrase ('submit')
- Mic button in composer toolbar (red pulse when recording)
- Integrated into MessageComposer via useComposerDictation hook
Signed-off-by: klopez4212 <klopez4212@gmail.com>
