Transcription and speech-to-text
Cacophony transcription surfaces, commands, and checks to verify STT health.
caco audio capabilities --json, run caco stt doctor --json, then prove transcription with FIXTURE="/tmp/caco-stt-fixture.wav"; caco audio transcribe --file "$FIXTURE" --json. Prefer file fixtures over microphone tests for regression work.
Quick commands
caco audio capabilities --json
caco stt doctor --json
OUTPUT="/tmp/caco-stt-fixture.wav"
caco audio speak \
--text "Cacophony transcription sample. The quick brown fox tests speech to text." \
--model azure/speech/azure-tts \
--voice 'en-US-Reed:MAI-Voice-1' \
--format wav \
--output "$OUTPUT"
caco audio transcribe \
--file "$OUTPUT" \
--model gpt-4o-mini-transcribe \
--language en \
--json
Use caco audio transcribe --live only when microphone permission, selected input route, and operator intent are known-good. Live mode depends on OS microphone permission, selected input routing, and voice activity detection. It does not read stdin; it spawns the configured capture backend itself. VAD-delimited speech is bounded by maximum segment duration and payload size, so uninterrupted speech flushes in smaller chunks instead of growing one unbounded in-memory audio buffer. For managed ambient daemons that need longer chunks, the live transcription path reads the bounded environment knobs CACO_LIVE_TRANSCRIBE_SILENCE_THRESHOLD_DB, CACO_LIVE_TRANSCRIBE_SILENCE_DURATION_SECS, and CACO_LIVE_TRANSCRIBE_MIN_SEGMENT_SECS; less-sensitive/longer chunking means a lower silence threshold, longer silence duration, or larger minimum segment, at the cost of higher latency and a greater chance of merging nearby turns. Prefer file fixtures for repeatable regression checks. For pipeline-style regression checks, caco audio transcribe --file - reads WAV bytes from stdin, and caco audio doctor --clip - uses stdin as the self-test clip.
caco audio transcribe --live --model gpt-4o-mini-transcribe --language en
INPUT_ROUTE="remote-pulse"
caco audio transcribe \
--live \
--input "$INPUT_ROUTE" \
--device source.default \
--model gpt-4o-mini-transcribe \
--language en
# Agent-buffer proof: route partial/final markers into an agent live buffer.
AGENT_ID="lark"
caco audio transcribe \
--live \
--input "$INPUT_ROUTE" \
--device source.default \
--agent-buffer "$AGENT_ID"
For a remote PulseAudio source, force mono 16 kHz capture, wrap it as WAV, then use the first-party daemon transcription path:
PULSE_SERVER="${PULSE_SERVER:?set PulseAudio server host:port}" \
timeout 5 pacat -r -d source.default --format=s16le --rate=16000 --channels=1 \
| sox -t raw -e signed-integer -b 16 -r 16000 -c 1 -L - /tmp/speech.wav
caco audio transcribe \
--file /tmp/speech.wav \
--model gpt-4o-mini-transcribe \
--language en
Do not feed a default s16le 2ch 44100Hz PulseAudio stream into transcription unchanged. The CLI/TUI/daemon path encodes captured audio as WAV before sending it to /api/v1/audio/transcription, so the sample rate is explicit.
Surface map
| Surface | Purpose | Guidance |
|---|---|---|
caco audio capabilities | List daemon-reported TTS/STT/realtime availability, models, voices, and devices. | Use for inventory only; model visibility is not proof that a backend works. |
caco stt doctor | STT-specific diagnostic and self-test. | Use before claiming local STT health. A failed self-test is actionable even when capabilities list a model. |
caco audio doctor | Broader audio doctor with --skip-self-test, --model, and --clip - for stdin. | Useful for capability triage; output shape differs from caco stt doctor. |
caco audio transcribe --file | Canonical file-based transcription through the local daemon; pass --file - to read WAV bytes from stdin. | Best default for repeatable fixtures and regression checks. |
caco audio transcribe --live | Foreground live microphone transcription with VAD. | Requires host microphone permission and correct input routing. |
caco audio live start / status / stop | Non-blocking live transcription session control for agent/MCP workflows. | start launches capture, status reports PID/scratch/log metadata, and stop requests graceful finalization. |
caco audio mcp stdio / plugins/caco-audio | Generated audio/STT MCP tools. | Exposes speech, file/live transcription, and non-blocking live-session controls through the scoped plugin. |
caco audio prewarm | Pre-download/prewarm a local STT model. | Relevant to local scribble; does not prove the daemon was compiled with local STT support. |
caco audio speak --output | Daemon-backed deterministic speech fixture generation. | Prefer this over host-provided tts binaries unless a local generator is declared and verified. |
caco audio realtime-turn --text | One-turn provider-backed realtime text-to-audio conversation. | Defaults to gpt-realtime-1.5; gpt-realtime-2 is also a first-party realtime model option when the provider is available. Surfaces daemon/provider error envelopes directly and returns audio under data.audio on success. The daemon advertises realtime WebSocket subprotocols first, then retries once without Sec-WebSocket-Protocol for OpenAI-compatible proxies that accept the beta header but omit a selected subprotocol. |
caco audio realtime-speech --file | One-turn provider-backed speech-to-speech conversation over the realtime WebSocket. | Sends input audio as realtime input_audio and receives output audio without forcing an STT → text → TTS round trip. Use --output reply.wav or --stdout; transcripts, when returned by the provider, remain command response metadata. It uses the same subprotocol-advertise-then-fallback handshake as realtime-turn. |
Standalone tts CLI | Host-provided generator outside the Cacophony contract. | Not recommended for managed fixtures unless it is declared in config and verified on that host. |
POST /api/v1/audio/transcription | Local daemon HTTP endpoint used by CLI and frontends. | Canonical provider-facing boundary; provider credentials stay in the daemon. Provider warmup/backpressure/connectivity failures return structured transcription_unavailable diagnostics with retry metadata instead of opaque gateway errors. |
| TUI push-to-talk / choices TUI STT | Send completed captures to the daemon STT endpoint. | Delivered text should follow the same path as typed messages, replies, or choice responses. |
| TUI / caco-web / Android Transcription tools hubs | Global and per-project Tools > Transcription surfaces for transcript-related feed events, speech/STT history, and transcript scratchpads. | These are operator observability hubs: TUI and caco-web load cached scratchpad note text directly in the hub, while other surfaces may link out to notes; mutation and provider calls still go through the canonical CLI/daemon surfaces above. |
| TUI voice conversation mode | Continuous record/transcribe/deliver loop for one attached agent. | Use when the operator intentionally wants live voice interaction with one agent. |
| Scratchpad dictation | Rolling chunk transcription plus smart append/final rewrite. | Useful for notes; keep exact chunks when evaluating accuracy. |
Checked-in ambient STT defaults
The checked-in project config imports shared STT defaults from stt.yaml (in the separate cacophony-config repository) through values.stt.imports. That file owns the default file/live model, speech.stt.language language hint (for example en), VAD tuning, local scribble settings, and per-target rewrite/smart-append behavior. The top-level language hint is overlaid through node config and is also the fallback language reported for local scribble transcription when speech.stt.scribble.language is unset. The host-specific services.caco-stt-daemon entries then select device/input routing and scratchpads; the current ms-mac and sgu24 ambient profiles use gpt-4o-transcribe-diarize, expand transcript buffers to 10,000 entries / 10 MiB, and route sgu24 capture through the sgu24-phone / source.phone input. Treat those as operator config defaults, not proof that microphone permission, Pulse/CoreAudio routing, or provider credentials are healthy.
Ambient instance health and text passthrough
See STT instance health and passthrough for per-instance status, supervision, external producers, durable cursors, mute boundaries and deployment limits.
Capture mode
See capture health and supervision.
Optional text-only mode
See external transcript sources, replay, mute and authentication.
Remote device-microphone STT input (device_mic)
A companion app (Android / iOS / watchOS) can act as a remote microphone for daemon STT. The app runs an in-app HTTP server exposing GET http://<host>:<port>/mic with an Authorization: Bearer <token> header, streaming 16 kHz mono PCM16 little-endian (Content-Type: audio/L16). The daemon consumes that stream as an STT input source. Reference it through the android://, iphone://, or watchos:// input-URI forms (which resolve to a device_mic source) or configure it directly with type: device_mic, scheme (android/iphone/watchos), host, port, and a bearer token.
Direct Azure authentication
providers.azure.auth selects key (the backward-compatible default) or entra for direct native Speech TTS, fast STT, and MAI STT. providers.azure.az is an executable path/name, defaults to az, and accepts an Azure CLI-compatible broker such as az-token. It is executed directly, not as a shell command:
az-token account get-access-token --resource https://cognitiveservices.azure.com/ --output json
Example direct-speech configuration:
providers:
azure:
auth: entra
az: az-token # or an absolute executable path; default: az
azure-speech:
base_url: https://your-speech-resource.cognitiveservices.azure.com
tts_base_url: https://eastus.tts.speech.microsoft.com/cognitiveservices/v1
resource_id: /subscriptions/YOUR-SUBSCRIPTION/resourceGroups/YOUR-GROUP/providers/Microsoft.CognitiveServices/accounts/YOUR-SPEECH-RESOURCE
# Existing mai.base_url / mai.deployment settings remain resource-specific.
The serving daemon's broker session must already be authenticated and authorized for the target resources; Caco does not run interactive az login.
Use the resource's Entra-enabled custom subdomain for fast STT and ensure the principal has the appropriate Speech/AI data-plane role.
Native TTS requires azure-speech.resource_id to construct the documented Authorization: Bearer aad#<resource-id>#<token> value; fast STT and MAI use the raw Entra bearer token.
See Microsoft's Speech Entra setup and Speech REST authentication.
The broker must return Azure CLI JSON with accessToken and expires_on (UTC epoch seconds, number or string), or legacy expiresOn (RFC3339 or CLI local wall time). The daemon caches tokens only in memory, coalesces concurrent acquisition, and refreshes lazily before expiry: five minutes early, or 10% early for shorter lifetimes. Both UTC and monotonic deadlines prevent clock changes from extending token use. A definite HTTP 401 permits one token refresh and one request replay; other provider failures are not retried by this auth layer.
Broker calls have a 30-second deadline, 128 KiB stdout limit, cancelled-child cleanup, and five-second failure backoff; stdout/stderr tokens are never echoed in diagnostics or exported into agent environments. Direct adapters do not follow redirects with credentials.
In key mode, existing azure-speech.api_key_file / api_key_env and the separate azure-speech.mai key references remain authoritative, and the broker is never invoked. In entra mode, those direct-adapter key files/environment variables are not used for authentication, and broker failure never silently falls back to a key.
OpenAI-compatible routes, including Helsinki LiteLLM and azure/speech/azure-tts(-hd), retain their existing proxy credentials and do not invoke this broker. Selecting Entra auth does not automatically switch a proxy model to a direct model, and it does not configure standalone Pi/cascade speech backends that bypass the daemon adapters.
For rolling config distribution, require the compiled azure_entra_auth_v1 activation capability before activating providers.azure; older daemons then retain their last compatible configuration rather than silently ignoring the new auth policy.
Azure-native STT (mai-transcribe-1.5)
In addition to the OpenAI-shape /audio/transcriptions models, the daemon can call the Azure Speech fast-transcription REST API directly. Configure the azure-speech provider (section 6.8 credential rules) with a region base_url (for example https://eastus.api.cognitive.microsoft.com) and api_key_file (optional api_version, default 2025-10-15, and locales, default ["en-US"]). In default key mode, the subscription key resolves to AZURE_SPEECH_API_KEY (via api_key_file or api_key_env) and the region to AZURE_SPEECH_BASE_URL, plumbed like the other provider secrets (Home Manager services.cacophony.providers.azureSpeech.apiKeyFile secretEnvScript, container-prelude AZURE_SPEECH_API_KEY_FILE). When configured, mai-transcribe-1.5 is advertised in caco audio capabilities and is the checked-in default STT model (stt.yaml in cacophony-config, the azure-speech provider must be configured for default STT to work), selectable explicitly like any other STT model (caco audio transcribe --file clip.wav --model mai-transcribe-1.5 --language en-US).
The daemon POSTs to {base_url}/speechtotext/transcriptions:transcribe?api-version=<version> with an Ocp-Apim-Subscription-Key header in key mode (or Entra Authorization: Bearer) and a multipart body of the audio bytes plus a definition JSON part ({"locales":[...],"enhancedMode":{"enabled":true,"model":"mai-transcribe-1.5"}}), and normalizes the Azure response (combinedPhrases[].text, falling back to phrases[].text) into transcript text. A missing/invalid key or region returns a structured azure_speech_unavailable / transcription_unavailable diagnostic rather than an opaque error.
Azure MAI STT (mai-transcribe-2)
MAI Transcribe 2 uses a separate Azure MAI resource API, not the regional Speech fast-transcription adapter used by 1.5. Declare the resource and deployment under the existing typed azure-speech provider:
providers:
azure-speech:
# Existing base_url/key/locales for 1.5 and native TTS remain unchanged.
mai:
base_url: https://harryaskham-sandbox-ais-eus.services.ai.azure.com
api_key_env: AZURE_EASTUS_API_KEY
deployment: harryaskham-sandbox-ais-mai-transcribe-2
# Optional: api_key_file: /path/to/your/mai-resource-key
# Or use base_url_env / api_key_env, defaulting to
# AZURE_STT_ENDPOINT / AZURE_STT_API_KEY.
The deployment is the resource's configured deployment name, not necessarily mai-transcribe-2. The example maps the nonsecret Agent Utils/Collective batch MAI binding explicitly: its resource is harryaskham-sandbox-ais-eus, its deployment is harryaskham-sandbox-ais-mai-transcribe-2, and its credential reference is AZURE_EASTUS_API_KEY. Other resources must supply their own deployment and endpoint; the East US 2 streaming binding and legacy regional Speech key are not substitutes. The runtime owner's normal secret resolver must make the selected key source available on the serving host. In key mode, the daemon resolves the MAI key file at request time (preferred over its configured environment variable), without reusing the legacy Speech subscription key or exporting secrets to clients. In Entra mode, it uses the shared Azure broker instead. An endpoint ending in .cognitiveservices.azure.com is normalized to .services.ai.azure.com. The base URL must be a resource root without credentials, query, or path; HTTPS is required except for localhost test fixtures. No default public OpenAI endpoint or automatic proxy fallback is used for this model.
caco audio transcribe --file clip.wav --model mai-transcribe-2 --language en --json
The daemon POSTs multipart/form-data to {resource}/mai/v1/audio/transcriptions with api-key (or Entra Authorization: Bearer).
The form has a file part holding the WAV bytes (filename audio.wav, content type audio/wav), a model part with the deployment name, and an optional language part.
The MAI Audio team asked for all MAI-Transcribe-2 traffic to use multipart, so the earlier JSON body with a base64 audio_url is no longer sent.
The request requires WAV framing and accepts at most 6 MiB of decoded audio within the existing 8 MiB daemon JSON body limit.
Responses must be JSON with a string text field and are bounded to 1 MiB.
The provider operation has a 60-second total deadline, cancels when its future is dropped, and does not follow redirects.
Only Entra's single refresh/replay after a 401 is retried, within that same deadline, and the replay rebuilds the whole form.
Provider error bodies are not echoed into diagnostics because they may contain credentials, audio, or transcript text.
Capabilities advertise MAI 2 when its endpoint and deployment plus either a key source or Entra auth are declared; discovery does not invoke the broker. CLI and daemon validation share built-in model recognition, so a known model with a missing binding returns azure_mai_unavailable rather than invalid_model; unknown models remain refusals. Configured STT command-template overrides still take precedence for MAI 2. Adding this binding does not change capture routing, mute state, ambient transcript retention, or the legacy MAI 1.5 protocol. Source support and model visibility are not live acceptance: verify a containing runtime, the serving daemon, and genuine new transcript/scratchpad advancement before declaring ambient STT restored.
Native Azure TTS (azure/speech/native-tts)
The proxy-based azure/speech/azure-tts and azure/speech/azure-tts-hd models (routed through the OpenAI-compatible /audio/speech path) are unchanged. In parallel, the daemon can also call the Azure Speech TTS REST API directly via azure/speech/native-tts and azure/speech/native-tts-hd, reusing the same providers.azure-speech key/region as STT — no extra config (caco audio speak --model azure/speech/native-tts --voice Mai-Voice-Finetuned-1:M --format mp3 --output out.mp3 --text "Hello.").
Native TTS is only served by the dedicated host https://<region>.tts.speech.microsoft.com/cognitiveservices/v1 (the STT cognitiveservices host returns 404 for TTS), so the daemon derives that host from the providers.azure-speech.base_url region (https://eastus.api.cognitive.microsoft.com → https://eastus.tts.speech.microsoft.com/cognitiveservices/v1). Set providers.azure-speech.tts_base_url only for a non-derivable/custom host. The daemon POSTs the same Azure SSML the proxy variants build, with headers Ocp-Apim-Subscription-Key (or the Entra authorization described above), Content-Type: application/ssml+xml, and X-Microsoft-OutputFormat (audio-24khz-48kbitrate-mono-mp3 by default, riff-24khz-16bit-mono-pcm for wav/pcm). These native models share the finetuned-Azure voice compatibility of the proxy variants and are advertised in caco audio capabilities only when providers.azure-speech is configured.
Azure embedding voices
A bare voice: embedding:<speakerProfileId> uses MAI-Voice-2.1-Flash as its base neural voice in both daemon SSML and direct Pi cascade speech. An explicit embedding:<baseVoice>:<speakerProfileId> always keeps its selected base, including older versions.
The model field still selects the provider transport, not the embedding base. Use the standard (non-HD) Azure transport and put an explicit base in the voice when needed:
model: azure/speech/azure-tts
voice: embedding:MAI-Voice-2.1-Flash:your-speaker-profile-id
voices:
- provider: azure
model: azure/speech/azure-tts
voice: embedding:MAI-Voice-2.1-Flash:your-speaker-profile-id
Omitting the base (embedding:your-speaker-profile-id) selects the same Flash default without pinning its version. model: MAI-Voice-2.1-Flash with voice: embedding:... is not yet a supported transport alias. just tts-default selects the non-HD mai-voice-2.1-flash-express profile from the shared configuration; it does not rewrite explicitly selected HD transports or custom embedding bases.
Capability vs health
caco audio capabilities --json answers what the daemon believes is configured. Provider-backed STT capabilities include gpt-4o-mini-transcribe, gpt-4o-transcribe-diarize, gpt-realtime-whisper, and whisper when OpenAI-compatible credentials are available; local scribble appears when a scribble model is configured. Diarized provider responses are normalized into speaker-prefixed transcript lines when the upstream returns segments, diarization, or utterances with segment-level fields such as speaker, speaker_label, speaker_id, speaker_index, or speaker_idx; numeric speaker IDs are rendered as stable speaker_N prefixes. When speaker metadata is present, those segment lines take precedence over a plain top-level text so the same text stored in transcription events, scratchpad/diff streams, and CLI JSON retains the speaker labels. The capability list does not prove a provider key is valid, a local model was compiled in, a microphone permission prompt was accepted, or a selected input route is audible. Use caco stt doctor --json and a small file fixture before claiming transcription is healthy.
If scribble is listed but a doctor or file transcription returns a local feature/backend error, treat that as a real local-STT health failure and route it to the owning implementation bead or operator queue.
Example fixture workflow
A useful smoke fixture has short text, a known transcript, and no microphone dependency.
OUTPUT="/tmp/caco-stt-fixture.wav"
caco audio speak \
--text "Cacophony transcription sample. The quick brown fox tests speech to text." \
--model azure/speech/azure-tts \
--voice 'en-US-Reed:MAI-Voice-1' \
--format wav \
--output "$OUTPUT" \
--json
caco audio transcribe --file "$OUTPUT" --model gpt-4o-mini-transcribe --language en --json
Record model, command, latency, exact transcript, and exact text diff or WER when benchmarking accuracy. Normalization such as “speech to text” becoming “speech-to-text” may be acceptable for a smoke fixture, but it should still be recorded when comparing models.
Diagnostic interpretation
caco stt doctor --jsonreports STT configuration and self-test state. Treat a self-test failure as an implementation/configuration problem, not a cosmetic warning.caco audio doctor --skip-self-test --jsonis useful for inventory without running an STT call. It still runs the capability probe, reportsself_test.status: "skipped_by_flag", and is not proof that a model can transcribe. Explicit--modelvalues must be one of the advertised STT models.- Long provider-backed TTS fixture generation can be slow. Use short text and keep
caco upas the first-party recovery path if the daemon is transiently unreachable after a long synthesis request. - Direct PulseAudio smoke tests should downmix/resample to mono 16 kHz WAV before calling
caco audio transcribe --file. transcription_audit_busymeans the daemon's shared store stayed busy for the whole 2-second audit wait. With phaseadmission_auditthe request was not dispatched to any provider. With phaseresult_auditthe provider had already finished, so do not replay that audio. Shorter contention is absorbed by the wait instead of refusing the request.
Current limitations
scribblecan be visible in capability output even when the local feature or model path is not usable. Confirm withcaco stt doctor --jsonor a file transcription.caco stt doctorandcaco audio doctoroverlap but intentionally expose different JSON shapes today.- Synthetic TTS fixtures may be provider-backed and slow. Keep text short and avoid treating a successful “played” status as proof that a physical speaker or microphone loopback was audible.
Where transcripts appear
caco audio transcribe --file ... --jsonreturns text indata.text, with model and language metadata. Configuredspeech.stt.command_templatemodel labels advertised bycaco audio capabilities --jsonare valid here; automation should not hardcode only provider model names.- TUI push-to-talk and choices TUI speech input deliver accepted transcription text into the same message, choice response, or direct-reply path as typed input.
caco audio live start/status/stopmanage non-blocking live sessions; agent-buffer mode appends partial segment updates and a final user-transcription marker. Foreground live transcription also accepts--agent-buffer <agent-id>to write those markers totranscription:<agent-id>:liveand connect that scratchpad to the target agent. On macOS,local-defaultandlocal-devicelive STT routes capture through native CoreAudio; named PulseAudio routes such as remote Android sources still use the PulseAudioparecpath.- Voice conversation mode continuously sends transcribed segments to the attached agent until detached.
- Scratchpad dictation appends raw chunk transcripts to the visible note, then optionally runs daemon-side smart append and final rewrite passes.
- The headless
caco stt daemonkeeps a bounded log of ambient transcript entries under$CACOPHONY_DIR/stt-daemon/.- Reading: listener profiles read new entries with
caco stt diff --cursor <n>.caco stt tail [-f|--follow] [-n N] [--stt-daemon <id>]is the passive view for people and pipes: it shows recent lines, can follow and merge every local instance, and never moves adiffcursor. - What gets written: segmentation favours longer conversational chunks, merges nearby useful ones and drops very short low-signal fragments, so only stable
[ambient transcription]utterances reach the scratchpad anddiff. - Waking consumers: with
transcript_wake_projectandtranscript_wake_targets, each utterance sends a lifecycle signal to consumers such astranscript-narrator. The signal carries no transcript text and is not a command or a chat message; consumers still readcaco stt difffrom their own cursor. - Supervision:
caco-stt-daemoninstances are non-critical services. The daemon writes its own PID file after its single-instance check; a supervisor that writes it first makes the daemon refuse itself and restart-loop. - This project's config: two unmuted English instances (
ms-macandsgu24), each keeping 10,000 entries or 10 MiB, writing totranscription:ambient:<node>scratchpads.
- Reading: listener profiles read new entries with
- Daemon speech history (
GET /api/v1/ui/snapshotspeech_historyand the corresponding UI/feedtranscription_finishedevent) includes the completed transcript text plustext_length, so frontend history can recover the visible transcript after reconnect. - TUI and caco-web
Tools > Transcriptionpages surface cached transcript scratchpad note text alongside feed/speech metadata. caco-web loads recent scratchpads via/api/v1/scratchpads?content_preview=4000, filters for transcript/STT/audio terms when possible, and renders bounded note-content previews in the hub so operators can inspect the transcribed text without first opening a separate scratchpad view. If no transcript-like note is found, caco-web falls back to recent scratchpads for the selected project scope.
Ambient listener policy
The STT daemon, caco stt diff, and passive caco stt tail are deliberately read/control surfaces. They never treat ambient transcript text as an authenticated operator command by themselves. Listener/router profiles that poll diff must declare their filing threshold or activation grammar, allowed targets, confirmation rules, capture mute/stop policy, possible TTS/status echo classification when Cacophony speech plays out loud rather than through headphones, and bounded audit record.
The repository now ships a transcript-bead-filer persistent observer profile for the conservative case: it polls each configured caco stt diff source by cursor (for this deployment, local ms-mac plus remote sgu24) and also inspects replicated scratchpad notes named transcription:ambient:*, such as transcription:ambient:ms-mac and transcription:ambient:sgu24. It remembers per-source cursors, ambient-scratchpad checkpoints, and recent-filing state in scratch, and only creates deduplicated draft beads with the from-transcript label when ambient speech contains a clear idea, bug report, task request, or call to action. It does not execute transcript commands, close beads, resolve choices, restart services, or message agents from ambient speech. It is declared on the persistent-observer stack in agents/cacophony_persistent.yaml (in cacophony-config), so it has reintegration.mode: none and should idle when transcript text is unclear, likely to be agent TTS/status narration, or likely to be loudspeaker feedback from recent Cacophony speech output. When in doubt, transcript readers should advance their cursor without filing rather than convert an echo into work.
The repository also ships a transcript-narrator persistent observer profile for the co-narrator case: it watches the same configured fleet transcript sources, treats every cursor advance as an interrupt, gathers quick useful context from the web or repository source, and speaks a concise earpiece-style assist back to Harry.
- Scope: Unlike the bead-filer it is not limited to Cacophony work; it can augment general conversation, technical terms, public facts, and codebase questions as they arise.
- Safety: Its safety boundary is that it speaks context only: it must not execute transcript instructions, mutate board state, or file beads, and it must suppress likely TTS/status feedback so it does not narrate its own narration recursively.
- Deployment: The checked-in declaration runs on helsinki alongside cluster control, uses explicit
ms-macandsgu24remote transcript sources, and has scratchpad read access so it can include relevant global/project notes in its spoken context while only writing its own narrator cursor notes. - Wake path: The normal wake path is STT-arrival lifecycle signals from the configured
caco-stt-daemoninstances, not chat messages and not a 10s or 60s fixed Pi loop; use a conservative fallback such as a 15m Pi loop only when event delivery is unavailable.
Without an explicit profile policy like that, ambient transcripts are diagnostic context only. Generic agents should not infer bead, choice, lifecycle, or infrastructure commands directly from caco stt diff output or transcription:ambient:* scratchpads.
JSON shape caveat
The audio namespace still exposes several JSON shapes. caco audio capabilities and successful caco audio transcribe use {ok,data,meta}. caco stt doctor returns command plus findings without a standard top-level ok. caco audio doctor has top-level ok, capabilities_ok, self_test_attempted, self_test_ok, capabilities, hint, and self_test, but not nested under data. The top-level ok reflects capability availability; inspect self_test_ok for the transcription self-test result. Automation should parse the exact command it invokes.
Regression checklist
- Capture
caco audio capabilities --json. - Capture
caco stt doctor --json, including self-test pass/fail. - Generate or reuse a short WAV fixture with known text.
- Transcribe with
gpt-4o-mini-transcribe. - For pipeline checks, run
caco audio speak --stdout ... | caco audio transcribe --file - --jsonand, when useful,caco audio speak --stdout ... | caco audio doctor --clip - --model <model> --json. - Transcribe with
whisperwhen provider budget allows. - Run local backends only when local support is expected to be compiled and configured.
- Record model, command, latency, exact transcript, and exact diff/WER.
- Avoid live microphone claims unless input permission and device routing were verified.
FAQ
- Which command should I use to test transcription? Start with
caco stt doctor --json; prove actual transcription withFIXTURE="/tmp/caco-stt-fixture.wav"; caco audio transcribe --file "$FIXTURE" --json. For generated fixtures,caco audio speak --stdout ... | caco audio transcribe --file - --jsonexercises the pipe path without touching disk. - Which model/provider is active? Use
caco audio capabilities --json, then verify the selected model withdata.modelfrom a real transcript. - How do I generate a deterministic sample? Prefer
OUTPUT="/tmp/caco-stt-fixture.wav"; caco audio speak --output "$OUTPUT". Do not use a host-provided standalonettsCLI unless a coherent package is declared in config and verified locally. - Where do transcripts appear? CLI transcripts appear in
data.text; TUI/choices STT deliver text into the same path as typed input; scratchpad dictation appends to notes. - What is currently broken or overlapping? Capability inventory can list local models that fail at runtime;
caco stt doctorandcaco audio doctoroverlap and use different JSON shapes; synthetic TTS fixture generation can be slow.