agent utilsPi package

speech

Talk to the agent, and let it talk back

Agent Utils turns Pi into something you can use with the screen off: it reads what you send, speaks replies or narrates its work, takes dictation, holds full voice conversations, and coordinates all of that across sessions and machines so two agents never talk over each other.

From words to a speaker

From words to a speakerOn the left, four sources of speech: read-aloud of what you send, verbatim spoken replies, one-line narration around tool batches, and spoken choices. Each goes to a configured audio provider, Azure, OpenAI, a speech daemon or a local command. Synthesised audio joins one machine-wide playback queue shared by every Pi session: a spool with atomic directory locks, one clip at a time with a two-second overlap. From the queue, audio plays through PulseAudio, where focus and solo routing can redirect it. Every request is also written to a durable speech feed that ag tts tail follows across the fleet. A mute policy per kind, with epochs, stops muted speech and keeps queued speech from replaying after an unmute. /read what you send /tts assistant replies /narrate tool-batch lines choices spoken questions provider azure · openai daemon · command playback queue machine-wide atomic dir locks 1 at a time 2 s overlap 1 GiB spool cap PulseAudio focus · solo routing speech feed every request, played or not mute policy, per kind read · tts · narrate · choices each mute advances an epoch; queued speech from an old epoch never plays
The speech path.Every voice surface shares the same providers, queue, routing, feed and mute policy, so they behave consistently and can be silenced together.

Speech out

CommandSpeaks
/readWhat you send, read from the editor as it is submitted. Text transformed by /pipe is spoken as finally sent.
/ttsEach finished assistant reply, verbatim. The agent never has to call a speak tool.
/narrateOne first-person line before and after each batch of tool calls, written by a small, cheap model. It never blocks tools, and its summaries are queued for your next real turn rather than starting one.
spoken choicesQuestions and their options, read out when an agent asks (see Sessions).

Sessions under pi-daemon get the same /tts, /narrate and choice audio from a durable module, which follows the session's events on the daemon host whether or not a client is attached.

Providers

Connections are configured once by name and selected per surface: Azure (with keys, or Entra tokens from an az broker), OpenAI, a speech daemon that can queue and play on a server, or any local command for text to speech.

{  "agentUtils": {    "audio": {      "providers": {        "studio": { "type": "daemon", … },        "cloud":  { "type": "azure", … }      }    }  }}

One queue per machine

Several Pi sessions on one machine would otherwise speak over one another. They share a playback queue with no daemon at all:

/tts focus and /tts solo temporarily route a session's speech to a shared focus output or play it alone, with ownership-checked cleanup afterwards.

Speech in

CommandDoes
/sttDictation with local voice detection: each turn is captured locally, wrapped as WAV and sent as one bounded HTTP request to an OpenAI-compatible transcription endpoint
/ptt, Ctrl+SpacePush-to-talk, toggled from the editor
/rtA full voice conversation with an OpenAI Realtime model, used as an ordinary Pi provider (openai-realtime/gpt-realtime-2 by default), with audio as a side channel
/cascadeA voice group chat: you speak once, then each agent answers in turn, hearing you and everyone before it, in its own voice

Local voice detection replaced an older streaming subprocess that could hang on "transcribing" forever. Because the whole candidate turn is already in hand, one request with a timeout and explicit errors is enough. The Realtime guide covers routing to a phone, voice-detection tuning, replays and diagnostics.

Muting, everywhere at once

Speech has four kinds (read, replies, narration and choices), and each can be muted on one machine or across the fleet without touching any session's settings:

ag tts mute                     # all four kinds, every enabled machineag tts mute --narrate --local   # just narration, just hereag tts statusag tts unmute

The policy lives in ~/.local/state/agent-utils/tts/mute.json. Each mute advances that kind's epoch, and queued or in-flight speech carries the epoch it started with, so muting and quickly unmuting can never replay stale speech. Each kind keeps its own PulseAudio client name, so a mixer can control them separately too.

Sharing audio

Speech is one pathway; deliberately shared audio is another. When an agent makes a song, proposes a sound or records a sample, audio_share puts it in a terminal audio reel in the session, with keyboard and mouse transport, and in a durable archive that ag audio gathers across machines. The reel never touches the speech queue, feed or mute policy, and speech never enters the reel.

♪ ▶ 2/5 Bossa riff in E minor (take 3)            ━━━━━━━━━━━●──────────────── 1:23 / 3:45  1 intro sting  [2 Bossa riff in E m…]  3 rain loop  4 UI click  vol 100% · /audio-reel