Daemon

caco daemon is the per-node service that orchestrates all Cacophony operations. One daemon runs on each machine in the cluster.

Responsibilities

• Loading and validating local configuration
• Managing canonical project checkouts
• Creating and supervising agents
• Serving the local control API
• Forwarding remote operations to peer daemons
• Maintaining the feed and SQLite indexes
• Proxying beads requests to the authoritative host
• Supplying the TUI with fast query and stream endpoints
• Resolving provider credentials and propagating to child processes
• Proxying audio (TTS/STT) requests to upstream providers

Canonical Checkouts

For each project the daemon keeps a clean default-branch checkout at $CACOPHONY_DIR/daemon/checkouts/<project>. No agent can be created until it exists; agents are cloned from it and never edit it. It is fetched periodically (every 60 seconds is recommended) and refreshed before a spawn unless that happened in the last 10 seconds.

• Identity: a git-ignored .cacophony/checkout-state.json records a secret-redacted fingerprint of the checkout-defining config: remote, default and state branches, bootstrap, environment and worktree settings, submodule policy, sparse spec, extra remotes, reintegration topology, sync policy and branch namespace. A mismatch found on startup, config reload, project sync or a spawn check resets and reclones the checkout before anything uses it, and logs exactly what differed.
• Exact status: absent, empty directory, daemon placeholder, Git without origin, origin without HEAD, dirty partial repository, non-Git evidence, policy exclusion or valid; existing is not the same as ready. Clone phase, owner PID, temp paths and errors persist outside the worktree for crash-safe resume, and guards refuse symlinks, unmanaged targets and the live config root.
• Fallbacks: while the canonical checkout is unhealthy, spawn can use a daemon-owned direct-remote checkout and direct reintegration can clone its publish checkout from upstream, leaving the bad evidence untouched.
• Equivalent remotes: scp-style GitHub SSH, ssh://git@github.com[:22]/... and ssh://git@ssh.github.com:443/... count as the same repository.
• SSH over 443: with a configured project identity, daemon-managed Git routes GitHub SSH through ssh.github.com:443, so refresh and reintegration work on networks that block port 22 without global SSH config. github_ssh_routing: auto|ssh_over_443|default_port overrides it per project or node, and the node wins. Direct reintegration uses the same bounded fallback for its preflight and target fetches, and failures name the stage that rejected them.

Canonical checkout refresh is non-destructive for real work: local commits on the tracked branch are rebased onto the fetched upstream default branch when possible, while arbitrary dirty or untracked project files produce a loud checkout-health error instead of being reset away.

• Generated-state cleanup: The safe cleanup exception is intentionally narrow: generated Cacophony state blockers and stale repo-owned .cacophony/... files may be removed only when the target branch already tracks that exact path, allowing authority/config migrations to converge without broad destructive resets. Before reintegration uses a daemon checkout as its canonical source, it may also fetch first and reconcile stale dirty tracked paths only when every path already exactly matches the fetched target tree and the local target branch has no local-only commits.
• HEAD rollbacks: A second narrow recovery handles current-target HEAD rollbacks observed during checkout synchronization: if the checkout is already at fetched origin/<target>, dirty entries are either staged-only rollbacks where the index matches HEAD^ and the worktree matches the index, or worktree-only rollbacks where the index still matches HEAD and the worktree matches HEAD^. In both cases the dirty paths must exactly match the paths changed by HEAD; the guard restores only those paths from HEAD and then requires the checkout to be clean. Otherwise the loud guard remains in place.
• Visibility: Project overview/status surfaces carry checkout-health blockers so operators can see why a daemon checkout is unsafe for spawning, profile resolution, or reintegration input.

For recorded session summaries, the daemon may also warm a read-only clone of each project's configured cacophony-state branch at $CACOPHONY_DIR/state-branches/<project>/. Summary list/show/raw endpoints prefer that cache so interactive CLI, TUI, web, and Android viewers do not fetch the state branch on every request. The cache is refreshed by checkout init/refresh paths and is not an editable code checkout.

Auto-Branch Creation

When the configured branch does not exist on the remote (e.g. a new project or a branch that hasn't been pushed yet), the daemon automatically creates it:

1. Detects the remote's actual default branch (via ls-remote --symref, falling back to main or master)
2. Clones using the actual default branch
3. Creates the configured branch locally from the default
4. Pushes the new branch to the remote (warning on failure; local branch is still usable)

This also applies to fetch_and_reset: if a fetch fails because the branch doesn't exist on the remote, the daemon attempts to push the local branch before proceeding with the local HEAD.

Service Lifecycle

caco up

caco up

Loads and validates config, ensures the local daemon is running, starts the beads service (if this node is the beads host), launches sidecars, and starts declarative always-on agents. Safe to run repeatedly (idempotent).

The command now streams real lifecycle progress instead of a blind timer. Operators see service convergence, health/stabilization, and daemon background-startup phases such as profile discovery, persistent-agent reconciliation, full-state sync, and stale-agent repair, including degraded or potentially stuck follow-up work. Startup full-state pull only runs when sync.state.pull.on_connect.enabled is true, while the periodic pull loop is gated separately by sync.state.pull.periodic.enabled; disabling one trigger no longer accidentally disables or starts the other.

Service ownership and warmup

• caco-web has exactly one port owner. services.caco-web.mode: standalone (the default) runs a separate PID-tracked caco web process; mode: embedded lets caco-daemon own the dashboard port instead. caco web status reports the mode, the served version and any version drift, and flags a listener the lifecycle does not own.
• Stale listeners are swept, never the daemon. Before respawning a PID-only service, a stale process holding its port is stopped, but cleanup refuses to touch the active or restarting caco-daemon.
• Warmup is not failure. Supervised services report starting or warming during a bounded grace period; convergence waits for or adopts the owner instead of spawning a duplicate, and only marks it unhealthy after the grace expires.
• Audio daemons write their own PID files after their singleton guard, so the supervisor never makes a child mistake itself for a duplicate.
• PID 1 in containers reaps orphaned helper processes between convergence cycles, and replaces a PID file that points at the supervisor itself once the grace period ends.

What goes in daemon-crash.log

Only the main caco-daemon writes stderr to daemon-crash.log. Child services keep their stderr in their own service logs, and routine diagnostics (HTTP 5xx records, sync notices, checkout lock messages, idle advisories, config reload errors) stay in daemon.log, the feed and the Errors view. The crash log is rotated about once a minute and keeps real panic and backtrace evidence; unknown stderr is still treated as crash evidence. Config reload failures carry a short hash you can find with caco log tail --service caco-daemon --json.

Warmup cache

The warmup cache holds redacted per-project/profile/model summaries for faster agent starts. Injecting a summary into a new agent's goal is off by default; when enabled, the block is age- and size-bounded, labelled advisory, and subordinate to the product contract, AGENTS.md, the profile and the operator's goal.

caco down

caco down

Stops services and agents for the current node only. Does not mutate peer nodes. Supports scoped shutdown by service or project.

caco status

caco status --json

Reports:

• Local daemon status
• Local sidecar status
• Beads host status (if applicable), preferring the canonical beads routing view from /api/v1/beads so it agrees with caco bd status after primary switchovers
• Local config hash
• Peer liveness summary
• control_plane_ok, timed_out_sections, and status_guidance in JSON output so core daemon/beads availability is separated from advisory load, scheduling, service, peer-summary, or agent-summary timeout diagnostics. Text output describes these as incomplete advisory diagnostics and recommends direct node/agent verification before treating a timed-out peer summary as a failed worker.
• Running agents
• Desired vs. actual declarative state
• Bounded diagnostic-only zombie-process rows when observed, classified as daemon, beads-daemon, service, tmux, inherited-by-init, or unclassified children with PID/parent evidence; status reports but does not kill or reap them
• Bounded lifecycle_decisions[] restart/action provenance in JSON output
• daemon.down_reason when the daemon is unreachable: planned restart, supervisor repair, deploy-helper restart, crash, native-service failure, API backpressure or unknown listener loss.
  • If the service is running but its API times out, status reports daemon.api_backpressured=true (with daemon.pid_file_pid when that process is alive) and trusts that over stale restart sentinels.
  • If the listener is down but the port is held or a lifecycle lock is active, the reason names the port holder, the lock owner and recent lifecycle decisions, and points to caco service status and the daemon log before any recovery.
  • If the sidecar's expected bind window has passed, the CLI says the restart looks stuck rather than still in progress.
  Clients and probes use the daemon's effective same-node bind host, so a healthy listener on a non-loopback address isn't misreported as restarting. Binding the local API to the literal tailnet alias is currently refused by validation; use the loopback local API plus an explicit cluster bind.

caco restart

1. Accepted at once

   caco restart

   POST /api/v1/restart answers accepted: true before the restart subprocess starts, so the caller gets its 200 OK before the daemon goes down and polls health for the outcome.

2. Phases you can watch

   Port release, service health and background startup are each reported, so healthy progress looks different from a stuck restart. Every restart records why it happened: caco status --json lists recent lifecycle_decisions[], and daemon.down_reason while the daemon is down.

3. Roll back if it is unhealthy

   If health checks fail, the restart tries the single-slot previous launcher, then walks the launcher archive under ~/.cacophony/launcher/archive/ from newest to oldest until a daemon is healthy or the archive runs out. With previous_config, the prior config is staged too and restored on failure. caco launcher list and caco launcher rollback manage the archive; it keeps one previous generation by default (updates.launcher_archive_count: 1) plus any entry a live process still uses.

4. Never loop on the same version

   If a restart for an update finds the binary version unchanged, because the new one could not be fetched or built, that target version is suppressed until a newer release appears.

• Watchdog exits: after repeated local-listener probe failures the health watchdog writes a sentinel (daemon-health-watchdog, with failure count, interval and exit code) before exiting, so the supervisor's respawn reads as planned maintenance.
• Hung daemons: a daemon alive but unresponsive past warmup and a bounded grace period is stopped and restarted by the supervisor.
• Flapping: status retries one short probe batch before calling a PID-held port unreachable. A peer whose config-hash probe is slow but whose data-plane probe succeeds is shown as reachable but degraded, not pending.
• Draining: on SIGTERM the daemon drains first; peer /beads/sync requests get HTTP 503 with Retry-After, and interrupted syncs are reported as retryable daemon_draining, not as daemon errors.
• Cleanup and rollback: services with their own ports, such as caco-web, clear stale port holders before respawning. Launcher rollback waits for slow multi-node startup phases such as full-state sync, and web, TTS or STT failures don't block daemon promotion.

API Endpoints

CLI daemon transport failures first probe the cheap local /api/v1/node endpoint before reporting broad daemon-down guidance. If that probe succeeds while the requested endpoint times out or resets, the JSON error is classified as daemon_endpoint_transport_error with the probed node/version so operators can treat it as endpoint-local backpressure or a focused daemon bug rather than reflexively running caco up. The mutating messages/send endpoint is stricter: it never retries the POST on transport failure, and it reports retryable msg_send_backpressured even when the bounded node probe is also unavailable, marking daemon_probe.ok:false so operators know both checks failed without risking duplicate accepted sends.

Safety and compact-status floors remain bounded under daemon-store contention. Incident-hold reads, mutations, and admission fail closed with HTTP 503 incident_hold_store_backpressured; current-choice reads return choices_store_backpressured rather than an unsafe empty set. Both include Retry-After, normalized endpoint/reason, lock-wait/timeout diagnostics, and state_known_clear:false. /api/v1/agents/summary uses the background agent read model and independently bounds optional assignment, planned-outage, peer-health, and message-delivery collectors, returning base counts with explicit degraded_sections when enrichment is unavailable. CACO_SAFETY_STORE_LOCK_TIMEOUT_MS tunes the shared default 1000ms store-lock budget.

Discovery

Feed

Agent Management

Use project-scoped endpoints when project membership must be validated, and global /api/v1/agents/... endpoints for cross-node read/lifecycle clients. There is currently no daemon /api/v1/agents/<id>/reintegrate route; reintegration is still driven by first-party lifecycle/CLI surfaces.

Audio

Config

Cluster (Cluster Listener Only)

UI Stream Events

The /api/v1/ui/stream endpoint produces a synthesized event stream consumed by both the main TUI and the choices frontend:

• feed_event
• choice_presented, choice_resolved
• speech_requested, speech_finished
• transcription_requested, transcription_finished (completed events carry transcript text and text_length)
• session_updated
• health_updated
• bead_updated
• notification
• message_context_updated
• audio_capability_updated

Periodic Snapshots

The daemon periodically emits daemon_state snapshots persisted at $CACOPHONY_DIR/daemon/state/<node>.json, containing node health, running agents, project inventory, and config hash.

UI snapshots are bounded-staleness read models, not beads authorities. Non-owner nodes fetch only interactive states through a short-lived, single-flight mesh cache; the proxy coalesces identical project reads and trims legacy unfiltered requests so concurrent clients cannot multiply full-board scans.

Structured Performance Events

First-party HTTP request_duration performance events keep the human-readable METHOD /path summary for direct inspection and also attach bounded-cardinality labels for method:, status:, and normalized route: family. Project, agent, bead, scratchpad, perf-record, numeric, and long hex/UUID-like path segments are replaced with placeholders such as {project}, {agent}, {bead_id}, or {id}, so Performance views can aggregate hot daemon routes without parsing freeform summaries or retaining per-operator identifiers as label cardinality. Transcription requests add bounded audio:transcription plus latency-bucket labels (latency:normal_lt_5s, latency:slow_ge_5s, or latency:slow_ge_10s) so STT/provider latency investigations can filter slow calls without scraping freeform summaries. The daemon also emits bounded process telemetry once per minute by default into the same performance store: CPU percent/seconds, RSS and max-RSS memory bytes, request/5xx throughput, and cumulative request count. Set CACO_DAEMON_PERF_TELEMETRY_INTERVAL_SECS=0 only for an intentional opt-out, and inspect samples with caco log perf-list --metric daemon_cpu_percent or the Performance dashboards.

Pre-Merge Validation

Agent profiles can define before_reintegration hooks that execute pre-merge smoke tests during caco agent complete. These run after before_complete hooks succeed but before any git merge or PR operations begin. If any hook fails, reintegration is blocked and the agent is notified.

Typical uses:

• caco config validate --strict — catches config changes that would break the daemon on restart
• caco daemon validate — validates startup and migrations in an owner-private temporary runtime with an in-memory non-credential and inert replicator, disables token/PKI/config-Git/external fingerprint writes, cleans temporary artifacts, and fails if configured high-risk paths or Git HEAD change
• Template expression linting — catches undefined variables that resolve to null

The --strict flag on caco config validate treats null-resolving template expressions as errors, which catches bugs like undefined variable references that would silently produce empty values in normal validation. Agents can also use --project-config-dir (or CACOPHONY_PROJECT_CONFIG_DIR) to validate config from an agent checkout without modifying the real project config.

The daemon runs no validation gate of its own before landing: profile before_reintegration hooks still run in the CLI, and GitHub branch protection merges a pull request only after its required checks pass. The projects[].integration.gate config block and the --force-skip-gate flag were removed. See the validation guide.

Beads Routing & Standalone Daemon

All beads operations route to the active primary. With service-object beads.primary, independently managed caco-bd-daemon exclusively owns every project beads.db and WAL. The main daemon proxies over same-node loopback or remote mTLS and fails closed instead of opening a store. Inspect the service with caco bd daemon status / restart.

The standalone store is SQLite-first: mutations commit transactionally and mark durable export-dirty state; a bounded pass later writes deterministic JSONL and commits only changed content. Ordinary reads never rebuild from JSONL. Admission limits, query deadlines, read connections, indexes, and single-flight status/UI caches keep expensive reads from starving CRUD.

Dispatch sends an immediate wake; the batch endpoint and slow jittered polling provide anti-entropy, while legacy per-project polling is rejected before store access. Single-primary deployments omit the multi-primary peer-probe mesh.

Recovered former primaries are fenced import-only until catch-up is proven. Only the standalone PID should hold beads SQLite files; persistent WAL growth, ownership-fence refusals, read-pool waits, or repeated route timeouts are incident evidence, never reasons for the main daemon to take ownership. Routing and intended-primary state remain visible through caco bd status and caco bd trigger-vote --dry-run.

Test & Build Queue

Heavyweight validation runs through daemon-local FIFO test/build queues (caco test run / caco build run), serial per project/node with bounded CARGO_BUILD_JOBS/RUST_TEST_THREADS caps. Jobs persist across restarts (re-pended, or converted to retryable infra errors), and ceiling kills on a saturated host classify as retryable host_saturation_timeout.

Project-Level Hooks

Projects can declare lifecycle hooks that fire on specific events:

• checkout_updated — fires when the canonical checkout moves to a new commit (e.g. after reintegration merge). Variable: $commit_sha (message actions), $CACO_HOOK_COMMIT_SHA (shell env).
• tag_pushed — fires when a new git tag is detected. Variable: $tag (message actions), $CACO_HOOK_TAG (shell env).

Hook actions include notify_agent (lifecycle-only persistent-agent nudge with normal message queues suppressed), shell (run a command), broadcast (project-wide message), and speak (TTS announcement). See Agents for configuration examples. Scheduled cron actions remain background automation; an explicit caco cron run waits for the action’s configured bounded timeout and returns the nested command’s actual execution receipt, stdout/stderr, and failure status instead of treating dispatch acceptance as successful delivery. Manual dispatch runs in a detached request-ID-keyed task: one exact-ID response retry observes the same terminal receipt, a per-cron gate prevents overlapping manual execution, and the cron-specific outer deadline (330 seconds by default, configurable with CACO_CRON_RUN_REQUEST_TIMEOUT_SECS) returns typed cron_run_backpressured phase evidence rather than an unclassified transport failure.

Test Agent Isolation

Choices and operator inbox items from test-type agents are automatically filtered from operator-facing surfaces. This prevents test data from leaking into the production operator inbox:

• GET /api/v1/choices excludes choices from test agents
• Test agents can still resolve their own choices through the standard resolution endpoint
• Filtering is applied at the API layer, preserving test data in persistent storage

Supervisor Integration

When running Cacophony under a process supervisor (systemd, launchd), use the following configuration to prevent duplicate daemon instances:

systemd (Linux)

[Unit]
Description=Cacophony Daemon
After=network.target

[Service]
Type=exec
ExecStart=/usr/local/bin/caco up --foreground
ExecStop=/usr/local/bin/caco down
Restart=on-failure
RestartSec=5

# Prevent duplicate spawning — systemd will not start a new
# instance while the previous one is still running.
# Type=exec ensures the service is considered started only after
# the exec() call succeeds.

launchd (macOS)

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>com.cacophony.daemon</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/local/bin/caco</string>
    <string>up</string>
    <string>--foreground</string>
  </array>
  <key>KeepAlive</key>
  <true/>
  <!-- LaunchOnlyOnce prevents launchd from spawning a second
       instance while one is still running. -->
  <key>LaunchOnlyOnce</key>
  <true/>
  <key>StandardOutPath</key>
  <string>/tmp/cacophony-daemon.log</string>
  <key>StandardErrorPath</key>
  <string>/tmp/cacophony-daemon.log</string>
</dict>
</plist>

Important: The TTS daemon has built-in startup guards that refuse to start if another instance is already running. If you see a "refusing to start" error, stop the existing instance with caco down before starting a new one.

Agent Retention & Cleanup

Completed and discarded agent checkouts are subject to a periodic retention sweep (every 10 minutes). Policy is configured per-project under agent_defaults:

Preview with caco agent prune --dry-run and clean up with caco prune run --delete. Pruning reclaims the checkout and any legacy logs/session.log; structured logs, agent.json, prompts, environment files and results always stay. Scheduled and emergency pruning skip agents whose project isn't in the current valid config, so a bad config render can't delete managed work.

Add --nonzero-only to any preview to hide empty rows.

Persistent Agent Recovery

Persistent agents that become idle or stuck are auto-restarted based on persistent_agent_idle_timeout_secs. The no-tool-activity threshold is configurable per-profile via stale_timeout_secs, falling back to the global agents.stall_timeout_secs. For agents whose tmux/runtime session is still live, that threshold is advisory: after a short grace the reconciler records and surfaces a no-tool-activity warning while preserving running and the live pane. It must not fail, restart, recreate, discard, roll back, or reassign the agent solely because tool activity or startup readiness has not advanced; ownership-changing recovery requires positive death evidence or operator approval. Aggregate agent-list caches also include peer daemon-state snapshot freshness so remote rows converge when a peer snapshot updates.

Feed Replay Resilience

Startup rebuilds daemon state from durable append-only logs plus SQLite indexes.

• Writing: New local feed and undelivered-event writes serialize each structured record as valid UTF-8 and exactly one trailing newline, rejecting any serialized record that would introduce embedded raw line breaks before touching feed.jsonl or undelivered.jsonl.
• Malformed records: A malformed or non-UTF-8 record in $CACOPHONY_DIR/daemon/feed.jsonl must not crash-loop the daemon: replay preserves the original feed file for forensics, skips malformed JSON and invalid-UTF-8 records instead of indexing a lossy replacement, and continues rebuilding valid feed entries. Retention pruning uses the same lossy reader only for timestamp decisions; when it keeps an invalid-UTF-8 line, it writes the original bytes back unchanged so pruning does not destroy forensic evidence while reclaiming old valid entries.
• Retention: The live feed ledger and feed_events SQLite index are short-horizon update state and are pruned to a 2-day window on busy nodes; durable operator context belongs in summaries, logs, git history, or explicit feed exports.
• Replication retries: Undelivered replication retries apply per-peer adaptive backoff (up to five minutes, with TLS failures starting at the maximum and dial failures starting above the ordinary retry interval) and report retry backoff active without consuming another attempt while the peer is cooling down. Body errors or exhausted retry counts are compacted out of undelivered.jsonl and surfaced as a warning feed event with kind: "replication_event_lost", affected peers/event types, queued-time range, and max-attempt metadata.

Materialized Config

The daemon writes a last-known-good config snapshot to $CACOPHONY_DIR/daemon/materialized-config.yaml on every successful startup. This allows post-mortem debugging when a config change breaks the daemon.