Controller Restart-Window Policy

Reference: .cacophony/profiles/controller.md and plugins/caco-agent/agents/controller.md contain the authoritative controller profile. This document summarizes the restart-window rules for operators and contributors.

Background

When the Cacophony daemon restarts (caco restart, caco up, caco down, caco update), the supervisor tears down the old process, waits for port release and file-lock cleanup, and then starts the new process. This window is routinely multi-minute — 2 to 5 minutes is normal during supervisor convergence, especially when the old process is slow to release its listener port or the local lifecycle loop needs time to reconverge.

Cluster controllers must treat this window as a normal operational phase, not an outage requiring escalation. When the old daemon is still reachable, caco restart publishes a bounded planned restart lifecycle signal before shutdown; caco status, fleet/router summaries, and beads proxy error envelopes use that signal to mark active beads-primary downtime as maintenance rather than an unexplained authoritative-beads outage. When the daemon is already unreachable, caco status --json also exposes daemon.down_reason plus recent lifecycle_decisions[] provenance so operators can distinguish a planned restart, supervisor self-healing, deploy-helper restart, crash, or native-service failure before filing a new incident.

Core Rule: No Stacking

A controller must never issue a second control-plane action while a prior one is still in flight.

In-flight command	Correct behavior	Incorrect behavior
`caco restart` still running	Wait for it to complete or time out	Issue `caco update` because reads are failing
`caco up` still running	Treat read failures as part of the window	File a remediation bead for “daemon unreachable”
`caco down` still running	Continue waiting	Issue a second `caco restart`

Stacking control-plane actions can cause supervisor races, double-bind failures, or cascading port conflicts that turn a routine restart into a real outage.

Recognizing a Restart Window

Symptoms that should be attributed to a restart window (not a new incident):

daemon unreachable errors
Loopback connection refused / reset
Port-release wait messages
Hook failures that appear once and do not reproduce after the daemon is back
caco-web shows Backend unavailable while its own shell and /health still respond, because daemon-backed APIs return 5xx during restart or /api/v1/ui/stream has failed for longer than the 15-second grace.
- Upstream 500/502/504 snapshot errors arrive as 200 OK daemon_proxy_upstream_error sentinels (with backend_unavailable:true and the original status in upstream_status and X-Caco-Upstream-Status), and still show Backend unavailable.
- An interrupted stream becomes a handled daemon_proxy_stream_error event with “stream interrupted; reconnecting” before the browser reconnects.
- A snapshot timeout is softer: a daemon_proxy_timeout sentinel (upstream_status:504) shows Snapshot delayed while retrying, or a snapshot-timeout message instead of zero counts if nothing has loaded yet. Any background refresh it triggers is logged, not toasted.
Any read failure occurring while a caco restart/up/down/update command has not yet returned
beads primary rows, authoritative-beads proxy errors, peer /beads/sync responses, or in-flight /beads/sync cancellations that include a planned restart maintenance window or local daemon shutdown drain for the active primary node. New sync requests that reach the cluster, legacy beads-host, or standalone beads-cluster routers during drain should return bounded HTTP 503 plus Retry-After; a daemon_draining / maintenance-shaped sync failure during SIGTERM is retryable restart-window fallout, not an Errors-tab daemon bug.
caco bd create reports indeterminate_create, caco bd claim reports indeterminate_claim, or caco bd create --claim true reports indeterminate_create / indeterminate_claim / claim_after_create_unverified; reconcile the authoritative bead state before retrying, assuming ownership, or filing a duplicate
caco status --json reports daemon.down_reason and recent lifecycle_decisions[] entries pointing at an in-flight restart, supervisor repair, or deploy-helper action

Controller Decision Flow

Is a control-plane command (restart/up/down/update) in flight?
  YES --> Wait. Do not issue another command. Do not file beads.
          Retry lightweight reads (agent summary, msg inbox) periodically.
          |
          Did the in-flight command complete?
            YES, daemon responsive --> Rehydrate fleet state. Resume normal monitoring.
            YES, daemon still down --> Escalate: one allowed update attempt (if restart
                                       was the in-flight command) or report to operator.
            NO, still waiting      --> Keep waiting unless active harm requires containment.
  NO  --> Normal monitoring. If daemon becomes unreachable with no in-flight
          command, follow the Major Action Gate escalation path.

Escalation Path (Degraded Health)

When health is degraded enough that caco choices cannot be presented:

Immediate containment of active harm is always allowed.
One caco restart attempt is allowed.
Wait for the restart to complete (multi-minute window is expected).
If restart completes but choices are still unavailable, one caco update attempt is allowed.
Do not loop. If choices still cannot be restored, report status and present operator choices as soon as the surface is available again.

Safe Manual-Control Mode

Use manual-control mode only when an operator explicitly wants to pause the native lifecycle supervisor and control local Cacophony services by hand during an incident. It is a bounded first-party escape hatch, not the normal recovery path. Routine convergence still uses caco up, caco down, caco restart, and service-family status commands.

Announce ownership and stop stacking actions. Make one controller the owner, record the incident/bead, and tell other agents to preserve WIP and stand down from local repair.
Pause the native supervisor with the first-party surface. Run caco service disable --json, then verify the supervisor is disabled or not loaded with caco service status --json. Do not use raw launchctl, systemctl, or supervisorctl while this surface is available.
Stop only the requested Cacophony services. Prefer caco down --services-only --json for the daemon/sidecar set, or caco down --service <family> --json when the operator named one service family. Avoid broad pkill/killall instructions; if a process must be contained manually, preserve the command, PID, listener, and reason in the incident record first.
Inspect before starting anything manually. Use caco ps --json, caco status --json, caco bd daemon status --json, caco web status --json, caco tts daemon status --json, and caco stt daemon status --json as applicable. These surfaces should show the disabled supervisor state plus active manual PIDs/listeners so operators can see who owns each port.
Start only the service the operator requested. Use the foreground first-party entrypoint in a named tmux pane or terminal so the process owner is visible: caco daemon, caco bd daemon serve, caco web --port <port> --daemon-url <url>, caco tts daemon, or caco stt daemon. If the goal is to restore the normal configured service set outside native supervision, run caco up --skip-update --json instead of piecemeal foreground processes.
Check dashboard ownership before touching port 11180. In standalone caco-web mode, exactly one standalone caco web process owns the dashboard port. In embedded mode, caco-daemon owns that port and a standalone web process would be a duplicate. Always read caco web status --json before starting or stopping a web listener, and treat embedded_daemon or stale/external-listener warnings as diagnostics to resolve rather than a reason to spawn another listener.
Restore supervision deliberately. Stop any foreground/manual service processes you started, run caco service load --json (or caco service start --json if the entry is loaded but inactive), and verify caco service status --json, caco up --skip-update --json, caco status --json, and service-family status outputs. Do not declare recovery complete until the native supervisor is healthy and manual PIDs/listeners are gone or explicitly adopted by the normal lifecycle.

For Contributors

When modifying controller profiles or control-plane recovery logic:

Preserve the no-stacking invariant. Any new recovery path that issues a control-plane command must check whether one is already in flight.
Restart-window duration is not fixed. Do not hard-code timeouts shorter than 5 minutes for restart-window detection.
Test restart-window behavior by simulating slow port release (e.g. a process holding the listener socket for 3+ minutes after SIGTERM).