Agents

Cacophony manages the full lifecycle of coding agents: checkout, terminal host (Herdr or tmux), profile loading, hook execution, status tracking, and reintegration.

Agent Identity

Tables and cards show the full ID, so workers with similar prefixes stay distinguishable; short labels appear only in cramped secondary controls.

Agent States

Checkout Strategy

Every agent works in its own checkout on its own branch, made by default from a canonical checkout the daemon keeps per project.

The canonical checkout records a secret-redacted cache identity: if project config changes the remote, branch, sparse spec, extra remotes or reintegration topology, the daemon reclones it before making new agents from it. How an agent's checkout is made is checkout_type; a persistent agent's declaration wins over projects[].agent_defaults, which wins over the profile.

Every clone the daemon makes, canonical, full or friend, takes its shape from config: the project's remote, SSH key, identity, git.config entries and extra remotes. caco git clone --project <name> --into <dir> makes the same clone by hand, with optional --branch, --filter, --depth and --sparse, and --check proves the identity, ls-remote and a dry-run push. The full table, the callers and the no-ad-hoc-clone rule are in the repo-shaping contract.

Friend projects with materialize: true become non-blocking clones of the friend project's own remote under friend-checkouts/<project>/, with their own objects, each with its own pending, ready or failed state. Writable friends require checkout_write, work on an isolated friend/<agent-id>/<project> branch, and land only through reintegrate --friend or complete --friend.

Terminal Hosts

Shell agents (Pi, Pi-Paratenic, Claude, Codex and other commands) run in the terminal host set by agents.runtime.multiplexer. Herdr is the default; tmux is available globally or per project.

agents:
  runtime:
    multiplexer: herdr   # herdr | tmux
    herdr:
      session: caco      # shared Herdr server name

projects:
  - name: example
    agent_runtime:
      multiplexer: tmux  # optional per-project override

A missing or incompatible Herdr is reported, never silently replaced by tmux. Existing agents keep the host they were created with, so changing the default never moves a live session, and caco agent attach --raw finds the right host on the owning node. caco agent register can borrow an existing Herdr pane; stopping a borrowed registration leaves the pane, its processes and the checkout alone.

CLI Usage

Spawn an Agent

caco agent new \
  --type codex \
  --profile worker \
  --goal "Implement the next ready bead"

To try a profile draft that is not yet under .cacophony/profiles/, pass --profile-path profiles/draft.md instead of --profile; the daemon loads that file through the normal profile bridge.

Where it runs

Each candidate node is scored, and ineligible nodes carry their reasons. An operator's choice comes first, then the highest-scored eligible node. The agent's status records why, and the daemon snapshot marks each node selected, ready or blocked for dashboards.

On what compute

Transient per-agent compute is opt-in per spawn: --compute local is the ordinary path, --compute default resolves agents.transient_compute.default, and any other value names a configured provider, such as the deterministic fake provider used in tests. A transient agent keeps the same ID and lifecycle; the daemon records its launch-spec hash, runtime and retry ceilings, admission and budget checks, and where its artefacts and logs will land.

caco agent new --type codex --profile worker \
  --goal "Implement the next ready bead" --compute fake

Now or in the background

With async_spawn: true or --async, caco agent new returns the agent ID straight away in starting while the checkout, bootstrap and launch continue; follow it with caco agent status or attach. --sync waits until it is ready. TUI spawns and Pi or debug-shell starts are asynchronous by default, so a slow provider or MCP server never times out the command.

List Agents

caco agent list

Attach to Agent Terminal

AGENT_ID="agent-id"
caco agent attach "$AGENT_ID"
caco agent attach "$AGENT_ID" --raw

By default caco agent attach opens the agent's own pane: status on top, its terminal below. --raw attaches straight to the underlying terminal, over SSH when the agent is on another node. A remote attach or status fails fast only when the peer is unreachable or its API probe is degraded; a config mismatch shows as cluster drift but does not block the request, so busy remote workers stay inspectable while config converges.

View Agent Status

AGENT_ID="agent-id"
caco agent status --id "$AGENT_ID"

Status reads reconcile with the agent's persisted agent.json first, so a recent liveness ping or terminal transition on disk clears a stale stalled or failed view; list and summary views do the same.

• Lineage: when an agent has lineage metadata, status shows the chain and can filter it with --lineage-filters, --lineage-participant, --lineage-transition, --lineage-source and --lineage-reason.
• Diffs are read with GIT_OPTIONAL_LOCKS=0, so refreshing a diff panel never races the agent's own git add/commit. The diff endpoint takes summary_only=true for counts and file rows, or max_diff_bytes=<N> to cap the patch.
• Heartbeats are delivered by the local daemon, configured under agents.heartbeat, projects[].agents.heartbeat or nodes[].agents.heartbeat with declaration/profile/id overrides, and suppressed when off, paused or finished. Optional conditions: [claimed_beads, unmerged_prs, pending_inbox] requires any matching condition in addition to enablement; omission inherits and conditions: [] retains unconditional controller timers. Empty/unknown conditional ticks advance the timer without a runtime wake. Pi normally records a non-model receipt; only an explicit trusted CACO_PI_HEARTBEAT_PROMPT opts into a task-specific model turn. Direct messages, assignments and queue returns keep their event-driven wake paths. When an idle agent's checkout holds committed work for an owned bead that is on no remote and not in the merge queue, the heartbeat adds a [caco-unpublished-work] line naming those beads and the agent's own submission command, and caco agent status shows blocked:unlanded_commits.
• caco heartbeat status explains the effective setting, suppression and schedule. status|on|off --agent <id> reaches the owning node automatically; without --agent it acts on the local node, and @node/@all still work for deliberate fleet operations.

Pane diagnostics never change anything by themselves. The daemon classifies each agent pane as healthy, frozen, rate-limited, stuck on a prompt, in a merge conflict, or with an exited or zombie child; an idle controller or observer with no recognizable blocker is watch-only, not restarted.

An optional remediation policy (off by default) picks the least invasive next step (observe, notify, nudge, dismiss a prompt, pause/resume, or ask the operator about a restart) based on attempts, cooldowns, role and explicit allow flags. Only non-destructive steps run directly; anything destructive becomes an operator choice. Attempts are kept for 30 days so cooldowns and post-incident review can see the history.

View Agent Logs

AGENT_ID="agent-id"
caco agent logs "$AGENT_ID"

Import transient-agent artifacts

Short-lived dynamic-compute workers can disappear before an operator inspects them. Use MANIFEST="/path/to/transient-result/manifest.json"; caco agent import-artifacts --manifest "$MANIFEST" to materialize an uploaded result-bundle manifest back into the normal local runtime layout so caco agent status, caco agent logs, caco agent replay, and caco agent diff remain the canonical read surfaces. Automatic dirty-work checkpointing before involuntary terminal transitions refuses large deletion-only staged changes (20+ deletions and no additions/modifications), resets the staging area, and leaves the checkout dirty for explicit first-party recovery instead of committing a likely broken whole-tree deletion checkpoint.

For microVM-backed workers, the same manifest may include crash/timeout checkpoint breadcrumbs. The importer preserves logs/microvm-console.log, logs/microvm-supervisor.log, microvm/checkpoint.json, optional checkpoint patches, git-bundle fallback, checkout tarballs, backend name, checkpoint kind, and checkpoint reason under the normal agent directory.

MANIFEST="/path/to/transient-result/manifest.json"
caco agent import-artifacts --manifest "$MANIFEST"
caco agent status --id "$AGENT_ID"   # shows backend/checkpoint breadcrumbs
caco agent logs --id "$AGENT_ID"     # includes microVM console/supervisor logs

Set Agent Status

caco agent status set "running tests"

Stop an Agent

AGENT_ID="agent-id"
caco agent stop "$AGENT_ID"

A stop that outlives the response budget returns HTTP 202 with stopped:false and completed:false; the CLI reports it as accepted, not completed. The original stop keeps the runtime-control lock until it records success or refusal, and a concurrent runtime operation receives retryable agent_stop_in_progress. Check caco agent status before acting on the runtime.

Resume an Intact Conversation

AGENT_ID="agent-id"
caco agent resume --id "$AGENT_ID"

Explicit resume also reopens completed, stopped and discarded-but-intact agents in the same checkout and conversation. It refreshes the managed environment/profile and injects one short [caco-resume] notice directing the agent to await the operator's next instruction. It does not require, reclaim, reopen or republish a bead. If the host session is absent, it reloads retained state or recreates the same logical identity and says so in the notice; a pruned or deleted checkout is refused, not silently replaced. In the TUI, a paused agent still shows Unpause (U), while bulk recovery does not reopen all delivered conversations.

caco agent recreate keeps a stopped agent stopped: when the source is Stopped (or a persistent declaration is operator-stopped), the replacement gets a fresh checkout, environment and record but no runtime launch, and the operator-stop intent is kept. Start or resume launches it later. Recreating a running agent still relaunches it immediately.

A completed AHP agent's local health is detached, not evidence that the host deleted its conversation. Older socket-era session_missing completion records are corrected on the read surface without claiming a fresh host probe. See conversation resume and retained-session health.

• Time budget: resume waits longer than ordinary calls so the runtime can relaunch and prove it is ready. If the response is lost while the daemon is reachable, the CLI returns retryable agent_resume_transport_error with may_be_in_progress: check status before retrying.
• Persistent agents retain their identity and checkout on an explicit conversation resume. Automatic recovery retains its separate relaunch path; a conversation resume is not permission to rebuild a pruned checkout.
• Other lifecycle calls return retryable agent_lifecycle_backpressured or inspect-before-retry agent_lifecycle_transport_error rather than a generic “daemon down”.
• caco agent nudge has a shorter budget because controllers use it as a health probe. --name <fragment> matches a persistent agent's configured name or ID, then a project-scoped suffix, then a substring; the configured name keeps working after recreation. Ambiguous matches fail with candidates in scripts and offer a picker in a terminal.

Fork a Live Agent Pane

AGENT_ID="agent-id"
caco agent fork --id "$AGENT_ID" --type pi

Fork launches a sibling runtime pane in the same per-agent tmux session without stopping the parent process. The fork inherits the same Cacophony agent ID, checkout, branch, and token scope, rewrites the runtime invocation to use --fork rather than --continue, and can select a supported backend with --type pi|claude|codex.

Resume Nudge

Agents record lineage: which agent an agent was retried, replaced, recreated, handed off, forked or resumed from, with reason, source and time. Fork and resume keep the same identity; the others name a successor. caco agent status --id <agent> shows the lineage chain, and its JSON carries the full records with filter facets. Reading lineage never rewrites it.

Decision points are bounded, redacted snapshots of an agent at a commit or an explicit capture: bead, branch, head and a short diff and session excerpt. caco decision-point rewind --agent-id <id> --decision-point-id <dp> --dry-run previews restarting work from one of them (checking bead ownership and that the ref still exists) without spawning anything. Handoff snapshots carry the same metadata for passing work to a successor agent; caco agent wip-handoff --dry-run describes the plan, and nothing is captured, patched or spawned by these previews.

Calls about a remote agent go straight to its owning node when that node is reachable, and fall back to the fleet collector otherwise; failures come back as structured errors with an HTTP status, a retryable flag, retry-after seconds and guidance. A single-agent status read never turns into a fleet-wide sweep. While an agent is starting or resuming, stop and discard stay safe, and a racing spawn, recreate or resume is refused or queued by policy.

Automatic work recovery and restart retain their existing recovery nudges. Explicit completed/stopped/discarded conversation resume sends the short [caco-resume] notice instead, without an old-goal nudge. For automatic recovery, phrasing depends on the stop reason:

• Operator-initiated stop (via caco down, caco restart, or explicit caco agent stop): nudge says "Your managed session was restarted."
• Crash or infrastructure failure: nudge says "Your managed session was revived after a crash."

Controller-scoped agents (those monitoring the fleet rather than working on beads) receive controller-appropriate phrasing directing them to rehydrate fleet state. Worker agents are told to continue an assigned bead only when the canonical agent record actually carries a bead assignment; if no assigned bead is recorded, the nudge says so, treats the missing assignment as potentially incomplete, directs the worker to audit the live board for its own in-progress assignment before idling or auto-claiming, then inspect its managed checkout and follow its profile idle/work-selection rules instead of inventing phantom bead work.

Merge Conflict Recovery

If direct reintegration hits a merge conflict or stale base, prefer AGENT_ID="agent-id"; caco agent rebase --id "$AGENT_ID" as the first-party recovery shortcut.

• Target: It rebases onto the same authoritative target branch that direct reintegration will validate, aligns the visible origin/<target> tracking ref to that fetched target so ordinary post-rebase status/diff commands do not show already-landed commits, reports the conflicting file list, and leaves any remaining conflicts active for git rebase --continue before retrying caco agent complete or caco agent reintegrate.
• Stale locks: Before fetch/rebase it also applies the managed-checkout stale .git/index.lock predicates to the agent checkout, removing only old unheld locks inside the configured agents root and otherwise reporting checkout-health diagnostics.
• Corrupt refs: Its fetch preflight also repairs Cacophony-owned corrupt local remote-tracking refs such as refs/remotes/origin/agent/*, the configured state branch such as refs/remotes/origin/cacophony-state, and corrupt local refs/tags/* tags when Git reports them as bad/missing objects, then retries with a bounded warning instead of requiring manual packed-ref cleanup.
• State-only commits: After a successful rebase, if every commit ahead of the fetched authoritative target changes only .cacophony/agent/** state or summary artefacts, the command resets the checkout to that target and reports the state-only cleanup so persistent agents do not keep carrying already-published summaries.

Spawn Request

{
  "node": "localhost",
  "type": "codex",
  "profile": "worker",
  "goal": "Implement ready work",
  "id": null,
  "env": {},
  "hooks": {
    "on_start": [],
    "before_tool": [],
    "after_tool": [],
    "on_complete": []
  },
  "reintegration": {
    "mode": "direct"
  }
}

If node specifies a remote node, the local daemon forwards the spawn request to that node's daemon via the cluster listener.

Agent Wrapper

caco agent run wraps the actual agent binary and handles:

1. Materialize checkout from canonical clone
2. Create tmux session
3. Load profile and install hooks
4. Launch underlying agent type
5. Inject resolved provider environment variables
6. Capture stdout/stderr
7. Record status changes
8. Write completion result

Startup and Activity Timeouts

A startup/readiness timeout does not automatically fail an agent while its tmux pane is still alive and attachable. The daemon preserves the pane in starting, surfaces an explicit warning, and lets operators attach or wait for slow bootstrap work such as first-run environment realization to finish. Immediately after creating a managed tmux session, the launcher also uses a bounded attachability settle loop instead of a sub-second fixed probe, so transient tmux transport failures under load preserve the final probe detail in diagnostics before the launch is classified as failed. A healthy running state is reported only after the configured runtime has actually launched; an accidental shell-only prompt is not treated as a completed worker launch.

Pi-family readiness also inspects the bounded post-launch stabilization window for deterministic provider rejection before reporting running. Invalid or missing credentials fail as provider_credentials_invalid; disabled or unavailable model access fails as provider_model_unavailable. Diagnostics name provider, model, and node but redact credential material, while transient network and provider 5xx failures remain retryable. Project provider/model defaults remain preferred unless a caller intentionally supplies an override.

Early bootstrap diagnostics are persisted under the agent directory at logs/bootstrap.log. The generated init.sh records named steps such as sourcing env.sh, applying shell/bootstrap snippets, configuring git excludes, writing alive/readiness sentinels, verifying the managed caco command, and execing the runtime. If startup exits before readiness, the log includes the failing step, exit code, PWD, and PATH, so operators can distinguish env quoting, checkout bootstrap, sentinel, and runtime exec failures even when the tmux pane dies before it can be inspected.

No-tool-activity timeouts are also advisory when tmux/runtime liveness is still positive. After the configured threshold and grace, the daemon records a warning while preserving running; it does not fail, restart, recreate, discard, roll back, or reassign that agent without positive death evidence or operator approval.

Transient Blocker Triage

Managed agents should treat local lock files, proxy read failures, stale cached node-health summaries, and restart-window timeouts as provisional signals until a bounded direct check confirms them. Before declaring a target node down, idling on a claim blocker, or filing broad outage work, cross-check the authoritative surface when possible. Useful checks include caco @node bd status --project <project> --json, caco @node agent list, and the target node's caco status --json.

If the direct authoritative check is green, agents should continue implementation or reintegration and record the local/proxy drift as diagnostics. If the direct check confirms a real blocker, file or update a specific bead with the failing command, node, bead ID, and observed authority state instead of waiting silently or broadcasting a vague outage.

When startup reconciliation does have positive failure evidence, structured log_error events and summaries include failure_cause labels so operators can tell the path apart from advisory timeout noise. Common labels include confirmed_tmux_session_dead, provider_runtime_exited, runtime_launch_failed, tmux_socket_dead, and timeout_or_starting_grace.

Credential Propagation

The wrapper passes these environment variables to spawned agents:

Completion

On completion, the wrapper writes result files:

PROJECT="project-name"
AGENT_ID="agent-id"
$CACOPHONY_DIR/agents/${PROJECT}/${AGENT_ID}/
  result/
    status.json         # Final state, profile, reintegration result, git info
    stdout.log          # Captured stdout
    stderr.log          # Captured stderr

Reintegration Policies

Reintegration is configurable per profile and per spawn request:

Profiles still default to direct when no policy is set, but a project can steer successful reintegration with projects[].integration. default_intent: review maps the default success path to pr_review, while backend: pull_request uses the configured fork/upstream remotes and gh to create the review surface instead of doing a local squash merge. During a short-term PR rollout fallback such as operators can keep the fork/upstream topology declared but set default_intent: direct and backend: local_merge so routine completed work lands on main while PR auto-merge / agent-merge is verified separately.

For persistent agents, PR-backed mid-flight reintegration is intentionally non-destructive: caco agent reintegrate --mode pr_review / pr_auto_merge keeps the runtime alive until the PR actually lands, emits an explicit warning/event that the agent was not recreated, and expects an operator or controller to run AGENT_ID="agent-id"; caco agent recreate "$AGENT_ID" afterward when a fresh runtime is desired. Profile-source changes also affect future materialization before they affect already-running persistent prompts; use caco agent refresh --id "$AGENT_ID" to rewrite managed env.sh, Pi/runtime config, and friend-checkout env/backfill, and reserve recreate/restart coordination for cases that need a fresh runtime.

Project policy is resolved as intent plus backend: default_intent chooses direct, review, artifact-only, or none; backend chooses local safe merge or PR-backed delivery for direct intent. The topology remains in remote, remotes, integration.reintegrate_target, and integration.pr_base.

Direct Mode Rules

• Target branch refreshed from remote before merge
• Agent branch validated against current target head
• Result lands as one squashed commit on the target branch
• Completion metadata includes the original agent branch and commit
• Safe publish-or-refuse checks reject dirty or stale branches and report success only when the landed commit is reachable from the configured remote target
• caco agent rebase is the explicit checkout-mutating recovery path; direct reintegration does not rewrite the live worker checkout, while rebase may clean up ahead commits only when they contain exclusively .cacophony/agent/** state or summary artefacts

PR-Backed Rules

• Push only to remotes whose project config allows pushes
• Publish to generated same-agent PR branches, never directly to the integration/base branch even when that branch's remote is writable
• Open or reuse a PR against integration.pr_base
• Reuse the project GitHub CLI wrapper (CACO_GH_COMMAND_OVERRIDE_*, CACO_GH_COMMAND_OVERRIDE, or checkout bootstrap) instead of inventing a second auth path
• Recorded .cacophony/agent/<id>/... summaries are published to cacophony-state and referenced from the PR body
• Manual reset/cherry-pick is exceptional diagnostics, not normal reintegration; agents should keep branches clean and retry through caco agent rebase plus complete/reintegrate

Agent Compatibility

Profiles use a canonical Claude-style format (Markdown + YAML frontmatter). Agent types that don't natively consume this format are supported through bridges:

Runtime types accepted by caco agent new --type:

Persistent Agent Restart Policy

Persistent agents (those declared in agents.persistent) support a granular restart policy that controls daemon auto-restart behavior:

agents:
  persistent:
    repo-observer:
      profile: repo-health
      restart: failure
      restart_backoff_secs: 300  # backoff before relaunch after an exit

restart_backoff_secs is a relaunch backoff, not the normal work cadence for a healthy persistent agent. Long-running Pi-backed specialists such as technical-writer keep running and use their composed runtime wake paths (for example pi-loop) for review cadence instead of completing every cycle just to be restarted.

Operator-facing status resolution treats persistent declaration IDs as first-class aliases for the live deterministic agent. If a declaration row such as persistent-69cd15d1 is stale but a matching persistent agent is already running for the same declaration name, node, and project, caco agent status --id persistent-69cd15d1 and aggregate persistent inventory should resolve to the live runtime rather than reporting agent_not_found or a stopped sentinel row. Agent summaries cross-check current in-progress bead claims, but closed or deleted bead IDs left in agent.json or historical assignee fields are watch-only stale_closed_bead_assignment residue rather than actionable stuck-worker evidence.

Pi Agent Runtime

Pi is a managed agent runtime supported as a Tier 2 bridge. For the Cacophony project itself, managed Pi agents compose a default stack of profile mixins:

Compose these as an ordered profile: list. Daemon-native direct delivery owns messages and configured heartbeat owns routine wakeups; use pi-loop only for a distinct recurring task. Speech/tool-call enforcement, the checkout wrapper, self-ops, independent inbox polling, automatic task-resume nudges, self-compaction, the image and profile-drift guards, and the sudo runner are retired. Their old profile names remain inert for external declaration compatibility, and refresh removes known generated entrypoints without changing the operator Pi home. Explicit /inbox inspection remains available. See profile lifecycle and runtime stacks for adoption and refresh guidance.

For non-Pi CLI sessions that need repeating narration, use PROJECT="project-name"; caco loop --project "$PROJECT" --interval 10m --body "..." instead of ad-hoc shell loops.

Package Pre-Installation

During Pi workspace materialization, the daemon reuses the host’s warmed ~/.pi/agent/git/ package cache (including node_modules) by linking it into the workspace when possible, then pre-clones any remaining git packages declared in the Pi settings.json packages array (e.g. git:github.com/owner/repo@tag). This prevents cold-start timeouts where Pi’s first-run package clone exceeds the daemon’s provider-ready or stall-detection window. Older workspaces that still contain copied .pi-agent/git caches can be relinked through caco prune run --pi-package-cache --current-agent or an explicitly selected prune run. Errors during pre-install are logged but non-fatal — Pi can still install packages on its own.

Spawn Concurrency

Nodes and projects can each declare concurrency ceilings for agent spawns:

nodes:
  - name: local-node
    concurrency:
      max_agents: 4

projects:
  - name: my-project
    concurrency:
      max_agents: 2

When a ceiling is omitted, that scope is unlimited. The daemon-local queue holds excess spawn requests until capacity frees up. Queued work is never re-routed between nodes.

Pre-Merge Validation

Profiles can define before_reintegration hooks that run after before_complete succeeds but before any git merge or PR operations. These are intended for pre-merge smoke tests such as config validation, template linting, and daemon startup checks. If any hook fails, reintegration is blocked and the agent is notified.

hooks:
  before_reintegration:
    - type: command
      command: caco config validate --strict
      timeout: 30

Project-Level Hooks

Projects can declare lifecycle hooks that fire on project events. These are configured under projects[].hooks:

Each event maps to a list of actions:

projects:
  - name: cacophony
    hooks:
      tag_pushed:
        - action: notify_agent
          agent: update-helper
          message: "New tag $tag pushed"
      checkout_updated:
        - action: speak
          body: "Checkout updated to $commit_sha"