Agent session logs

Cacophony exposes live agent output through PTY attach surfaces and historical output through runtime-owned structured logs. The old unbounded tmux pipe-pane capture path has been removed.

Current surfaces

Interactive web PTY

Current web terminals use the structured interactive /api/v1/agents/{id}/pty WebSocket.

TUI client diagnostics

$CACOPHONY_DIR/tui/tui.log is a client-local diagnostic log, not durable operator history. The TUI rotates runaway logs before appending and samples known high-frequency duplicate-message debug paths so normal navigation cannot grow the file beyond log-monitor thresholds. Use it for short-horizon TUI troubleshooting; use daemon logs, feed, summaries, or structured agent logs for durable operational history. If a live or stale pre-fix TUI log is already oversized, preview first with caco prune run --tui-logs --dry-run; the non-dry-run form trims the log to a bounded tail without deleting TUI state.

Legacy cleanup

Agents spawned after no longer create logs/session.log. Existing large files are safe to delete, and first-party pruning treats them as reclaimable artifacts for terminal agents: caco prune list includes their bytes in the reclaimable estimate, while caco agent prune and the automatic retention sweep remove them without deleting structured logs or metadata. For ad-hoc cleanup, review matches first:

# dry-run first
find ~/.cacophony/agents -type f -name session.log -size +10M -print
find ~/.cacophony/trash -type f -name session.log -print

# delete only after reviewing the output
while IFS= read -r -d '' LOG_FILE; do
  rm -- "$LOG_FILE"
done < <(find ~/.cacophony/agents -type f -name session.log -size +10M -print0)
while IFS= read -r -d '' LOG_FILE; do
  rm -- "$LOG_FILE"
done < <(find ~/.cacophony/trash -type f -name session.log -print0)

Terminal broker design

captures the intended Cacophony-owned terminal session broker for future embedded panes. The broker should own live PTY, tmux, SSH, and AKS stream creation, resize, detach, authorization, audit, and cleanup while renderers such as the TUI, web dashboard, and native desktop apps remain replaceable clients.

Daemon feed and crash-log hygiene

Daemon feed and undelivered-message ledgers are JSONL files: each structured record is serialized as valid UTF-8 and written with exactly one trailing newline. If serialization would produce an embedded line break in the raw record, the write is rejected rather than corrupting daemon/feed.jsonl or daemon/undelivered.jsonl. Replay remains forensic-first for older damaged records: malformed JSON or invalid UTF-8 is skipped/preserved as described in the daemon guide instead of crash-looping startup. The live feed ledger and matching feed_events index are pruned to a 2-day window; export or summarize anything that must outlive the short-horizon update stream.

daemon-crash.log holds only actionable main-daemon failures: stderr, panics, fatal errors and fatal summaries. Unknown daemon stderr is treated conservatively as crash evidence.

• Kept out: routine diagnostics already recorded in daemon.log, the feed and the Errors view, such as HTTP 5xx route records, model discovery, replication and sync notices, queued-dispatch and proxy errors, hook completions, checkout and index.lock diagnostics, snapshot backpressure, transient cluster errors, TLS “fatal alert” handshakes and persistent-agent idle advisories. Non-fatal transcription 502s are logged at warning level, and PID-only audio watchdog messages (diagnostic=pid-only-watchdog) stay in service logs.
• Self-cleaning: about once a minute, rotation prunes known non-crash tails (including stale “previous stderr” banners) while keeping real panics and backtraces. Log-monitor profiles should treat these shapes as recurring signatures, not new beads.
• HTTP 5xx records carry method, path, status, request ID, error.code and a bounded message, a redacted body preview, retryability and retry-after, so incidents can be diagnosed from the Errors view without grepping logs.
• Other services such as caco-web and the audio daemons keep their stderr in their own service logs.

Service log sources and cross-node queries

caco log tail --service caco-daemon-sidecar reads the actual sidecars/caco-daemon/sidecar.log; names ending in -sidecar resolve to their owning service directory. Missing or unreadable files are errors, not successful empty histories. Pi Daemon and its web, watchdog, and Paratenic connector services use native logs rather than Caco sidecar directories. On systemd, query journalctl --user -u 'pi-daemon*' -u 'paratenic-pi-daemon*'; launchd and supervisord use their configured stdout/stderr paths. caco service logs reads Caco's native supervisor output.

For cross-node diagnostics, put wrapper options before the command: caco foreach node --concurrency 2 log exceptions --since 3d --limit 50 --json. All arguments after the nested command starts belong to that command; use caco foreach node --json -- log exceptions --since 3d for an outer JSON result envelope. Agent iteration retains its existing selector parsing; use an explicit -- to separate nested command arguments from wrapper selectors.

Runtime recovery and snapshot freshness

Pi Daemon owns the latest generation of a configured session, including dormant generations requiring credential reprovisioning after a host restart.

• Adoption: Caco adopts a higher generation only when the returned session identity and retained target/spec or persistent-declaration provenance match; it never treats a reused session ID alone as proof. Automatic reconciliation preserves an existing logical persistent agent's inventory row and resumes it instead of discarding it before a replacement launch.
• Credential recovery: A retained credentials_required quarantine is reprovisioned only when the authenticated host advertises the complete same-generation recovery contract. Caco resupplies the exact saved declaration with host-incarnation, generation, revision, ETag, and retained-policy-digest fences; it does not replace the conversation or increment the generation. A lost terminal response is reconciled by reading that same identity, not by replaying a mutation.
• Otherwise: Unsupported hosts, missing policy evidence, and other recovery conditions still return session_reprovision_required without a replacement PUT. This preserves the current generation and conversation but does not restore inventory rows already lost before this safeguard.

UI mesh snapshots retain each failed project's last good bead rows while accepting successful updates and authoritative empty lists from other projects. Partial or failed attempts back off without becoming fresh data, and the UI deadline leaves one bounded-concurrency refresh running rather than cancelling and duplicating it. This is a read-model cache only; authoritative bead writes and ownership checks are unchanged. Reconciliation discovers unknown agent records off the async executor and reuses the manager's existing inventory instead of rehydrating every terminal record on each pass. Active records are still validated from disk before terminal-state transitions. Checkout-size cache misses use two shared workers and a bounded queue rather than spawning one recursive walker per agent; saturated admission leaves no permanent zero placeholder. Size walks accept an explicitly symlinked root but count descendant symlinks themselves rather than following external trees or cycles, and do not double-count friend checkouts already inside that root.

A missing same-origin configured remote alias can be repaired without a historical checkout-state.json when actual origin fetch/push URLs, branch, and upstream tracking prove the repository identity. caco checkout repair-remote --project <project> --dry-run reports identity_proof; normal startup uses the same locked additive repair before compatible-checkout metadata backfill. Malformed metadata, conflicting identity, dirty/ahead state, foreign Git common directories, and ambiguous remotes remain refused; this does not authorize a reset or a general metadata override.

Generic-provider discovery uses that provider's declared api_key_file or api_key_env, with file precedence and no fallback to another provider's credentials. An unreadable declared file preserves the last good catalog as stale and reports a credential error rather than issuing an anonymous request. Copilot aliases must explicitly configure /models, not the generic /v1/models default; account-specific token provisioning remains separate. Generic model_headers carries non-secret protocol metadata such as Copilot integration/editor compatibility headers; authentication and authority overrides are refused there. Model discovery requires an actual catalog envelope and bounds response bodies to 4 MiB, including chunked responses, so malformed successes or oversized responses cannot erase cached models or allocate unbounded memory.

Performance telemetry reports

Structured performance records live in the same daemon-local store as structured error records and are queried through caco log perf-list or the Performance surfaces in TUI/Web. In addition to sampled HTTP request_duration events, the daemon records its own process audit metrics once per minute by default:

• daemon_cpu_percent and daemon_cpu_seconds_total for process CPU use.
• daemon_memory_rss_bytes and daemon_memory_max_rss_bytes for resident memory.
• daemon_http_requests_per_sec, daemon_http_5xx_per_sec, and daemon_http_requests_total for throughput.

Use caco log perf-list --project cacophony --top for recorded request-duration hotspots, or select another metric explicitly, for example caco log perf-list --project cacophony --top --metric daemon_memory_rss_bytes. Groups keep metrics, units, and components separate, with numeric ranking only within a metric/unit pair; counters are never combined with ratios or durations. On glibc systems exposing mallinfo2, daemon_allocator_allocated_bytes and daemon_allocator_free_arena_bytes distinguish in-use allocations from reusable allocator arena space. These are allocator accounting, not RSS; compare them with resident-memory and request-load samples before diagnosing a leak or choosing cache infrastructure. --limit controls the number of groups in top mode. --since filters a bounded retained sample, not necessarily every event in the requested window; sample percentiles are not whole-traffic SLA percentiles. Set CACO_DAEMON_PERF_TELEMETRY_INTERVAL_SECS=0 only for a deliberate diagnostic opt-out; otherwise the default bounded cadence keeps telemetry available without writing on every request.

iOS/Android app error reports (platform-tagged)

Companion-app runtime errors — failed button actions, failed daemon requests, response decode failures — flow into Cacophony as first-class platform errors through the same structured-error ingest endpoint the CLI uses, so iOS/Android surface devs can analyse them in the Errors surfaces (caco log exceptions, the TUI/Web Errors tab).

Endpoint. POST /api/v1/logs/error — the same path caco log error uses, with fields component, summary, detail, severity, fingerprint, labels. The endpoint already exists (handle_log_error); clients do not need a new route.

Auth. The companion apps authenticate with their existing node/operator token. A node-identity caller is authorized for every endpoint (CallerIdentity::Node resolves to an allowed scope check), so the app's token can POST /api/v1/logs/error without a new scope grant; worker-scope tokens are not used by the apps.

Platform-tagging convention (stable — the iOS/Android client beads target this):

Recommended additional labels:

• surface:<view> — the originating screen, e.g. surface:ContentView, surface:AgentDetail.
• kind:<class> — the failure class: kind:button (a failed UI action), kind:request (a failed daemon request), kind:decode (a response-decode failure).

Fingerprint guidance (dedup). Set fingerprint to a stable signature of the recurring failure, e.g. <component>:<surface>:<kind>:<stable-error-key>, omitting volatile detail (timestamps, request ids), so a repeatedly-failing button aggregates into one Errors row with an occurrence count instead of flooding the surface. When fingerprint is omitted the daemon derives one from the component and summary.

Filtering. Narrow the Errors surface to one platform with the existing component filter — caco log exceptions --component ios-app / --component android-app — plus the platform: / surface: / kind: labels for finer analysis.

Example (verified end to end):

caco log error --project cacophony \
  --component ios-app \
  --summary "Save button failed" \
  --detail "ContentView save -> POST /api/v1/agents/<id>/nudge returned 500" \
  --severity error \
  --labels "platform:ios,surface:ContentView,kind:button"
# surfaces in: caco log exceptions --component ios-app

Operator takeaway

If you want to watch an agent live, use attach/terminal surfaces. If you want history, use the structured log views. Do not expect session.log to exist for new agents; future embedded terminal panes should use the broker contract rather than adding new unbounded capture files.