Testing

Cacophony uses a four-lane integration testing strategy with a committed fixture for whole-system validation.

Test Lanes

The repository defines four end-to-end validation lanes:

Integration Fixture

The committed fixture at tests/fixtures/config.integration.yml defines a single-node cluster with all ports shifted by +20000 to avoid collisions with development instances:

Hermetic Validation Flow

The hermetic localhost lane runs against an isolated temp $CACOPHONY_DIR and temp git remote:

1. caco config validate --config=... — config valid
2. caco cert init --config=... — CA created
3. caco cert issue --config=... --node localhost — node cert issued
4. caco up --config=... — node converges
5. caco status --config=... --json — reports healthy daemon, beads, sidecars
6. caco bd create, list, claim, close — proves beads proxying and persistence
7. caco msg send, broadcast — proves messaging, feed emission, scoping
8. TUI/choices contracts validated via UI SSE stream
9. Synthetic speech fixture generated through caco audio speak using explicit command-template fixture model labels, including file, JSON, and stdout/piping surfaces; transcribed through caco audio transcribe, caco audio doctor, and POST /api/v1/audio/transcription; and asserted in CLI JSON/text, daemon responses, doctor/self-test output, and UI snapshot speech_history entries with transcript text plus text_length
10. Agent lifecycle exercised via deterministic test bridge
11. Agent completion updates project state, feed, logs, git reintegration
12. caco down --config=... — clean shutdown

Credentialed Acceptance Lane

Uses the same --config override and fixture, with real provider credentials supplied via the managed-service model:

• At least one real agent launch through a supported production bridge
• At least one real TTS and one real STT via the daemon audio proxy
• Message delivery readable aloud via the configured speech provider

Docker Compose Multinode Lane

The docker compose multinode lane (cacophony-spec section 24.7) validates three-node cluster operation using the fixture at tests/fixtures/compose/:

• Feed replication across nodes
• Remote daemon forwarding
• Remote beads proxying
• Project-vs-global chat visibility across nodes
• Config-hash mismatch reporting
• Recovery after a temporarily unavailable peer

The compose fixture and the operator-facing deploy/compose/ share the same runtime/config contract but are kept separate: the fixture is a deterministic validation environment, while the deploy path is for operator use.

Relay Transport Validation Lane

cacophony-spec section 24.8.1 requires a relay-specific multinode validation lane once any runtime implementation of relay transport exists. The lane can live in the compose multinode harness, the hermetic microVM multinode harness, or a dedicated committed fixture, but it must use first-party caco commands and isolated runtime directories. It should validate relay-peer selection and invalid-peer rejection, mTLS bootstrap identity, persistent outbound connection lifecycle, routing for feed replication, full-state sync, messaging, beads proxy/read operations, remote daemon forwarding, source-identity preservation, dispatch forwarding, relay failure semantics, and observability for relay node, relay peer, connection lifetime, queue/backpressure state, and failure phase. Until relay runtime exists, the harness may skip only with a machine-readable relay_runtime_unavailable reason so CI can distinguish “not implemented yet” from missing validation.

Running Tests

Managed Agent Validation

When Cacophony’s configured queue lane has authoritative GitHub Check & Lint and Fast Tests (unit), those checks own final ready-time validation. Authors may run the smallest exact package/target plus test-name smoke needed to answer a concrete development uncertainty, but should not rehearse broad Cargo, clippy, build, or docs gates immediately before central CI repeats them on the same capacity. The managed pre-tool guard intercepts broad direct commands and points them at the first-party queue. Commit ready work, submit it through the canonical lifecycle surface, and move on. Projects without equivalent central enforcement keep their configured validation policy.

The managed hook layer still intercepts heavyweight compiler, Nix, emulator, and build/test commands on shared hosts and denies high-risk shell operations including sudo, firewall tools, service stop/restart commands, broad process kills, destructive runtime-state deletion, force pushes, and hard resets. Route explicit exceptions through the first-party queue with bounded concurrency:

# Tiny focused development smoke when one uncertainty warrants it
caco test run --wait --command "cargo test -p caco-daemon --lib exact_test_name"

# Uncovered-platform build through the queue
caco build run --wait --command "cargo build --jobs 2" --timeout 900

# Source-only checks remain fine when they do not duplicate hosted gates
# or mutate privileged host/cluster state
git diff --check

Queued jobs execute in the resolved cwd recorded in the enqueue payload. For agent-branch validation, run the queue command from the agent checkout you intend to validate (or pass --cwd "$PWD") and use caco test run --print --command "<same command>" to preview the checkout before enqueue. Uncommitted files are visible only if they already exist in that resolved checkout when the daemon starts the job; the queue does not copy WIP into the canonical daemon checkout.

Restarts, timeouts and diagnostics

• Daemon restarts don't lose jobs. Queued jobs are re-pended in order. A running job whose process is still alive is re-adopted; one that finished while the daemon was down is finalized from its recorded exit status. Only an unrecoverable job fails, as a retryable infrastructure error (failure_kind: "daemon_restart_recovered") with a retry command, so automation can tell it apart from a real test failure. The reintegration gate is a queued test job and survives restarts too.
• Zero-test runs fail. A cargo test whose filter matched nothing fails validation instead of passing vacuously; Cargo filters are substring matches, and only one positional filter is allowed. Set CACO_ALLOW_ZERO_EXECUTED_CARGO_TESTS=1 in the command when that is intentional.
• Progress while waiting. --wait reports queue position and running counts; caco test logs / caco build logs show a start marker before the job prints anything.
• Silent Cargo jobs explain themselves. An early package-cache-lock note appears after 60 seconds (or 25% of the runtime cap) without output, and at 80% of the cap the job appends the target directory, lock files and a snapshot of running cargo, rustc and sccache processes.
• Full disks are retryable. ENOSPC in artifacts or output marks the job failure_kind: "disk_exhaustion", with a hint to run caco prune run --cargo-targets or use an isolated CARGO_TARGET_DIR.
• Timeouts kill the whole process group, so no orphaned compiler keeps consuming the host. caco build run --timeout <seconds> overrides the runtime cap per request.

caco test show and JSON test show/list expose each job’s bounded admission observation from the owning node’s drain_pending pass, with observed_at and a typed decision.predicate (plus the exact shared-heavy refusal reason). This is historical evidence, not current eligibility or proof of a leaked reservation. Node capacity, disk floor, project capacity, gate reservation, shared-heavy capacity/memory/release reservation, and load-governor deferrals remain enforced. Predicates after the first refusal are not evaluated; not_selected means the pass stopped before attempting that job, and not_evaluated with no timestamp means no observation in this service epoch. Observations reset on daemon restart, and reads neither acquire permits nor trigger admission. The enclosing response identifies the source node and each record retains its original job/enqueue identity. Cross-node placement and pre-enqueue policy refusals are separate from these owning-node queue observations; there is no implied cluster-wide free-slot guarantee. The supervised drain retries periodically, while completion also drains pending work.

Test launch evidence

New test executions persist a bounded execution object in the existing test job record. Read it with caco test show --project <project> --id <job-id> --json; JSON test list and terminal caco test run --wait receipts carry the same object, and text show/wait render its main fields. These use the existing project-scoped routes and caller node selection: no new cross-node lookup, identity fallback, process scan, or read-triggered admission is added. An unavailable job on that route remains unknown, not proof of global absence.

Absent execution means legacy or not started; null fields mean unobserved, never an invented negative observation. Preparation precedes the existing child-runtime timer, so a preparation snapshot does not assert that timer is armed. The runner persists observations; show/list only read snapshots and do not extend deadlines or change terminal results. Cancellation remains terminal even if later cleanup enriches its evidence. Timeout and execution-error notes append to durable logs instead of truncating the captured output; the rare piped fallback retains its existing partial-output limitation on timeout. This instrumentation only improves future evidence: it cannot recover an old job’s missing launch history or establish its cause. CLI/API support is shared; dedicated TUI, web, and Android presentation remains follow-up work.

Queued Cargo jobs reuse their owner's warm target, target/caco-queue-warm/<key> in the job's checkout. The key separates toolchain identity (launcher, the flake's nixpkgs/rust-overlay/fenix pins, rust-toolchain files and +toolchain selectors) and whole-build compiler flags such as RUSTFLAGS and CARGO_PROFILE_*; package selection, features, profiles and target triples share a key because Cargo already keeps their artifacts apart. An exclusive lease bound to the job's process-group leader lets one live job use the target at a time, including a job that survives a daemon restart. A concurrent job of the same owner gets a fresh per-job target instead of waiting. Completion releases the lease and keeps the build, so the next job compiles incrementally, an interrupted compile keeps its finished units, and the built test executables remain until the owner's next job rebuilds them or the target is reaped. An explicit CARGO_TARGET_DIR anywhere in the command is respected. An unchanged rerun in a leased warm target may finish without compiling anything, so the instant no-compile false-green guard (false_green_no_compile) applies only to per-job and unmanaged targets. An hourly sweep removes unleased warm targets idle for CACO_QUEUED_CARGO_WARM_TARGET_IDLE_HOURS (default 6h) and then the oldest idle ones above CACO_QUEUED_CARGO_WARM_TARGET_MAX_TOTAL_GB (default 60 GiB).

Simple commands and bounded compound lists such as cd crates/caco-config && cargo test or export RUST_MIN_STACK=33554432; cargo test receive the same devshell launcher, parallelism caps and target; the compound form receives them as one leading export. Command substitution, subshells, control flow, background jobs and here-documents are not reinterpreted.

Per-job target/caco-queue-jobs/<job-id> directories are removed on completion. To bound directories orphaned by daemon restarts, the daemon also runs an hourly cross-checkout cleanup based on the newest Cargo build activity (default age floor 6h via CACO_QUEUED_CARGO_TARGET_MAX_AGE_HOURS). On Linux, a /proc command-line guard excludes live job IDs before an additional oldest-activity-first pass reduces the aggregate inactive footprint to CACO_QUEUED_CARGO_TARGET_MAX_TOTAL_MB (default 10240 MiB / 10 GiB); set that cap to 0 to disable the size pass. Hosts without readable /proc automatically use the age-only path. The age setting's 0 value makes every candidate age-eligible; it is not the disable value.

When a concrete uncertainty, test-infrastructure bead, or returned CI failure requires a narrow caco-daemon helper reproduction, use just caco-daemon-helper-test <filter> or just caco-daemon-helper-test-exact <path>. The wrapper queues RUST_MIN_STACK=33554432 cargo test -p caco-daemon --lib ... -- --test-threads=1 without compiling broader daemon targets. The equivalent focused caco-cli file-cache diagnostic is just caco-cli-file-cmd-test [<filter>]. These are diagnostic tools, not mandatory pre-submit rituals.

Projects without interception or hosted required CI may use the direct foreground validation their own policy specifies. Full suites remain explicit, justified work rather than a default managed-agent preflight.

Do not wrap validation in run_in_background or stray background tasks; the hook guards block that session-stranding and host-contention pattern.

Shell fixtures on NixOS hosts

On NixOS fleet hosts /bin/sh sources /etc/set-environment unless __NIXOS_SET_ENVIRONMENT_DONE is set, which replaces PATH, GIT_SSH_COMMAND and PI_* with the operator's values. Cargo sets the guard for every process it runs (.cargo/config.toml), and queued jobs set it too, so a #!/bin/sh fixture keeps the environment its test built. A fixture that runs outside Cargo should still name its transport explicitly, for example GIT_SSH_COMMAND=... exec git ....

Direct Test Execution

Use direct commands only on a developer machine or a project without interception, not on shared managed hosts where the hook asks for queued execution.

# Component tests
cargo test

# Integration tests (hermetic lane)
cargo test --test integration

# With config override
CACO_TEST_CONFIG=tests/fixtures/config.integration.yml cargo test