micro-0 full-node microVM rollout runbook

Operator-facing, step-by-step guide for bringing up the first persistent full-node Cacophony microVM (micro-0) on a Linux KVM host. It ties together the read-only prep surfaces (caco microvm node plan / status / up --dry-run), the host preflight, the per-variant Nix images, and the caco microvm node up / down / destroy wrappers around the repo-owned proof harness.

For the image internals (flake outputs, runtime shape, security model, and harness mechanics), consult the internal investigation note in this repository.

Scope. Booting a microVM requires Linux + KVM; it cannot run on macOS. The prep surfaces in steps 2–4 are deliberately KVM-independent and safe to run anywhere (including a macOS operator host) so the launch can be inspected and previewed before a host is provisioned.

One-shot vs manual: cs node bring-up & recreate

This is the at-a-glance audit of which steps are automatic vs which still need an operator or authority action. Detailed procedures are in the sections below.

Automatic (cold-start / cs-node-up / flake oneshots)

One-time / authority / operator actions

Recreate an existing node (vs new node)

A recreate keeps the persistent volume, so disks, certs, git-key source paths, and gh auth typically survive — cold-start + cs-node-up bring it back to 6/6 on their own. A brand-new node additionally needs the one-time authority actions above (cert issuance + first git-key staging) before its checkouts can materialize. A node that has lost its guest SSH key cannot have access re-injected by mounting disks — re-provision cleanly with the bootstrap store in place from the start.

1. Current ready state

The config and certificate contract for micro-0 is already in place; only the KVM host and the real boot remain:

• Declared static node. micro-0 is declared in the separate cacophony-config repository (host micro-0.example.ts.net, daemon bound on 0.0.0.0, marked expected_unreachable until it boots and joins the Tailnet).
• TLS pre-issued. Reusable node TLS material was issued on the PKI authority (helsinki) with caco cert issue --node micro-0, producing pki/nodes/micro-0.crt and pki/nodes/micro-0.key. No bootstrap token, provider key, SSH key, or certificate is baked into the image.

Confirm the read-only contract at any time, from any host:

caco microvm node status --name micro-0

On a host without KVM this reports launcher state: not_provisioned, can boot here: no, the resolved state dir, and the cert contract — all without booting anything.

2. Provision a Linux x86_64 KVM host

Pick a Linux node with /dev/kvm. The current fleet punch-list:

On the chosen host, install the hypervisor and (re)check readiness:

# On the KVM host:
nix profile install nixpkgs#cloud-hypervisor

# Seed the cache directories if missing:
mkdir -p "$CACOPHONY_DIR/microvm/cache"/{kernels,rootfs,snapshots}

# Run the host preflight (treat ready=false as a hard placement blocker):
CACHE_DIR="$CACOPHONY_DIR/microvm/cache"
caco microvm preflight --cache-dir "$CACHE_DIR"

caco microvm preflight gates on KVM, tun/tap, vsock, cgroup v2, nftables, the hypervisor binary, and the cache directories. Do not proceed until ready=true.

3. Preview the launch (no boot)

From any host, emit the structured launch plan to review exactly what up will build and run:

caco microvm node plan --name micro-0 --variant isolated

This prints the target system (x86_64-linux), whether the current host can boot it, the per-variant Nix attributes (.#caco-microvm-node-isolated-runner and .#caco-microvm-node-isolated-image-info), the cert/identity paths, and the exact boot-harness invocation — without building or booting anything. JSON is available with --json true.

Variants:

4. Dry-run the real preparation (no boot)

On the provisioned KVM host, run the real up preparation phase — variant validation, the preflight boot gate, Nix attribute resolution, and the boot plan — while still stopping before any Nix build or boot:

caco microvm node up --dry-run --name micro-0 --variant isolated

When the host preflight is ready this reports would boot: yes; when it is not (for example on macOS) it reports would boot: no with a refusal reason, mirroring what a real up would do. This is the safe last check before booting.

5. Boot micro-0

On a ready Linux/KVM host, the first-class caco microvm node up path now wraps the repo-owned boot-proof harness in detached, keep-running mode and writes a launcher.json marker for caco microvm node status:

caco microvm node up --name micro-0 --variant isolated

The command refuses before booting when preflight is not ready, creates the persistent state directory under $CACOPHONY_DIR/microvm/nodes/micro-0 unless --state-dir overrides it, invokes scripts/microvm-node-boot-proof.sh with --keep-running --detach, and records the tmux session plus boot.json report path in launcher.json. Inspect launch state with:

caco microvm node status --name micro-0
tmux attach -t caco-microvm-node-micro-0

For lower-level incident proofing, the harness is still callable directly:

scripts/microvm-node-boot-proof.sh --variant isolated --json-report micro-0-boot.json

The harness builds the runner and image-info outputs, creates a proof workdir, runs caco microvm preflight, starts microvm-run under a wall-clock bound, captures console.log, waits for the Cacophony-node and rich-tool readiness patterns, records proof.env / the JSON report, and shuts the VM down unless --keep-running is passed. On Linux hosts with /dev/vhost-vsock, ssh, a VSOCK-capable socat, and guest SSH credentials, add --guest-probe required for a stronger in-guest proof. The default guest probe runs caco-microvm-health-check, which captures caco status, verifies the cacophony-node service, writes /tmp/caco-dbus-daemon-count, and fails if more than three dbus-daemon processes are present so session-bus runaway regressions are visible in boot proof evidence.

For real cluster membership (not the single-node fallback), mount or materialize a joined config plus the pre-issued PKI material under /var/lib/cacophony before/at boot so micro-0 comes up as the declared Tailnet node rather than a standalone single-node config.

6. Stop without deleting state

To stop a registered persistent node after validation or incident proofing, run:

caco microvm node down --name micro-0

down is idempotent and safe on non-KVM hosts: if no launcher.json marker exists, it reports a clean no-op. When a launch is registered, it best-effort kills the detached tmux session, runs the runner microvm-shutdown helper when runner.path is present, clears launcher.json, and leaves the state directory for future launches.

7. Destroy when state should be discarded

down preserves the state directory for the next launch. To delete that state, preview first:

caco microvm node destroy --name micro-0

Then pass --force only when the persistent state should be removed:

caco microvm node destroy --name micro-0 --force

The forced path best-effort stops a registered tmux session, runs runner microvm-shutdown when available, and then removes the state directory.

8. Validate

After boot, from the guest (or via the harness VSOCK probe):

caco status --json
REPORT="microvm-validation.json"; caco microvm validate --report "$REPORT"

caco microvm validate records the isolation/limit/cleanup evidence (host paths outside projected mounts denied, unprojected sockets inaccessible, CPU/memory/ disk ceilings hit, tap/vsock/cgroup cleanup, startup/teardown timings).

Status surfaces summary

Tailnet cs-N ghost devices

Tailscale does not free a device NAME when the device goes offline. When a cs-N microVM is recreated and its persisted tailscaled.state is lost (fresh state disk, removed device key, etc.) it re-registers FRESH, and because the offline cs-N ghost still holds the name, Tailscale appends a suffix (cs-N-1, cs-N-2, ...). Those suffixed devices are invisible to the mesh config (which only knows cs-N), so the node effectively drops out. Cold-start (cs-microvm-coldstart.sh) tries to avoid this by preserving tailscaled.state, but that is fragile; the durable fix is to DELETE the stale offline ghost so the canonical cs-N name is freed before the fresh node joins.

deploy/codespaces/cs-tailnet-ghosts.sh makes this visible and actionable:

# report (read-only, no creds; runs on any node that sees the tailnet):
deploy/codespaces/cs-tailnet-ghosts.sh                 # cs-0,cs-1,cs-2
deploy/codespaces/cs-tailnet-ghosts.sh --node cs-0     # one base node

# delete offline ghosts (frees the squatted cs-N name) — needs admin API creds:
TS_API_TOKEN=tskey-api-... TAILNET=example.ts.net \
  deploy/codespaces/cs-tailnet-ghosts.sh --delete --node cs-0 --yes

The join auth-key (tskey-auth-...) cannot delete devices and the tailscale CLI has no device-delete; deletion requires a Tailscale admin API token (tskey-api-... or an OAuth client with device scope) plus the tailnet name. Online devices are never deleted. Operator dependency: cs-microvm-coldstart.sh now runs a best-effort ghost-delete before the guest boots when both $BOOT/tailscale-api-token and $BOOT/tailnet are present in the bootstrap store (it shells out to cs-tailnet-ghosts.sh --delete --node cs-N --yes), and is a strict no-op / tailscaled.state-preservation fallback when they are absent. The only remaining operator step is to provide the token: commit the Tailscale admin API token (tskey-api-... or an OAuth client with device scope) plus the tailnet name once as a SOPS-encrypted file under deploy/codespaces/secrets/*.sops.yaml (the caco / caco-work age identities are already readers; see .sops.yaml) and have the bootstrap-store population decrypt it into $BOOT/. Until that token exists, ghost cleanup stays a manual operator action and recovery depends on tailscaled.state durability.

cs daemon cluster-bind (bind_host)

The cs guest frequently runs a minimal fallback config.yaml whose node host: is a hostname (cs-N) or a churned/stale ghost IP and whose bind_host is null. With a non-local/hostname peer address the daemon aborts at startup: invalid cluster bind address ... must set nodes[].services.caco-daemon.bind_host to a bindable local address such as 0.0.0.0. This silently leaves cs-N at 0 agents (daemon never started) after a recreate — distinct from the repo node config, which already sets bind_host: 0.0.0.0 for cs-0/cs-1/cs-2. cs-node-up.sh now patches bind_host: 0.0.0.0 into the guest config before starting the daemon (idempotent; persists on the /var/lib/cacophony volume), so the cluster listener binds locally regardless of the advertised dial address. If a cs daemon won't start, check caco status for this error and confirm the guest config's bind_host.