AKS Deployment

Operate Cacophony on Azure Kubernetes Service without public daemon ingress or local image builds.

Current cloud path. AKS is the active managed-cluster direction for Cacophony. ACA remains legacy/reference material unless an operator explicitly reassigns ACA maintenance.

Production-style cluster 101

Use explicit variables for production-style AKS targets so commands do not accidentally select a local daemon or a different Cacophony cluster:

KUBE_CONTEXT="example-aks"
NAMESPACE="cacophony"
HELM_RELEASE="cacophony"
MASTER_POD="cacophony-master-0"
NODEPOOL="system"

Use the explicit cluster target when you want the local CLI to hop through Kubernetes instead of selecting a local project, node, or agent target:

caco "@cluster:${KUBE_CONTEXT}" status
caco "@cluster:${KUBE_CONTEXT}" bd status
caco "@cluster:${KUBE_CONTEXT}" tui
caco "@cluster:${KUBE_CONTEXT}/${NAMESPACE}" status --json true

The @cluster:<kube-context>[/namespace] target means: use the caller's kubeconfig context, pick a Cacophony pod in that namespace, and run the in-cluster caco over private kubectl exec. It does not require public daemon, web, or SSE ingress. Set CACO_AKS_POD when you need to target a specific pod for diagnostics; otherwise the CLI prefers the master pod.

TUI needs a real terminal. caco "@cluster:${KUBE_CONTEXT}" tui uses kubectl exec -it. Run it from a normal terminal or tmux pane and exit with Ctrl-C. Avoid wrapping the TUI in a hard non-interactive timeout, because killing the local side can leave an in-pod caco tui process behind.

Raw Kubernetes access

Prefer the repo-owned Nix shells so kubectl, helm, Azure CLI, and helper scripts come from the expected toolchain instead of the ambient host:

nix develop .#aks-lite --command kubectl --context "$KUBE_CONTEXT" get nodes
nix develop .#aks-lite --command kubectl --context "$KUBE_CONTEXT" -n "$NAMESPACE" get pods
nix develop .#aks-lite --command helm --kube-context "$KUBE_CONTEXT" -n "$NAMESPACE" status "$HELM_RELEASE"

Use .#aks-lite for fast read-only status and dry-run work. Use .#aks for heavier Azure/ACR/nodepool operations.

For the remote web dashboard, keep the cluster private and open a local tunnel instead of exposing ingress:

just aks-web
# then browse http://127.0.0.1:11189

just aks-web forwards local port 11189 to the selected AKS Cacophony pod's dashboard port by default. Override with CACO_AKS_CONTEXT, CACO_AKS_NAMESPACE, CACO_AKS_POD, CACO_AKS_WEB_LOCAL_PORT, or CACO_AKS_WEB_REMOTE_PORT.

Where configuration lives

deploy/aks/: AKS rendering, validation, topology, and production rollout notes.
deploy/helm/cacophony/: Kubernetes manifests and Helm defaults for the Cacophony role StatefulSets.
justfile: operator commands such as aks-web, aks-toolchain-check, aks-deploy-check, aks-deploy-main, aks-nodepool-show, and aks-nodepool-scale.
flake.nix: .#aks-lite and .#aks development shells.
.cacophony/profiles/caco-vm.md and projects.yaml in the separate cacophony-config repository: project/profile wiring for the persistent virtualization owner that retains AKS rollout history while also covering microVM and dynamic-node worker paths.
aks/pool.yaml (in cacophony-config): future Tailscale-backed dynamic pool-dyn-N worker template imported by the checked-in config. The current production proof lists pool-0, pool-1, and pool-2 as static Tailscale-IP nodes in the root config so they do not collide with the dynamic regex. Rotate the tagged/pre-approved auth key with just set-secret-caco-aks-tailscale 'tskey-auth-...', and patch operator SSH public keys into the existing node-secret shape with just set-secret-caco-aks-authorized-keys.

Inside the pod, runtime state is under /var/lib/cacophony, mounted config is at /etc/cacophony/config.yaml, and the canonical project checkout is /var/lib/cacophony/daemon/checkouts/cacophony. The chart mounts writable runtime paths including /home/caco/.ssh and /run, so the shared prelude can materialize SSH identities and sshd can create /run/sshd while the image layer stays read-only. Current images also include /opt/cacophony-seed-checkout; the shared prelude copies that lightweight Git checkout into the daemon checkout path only when the target is missing, leaving existing .git checkouts and non-empty directories untouched. That breaks the bootstrap cycle for fresh PVC nodes whose home config imports repo-managed .cacophony/* files before Git credentials are available. Inspect the live image tag through Helm or Kubernetes status instead of hard-coding registry-specific image names in public runbooks.

Relay, master, and worker pods run long-lived caco supervisor as Kubernetes PID 1. The supervisor performs non-blocking child-reap sweeps between convergence/poll cycles and logs any orphaned helper children it reaps, preventing escaped Git or hook subprocesses from accumulating as zombies until a pod restart.

Updating config or image

When reintegration and deployment are safe, check the decision first and only then run the guarded deploy:

nix develop .#aks --command just aks-toolchain-check acr
nix develop .#aks --command just aks-deploy-check
nix develop .#aks --command just aks-deploy-main

The toolchain preflight proves the host can run the macOS/Nix-sensitive tools, kube context/API probe, Azure CLI login, Python/PyYAML, kubectl, helm, git, and bash needed for rollout. Each subprocess check is bounded by CACO_AKS_TOOLCHAIN_CHECK_TIMEOUT_SECS (default 60) so broken Python/Azure/kube tooling fails before any production mutation. The guarded deploy also runs the relevant preflight automatically so broken hosts fail before ConfigMap/Helm mutations or ACR uploads. The guarded deploy separates three freshness axes: image inputs, Helm/config inputs, and the in-pod repo checkout. It skips the slow ACR remote build for docs-only or checkout-only drift, applies Helm-only changes without rebuilding, and refreshes the in-pod checkout when a source checkout update is enough. If Azure Container Registry source upload fails before build execution with AuthenticationFailed / Signed expiry time SAS-expiry output, just deploy-remotes retries the az acr build upload path with bounded defaults. Tune that with CACO_ACR_BUILD_UPLOAD_RETRIES and CACO_ACR_BUILD_RETRY_DELAY_SECS; this retry is for source-upload SAS expiry only, not failed image builds.

Scaling and node recovery

Use AKS nodepool operations for shutdown or unreachable nodes; do not try to recover production capacity by only clearing Kubernetes taints.

RESOURCE_GROUP="rg-cacophony-example"
CLUSTER="example-aks"
COUNT="11"
nix develop .#aks --command just aks-nodepool-show "$RESOURCE_GROUP" "$CLUSTER" "$NODEPOOL"
nix develop .#aks --command just aks-nodepool-scale "$RESOURCE_GROUP" "$CLUSTER" "$NODEPOOL" "$COUNT"

A recent recovery scaled a system nodepool up before rerunning the Cacophony health smoke. Post-recovery validation should confirm all expected nodes are Ready, all Cacophony role pods are Running, Helm is at the intended revision, beads are fresh, the in-pod checkout matches GitHub main, and no lingering in-pod caco tui process remains after a controlled TUI smoke.

Quick health checklist

nix develop .#aks-lite --command kubectl --context "$KUBE_CONTEXT" get nodes
nix develop .#aks-lite --command kubectl --context "$KUBE_CONTEXT" -n "$NAMESPACE" get pods
caco "@cluster:${KUBE_CONTEXT}" status --json true
caco "@cluster:${KUBE_CONTEXT}" bd status --json true
caco "@cluster:${KUBE_CONTEXT}" exec -- bash -lc 'cd /var/lib/cacophony/daemon/checkouts/cacophony && git rev-parse --short=12 HEAD && git ls-remote origin refs/heads/main | cut -c1-12'

Operator flow

1. Deploy guarded

Prefer just aks-deploy-main: it fetches origin/main, compares image, Helm, and rendered-config inputs, pushes config-only drift, skips slow ACR builds when possible, and refreshes the in-pod checkout when no image, Helm, or config rollout is needed.

2. Inspect privately

Use nix develop .#aks-lite for fast read-only kubectl, helm, and helper commands without pulling in Terraform or a local Cacophony package build.

3. Isolated board standby

Production AKS defaults the cacophony project to isolated aks-beads until canonical beads catch-up is proven safe; keep AKS free of autonomous bead-writing agents, and set CACO_AKS_BEADS_BRANCH=beads only for explicit canonical-board validation or recovery drills.

Guarded rollout and remote build

git fetch origin main
nix develop .#aks --command just aks-toolchain-check acr
nix develop .#aks --command just aks-deploy-check
nix develop .#aks --command just aks-deploy-main

Use aks-toolchain-check acr before routing rollout work to a host; the mutation recipes run it automatically and stop early on broken macOS/Nix code signatures, missing kube context, stale Azure login, or missing Python/PyYAML/kubectl/helm. Use aks-deploy-check first when you want the decision without mutating AKS; it reports whether the guarded path would refresh the checkout, push the rendered ConfigMap, apply Helm-only drift, or run an ACR image build.

Default path: aks-deploy-main is the default AKS freshness path: it avoids an image rebuild for documentation, bead, Android/macOS, web-only, profile-note, checkout-only, and Cargo-version-only drift by default; pushes AKS config changes with just aks-push-config; applies Helm chart changes without rebuilding the image; and rebuilds for Dockerfile changes, first-party runtime crate changes used by the pod, or shared container prelude changes.
Config changes: Reintegrating a repo config change updates the source of truth; the running cluster observes it only after the rendered ConfigMap is replaced and the StatefulSets restart. The self-contained renderer also filters repo-owned Pi overlay mixins that the current in-cluster image may not ship yet, including pi-inbox, from project agent defaults and interactive Pi defaults; validate-self-contained-config.sh asserts those unsupported profiles are absent before a ConfigMap is applied.
Forcing and pinning: Set CACO_AKS_BUILD_ON_VERSION_ONLY=1 when a Cargo metadata-only diff should force a rebuild. Set CACO_AKS_DEPLOY_REF=<sha-or-ref> when a production rollout must pin the ConfigMap diff, ACR archive/image tag, and Helm image tag to one captured commit instead of chasing a moving origin/main during a long ACR build. Use just deploy-remotes only when an unconditional ACR remote build is required; that path archives the selected deploy ref (default origin/main), uses configured ACR-hosted base image mirrors, and waits for the Helm release to converge.
Uploads: The repo .dockerignore excludes root static/ artwork and .cacophony/agent/** state-branch summaries from Docker/ACR source uploads; live pods consume those from checkouts/config or the Cacophony state branch, not from the canonical image.
Overrides: Common overrides include CACO_AKS_DEPLOY_REF, CACO_AKS_HELM_RELEASE, CACO_AKS_NAMESPACE, CACO_AKS_HELM_TIMEOUT, CACO_AKS_RELAY_PERSISTENCE_SIZE, and CACO_AKS_BEADS_BRANCH.

Live binary update path

Pool runtime, updates and SSH

Live updates on a read-only image. The container prelude points CACO_INSTALL_DIR at PVC-backed /var/lib/cacophony/bin and puts it first on PATH, so caco update can stage and promote a new binary without touching the image's /usr/local/bin/caco. An empty directory falls back to the image binary. Prefer a guarded ACR image rebuild plus Helm rollout while published Linux binaries are not guaranteed to match the image's glibc.
Private release checks. Set github.tokenSecretName (and optionally github.tokenKey) in the chart to project a token as GITHUB_TOKEN / GH_TOKEN for release metadata; do this only for an intentional rollout.
First-boot checkout. The prelude seeds the daemon checkout from /opt/cacophony-seed-checkout; set CACO_CONTAINER_SEED_CHECKOUT_ENABLED=0 only when a deployment supplies the checkout another way.
SSH. The chart projects authorized keys and runs sshd on port 2222 over the pod's Tailscale IP (password login disabled). SSH sessions inherit CACOPHONY_DIR=/var/lib/cacophony and the pod's CACO_NODE, so caco ssh pool-N followed by plain caco status reads the supervised runtime.
Pool rollout. just aks-pool-dry-run renders the separate caco-aks-pool release; CACO_AKS_IMAGE_TAG=<tag> just aks-pool-upgrade rolls it out with checkout-backed live config, so pool config converges with the root cluster. caco @pool-N status --json should succeed before and after.

Scheduled image refresh

The production cron aks-image-refresh runs the same guarded path through scripts/aks-image-refresh.sh / just aks-image-refresh. It takes a lock to avoid overlapping ACR builds, defaults to the live AKS context and namespace, preserves StatefulSet PVCs such as /var/lib/cacophony, and disables retired ACA rollout unless explicitly overridden. Use CACO_AKS_IMAGE_REFRESH_DRY_RUN=1 or just aks-image-refresh-check for no-mutation validation.

Private inspection

KUBE_CONTEXT="example-aks"
NAMESPACE="cacophony"
nix develop .#aks-lite --command kubectl --context "$KUBE_CONTEXT" get nodes -o wide
nix develop .#aks-lite --command env CACO_AKS_CONTEXT="$KUBE_CONTEXT" CACO_AKS_NAMESPACE="$NAMESPACE" just aks-local-status-lite

The aks-lite shell intentionally excludes Terraform and the local caco package build. Config-rendering helpers use ambient caco on PATH or an explicit CACO_BIN. Use nix develop .#aks for the heavier Azure/ACR deploy shell when you need image rollout or nodepool recovery commands.

Private `@cluster` target

The @cluster:<kube-context>[/namespace] target executes the in-pod caco through the caller's kubeconfig. It is private Kubernetes exec, not public daemon, SSE, or web ingress.

KUBE_CONTEXT="example-aks"
caco "@cluster:${KUBE_CONTEXT}" status --json
caco "@cluster:${KUBE_CONTEXT}" bd sync
caco "@cluster:${KUBE_CONTEXT}/cacophony" bd list --project cacophony --status open --limit 20

TUI needs a real TTY. One-shot commands such as status and bd sync can run non-interactively. caco "@cluster:${KUBE_CONTEXT}" tui uses kubectl exec -it, so run it from a real terminal or tmux pane. A non-interactive failure such as Unable to use a TTY is expected.

Current validation snapshot

For each AKS rollout smoke, verify that all role StatefulSets converge, expected cert-authority health annotations are understood, the configured beads branch reports fresh sync (isolated aks-beads by default, or canonical beads only for an explicit validation/recovery drill), unsupported Pi-only profile mixins are absent from the self-contained ConfigMap, and the private TUI path works through a real tmux TTY via kubectl exec -it without exposing daemon, SSE, or web ingress publicly. The first Tailscale pool proof ran pool-0, pool-1, and pool-2 as a separate Helm release with stable tailnet IPs and beads_host.reachable=true via the root cluster; automatic bootstrap renewal still needs the bootstrap-join EOF fixed before this is hands-off.

Treat each validation as a fresh run: fetch origin/main, run aks-deploy-check to review the planned action, run the guarded AKS freshness path when mutation is intended, inspect role readiness, check the expected beads-branch sync mode, and rerun a smoke test for caco "@cluster:${KUBE_CONTEXT}" tui from a real terminal.

Pool nodes pool-0 to pool-2 are ordinary nodes in the root config with Tailscale IPv4 hosts; pool-dyn-N is reserved for future dynamic leases. Rollout history and health sweeps live in deploy/aks/PRODUCTION-ROLLOUT.md in the repository. If you connected to a pod before SSH was enabled, remove its stale [<tailscale-ip>]:2222 host key before retrying.

Nodepool recovery

If all role pods are pending because AKS nodes are NotReady or tainted node.kubernetes.io/unreachable, recover the nodepool first instead of only clearing Kubernetes taints.

RESOURCE_GROUP="rg-cacophony-example"
CLUSTER="example-aks"
NODEPOOL="system"
COUNT="6"
just aks-nodepool-show "$RESOURCE_GROUP" "$CLUSTER" "$NODEPOOL"
just aks-nodepool-scale "$RESOURCE_GROUP" "$CLUSTER" "$NODEPOOL" "$COUNT"

When the fresh VMSS nodes become Ready, Cacophony role pods should reschedule and the next Helm upgrade can converge the release record. The nodepool helpers classify Azure AuthorizationFailed output so operators can tell when the hermetic toolchain works but the current principal lacks AKS cluster / agentPool RBAC; a privileged Terraform apply can set operator_recovery_principal_ids in deploy/aks/terraform.tfvars for declarative AKS-scoped nodepool inspect/scale grants.

PKI and health expectations

The CA role may intentionally run caco cert serve instead of caco daemon. In that topology daemon health can appear unreachable for the CA while bootstrap HTTPS is healthy; mark that as an expected health annotation rather than treating it as a cluster outage.

The root cluster's pki.bootstrap block is optional. Independent AKS clusters can run their own CA/bootstrap surface and should not inherit a deleted or unrelated root-cluster bootstrap endpoint.

AKS Deployment

Production-style cluster 101

Raw Kubernetes access

Where configuration lives

Updating config or image

Scaling and node recovery

Quick health checklist

Operator flow

1. Deploy guarded

2. Inspect privately

3. Isolated board standby

Guarded rollout and remote build

Live binary update path

Pool runtime, updates and SSH

Scheduled image refresh

Private inspection

Private @cluster target

Current validation snapshot

Nodepool recovery

PKI and health expectations

Further reading

Private `@cluster` target