GitHub Codespaces Nodes

Run a temporary, browser-accessible Cacophony node from GitHub Codespaces.

When to use this path

• Spin up a clean-room node for a one-off experiment.
• Share a session with a collaborator who has repository access.
• Burst agent work away from your local CPU.
• Attach from the web, TUI, or Android surfaces through the rendezvous path.

Minimal flow

REPO="owner/repo"

# 1. Resolve bootstrap material, create the Codespace, install caco there,
#    and enroll it as a dynamic Cacophony node.
caco codespace new --repo "$REPO" --display-name my-experiment

# If your org does not offer the default machine (`basicLinux32gb`), pass
# --machine explicitly or export CACO_CODESPACE_MACHINE.

# To target an enterprise GitHub host (e.g. ghe.example.com), pass --host or set
# CACO_CODESPACE_HOST; it also derives from the --repo/--source-repo URL host.
# caco codespace new --repo "owner/repo" --host ghe.example.com

# 2. Verify the enrolled `cs-<hash>` node is visible.
caco node list
caco codespace ls

# 3. Spawn an agent on the new node.
CS_HASH="8hexhash"
AGENT_ID="agent-id"
caco agent new --node "cs-${CS_HASH}" --type claude --project myproj

# 4. Attach from the usual surfaces.
caco agent attach --id "$AGENT_ID"

What the command does

1. Resolve bootstrap material first

   The bootstrap URL comes from --rendezvous, CACO_BOOTSTRAP_URL, legacy CACO_RENDEZVOUS_URL or pki.bootstrap.public_url; the token from CACO_BOOTSTRAP_TOKEN, legacy CACO_ENROLL_TOKEN, CACO_BOOTSTRAP_TOKEN_FILE, ~/.cacophony/tokens/node.token or the configured pki.bootstrap sources. If either is missing, nothing is provisioned. The URL must be reachable from GitHub: private, loopback, RFC1918, unique-local, link-local and Tailscale/CGNAT literal IPs are refused unless CACO_CODESPACE_ALLOW_PRIVATE_BOOTSTRAP=1 marks an operator-managed tunnel.

2. Create the Codespace

   gh codespace create --machine "$MACHINE" --default-permissions, with no interactive prompts; MACHINE is CACO_CODESPACE_MACHINE or basicLinux32gb. For an enterprise host, --host or CACO_CODESPACE_HOST (or the host in --repo) sets GH_HOST for creation, secrets and the SSH bootstrap alike.

3. Project secrets for first start

   gh codespace user-secret set stores the bootstrap token and URL, CACO_NODE and readable provider material such as CACO_LITELLM_MASTER_KEY, CACO_GOOGLE_API_KEY and CACO_PI_AUTH_JSON as per-Codespace secrets, passed through GH_USER_SECRET_VALUE rather than process arguments. During bootstrap they move into owner-only files under ~/.cacophony/secrets/codespace/, and the CACOPHONY_*_FILE variables point there.

4. Enroll over SSH

   Unless --no-wait is given, it opens gh codespace ssh, clones Cacophony alongside when the target repository lacks its devcontainer hook, installs caco if missing (falling back to cargo install --path crates/caco), runs caco codespace enroll, which joins through /v1/bootstrap/join, writes ~/.cacophony/bin/run-caco-daemon.sh and runs caco up --skip-update. The script is sent as a remote shell command, so repositories that are not Cacophony enroll reliably too.

5. Report

   It prints the Codespace name, the derived cs-<hash> node ID, the secret projection status and the enrollment result. Confirm the node with caco node list or caco codespace ls, then remove or rotate the temporary Codespace secrets.

Security notes

• Bootstrap tokens are sensitive operator bootstrap material and should be rotated or removed from Codespaces secrets after enrollment.
• Do not paste full env / printenv output from a Codespace into logs or beads; first-party bootstrap failure output redacts obvious token/key assignments, but manual shell diagnostics can still expose secrets.
• Operator-host mesh master keys stay on the operator host.
• caco codespace new projects only the repository runtime provider material it can read from the operator host. Add per-task or project-specific extra secrets through GitHub Codespaces user secrets or the repo-owned container secret-file path.
• Routine mesh traffic should continue to use the relay/rendezvous path rather than exposing arbitrary ports.

Implemented commands

Lifecycle commands

codespace_nodes pool (planned, design-stage)

caco codespace new provisions one Codespace at a time, and each becomes a dynamic cs-<hash> mesh peer. For recurring pools, the codespace_nodes config key declares the desired set of Codespace-hosted microVM nodes instead.

• Shipped: the codespace_nodes key and schema, expanded into ordinary node entries at load time; caco codespace status, which compares each declared member with its live mesh state; and caco codespace reconcile --dry-run, which shows which members would be created, left alone, left suspended or skipped. Both are read-only, and reconcile without --dry-run refuses rather than pretending to act.
• Not yet built: the commands that create, suspend or delete Codespaces (reconcile for real, init, down, destroy). They need the operator's GitHub identity plus Tailscale and certificate projection; treat the commands described below as design intent, not something you can run. The design reference is docs/investigations/codespace-nodes-design.md.

Design summary

codespace_nodes:
  cs:                                          # prefix; becomes cs-0, cs-1, ...
    github_identity: harryaskham_microsoft     # which gh account/token
    host_project: infinity-microsoft/picasso   # repo the codespace is created from
    machine: largePremiumLinux                 # 16c/64GB/128GB (verified available)
    replicas: 3
    # optional: variant (isolated default), ts_tag, region/idle_timeout, ...

At config load the codespace_nodes.<prefix> map with replicas: N desugars into N real NodeEntrys merged into config.nodes named prefix-0 .. prefix-(N-1). Each entry gets the standard static-node shape (Tailscale MagicDNS host, microVM SSH, daemon service, speech: disabled, role: default, and health_expectations.expected_unreachable: true so the fleet/health surfaces treat Codespace auto-suspend as expected), plus a provenance marker so the reconcile path and operator surfaces know the entry is Codespace-backed. This is the key design choice: desugar into nodes, not into dynamic_nodes (dynamic_nodes are regex join-templates; codespace_nodes are concrete known-name static nodes). Every existing node resolver — caco node list, caco ssh, @prefix-1, fleet/health, beads routing, TUI Nodes — then sees them as ordinary static nodes with no Codespace-awareness required.

Reconcile commands (read-only views implemented; mutating verbs planned)

The only Codespace-aware operator surface is the reconciler. Its read-only desired-vs-real views have shipped; the mutating convergence verbs remain planned. Each invocation compares the desired pool against live mesh state:

The in-Codespace bootstrap is the same proven path captured in the micro-0 rollout runbook (/dev/kvm access, cloud-hypervisor plus seed cache, caco microvm preflight, guest Tailscale join, caco microvm node up, then caco up --skip-update). See docs/microvm-node-rollout.md for the verified per-node sequence the reconciler will drive.

Why static-node desugaring, not dynamic

The verified feasibility pass (this design's source) confirmed that a GitHub Codespace on Azure exposes /dev/kvm, kvm_intel nested=Y, vmx/svm, cgroup v2, /dev/net/tun, and /dev/vhost-vsock, with passwordless sudo; caco microvm preflight passes every required hardware/kernel check. The only bootstrap gaps are the hypervisor binary and the cache directories, which the per-pool init handles. Because each prefix-K is a concrete known-name node with a stable Tailscale MagicDNS hostname, treating them as static NodeEntrys lets every existing node resolver work unchanged. The separate dynamic_nodes template surface stays reserved for regex-named join patterns (for example the existing cs- rendezvous path used by ad-hoc caco codespace new Codespaces); the two surfaces are complementary, not overlapping.

Decomposition (planned beads)

• cs-config-schema: add the codespace_nodes.<prefix> model + schema entry + unknown-key validation + the expand_codespace_nodes() load-time pass that appends synthetic NodeEntrys to config.nodes with provenance.
• cs-cert-join-projection: authority-side cert issuance plus Codespace secret projection for the joined config, Tailscale auth key, and bootstrap inputs the in-Codespace path expects.
• cs-codespaces-reconcile: the caco codespaces reconcile / init / status / down commands and their convergence loop.
• cs-first-real-init: end-to-end provisioning of the first cs-N node through this path on a 64 GB Codespace.
• cs-micro0-decommission: retire the static micro-0 proof node in favour of the cs-N pool (config slice landed in 7eb8d47164; operator-action destructive cleanup tracked in).

Container-hosted operation

The repo-owned deploy/codespaces/ path runs an already-published Cacophony image inside the Codespace. It is runtime-only: CACO_CODESPACE_IMAGE must point at an image produced by Azure remote build, and the compose file must not grow a local build: stanza.

mkdir -p deploy/codespaces/runtime/{state,secrets}
# Write secret values directly to files instead of exporting plaintext tokens.
# Example: your-secret-command bootstrap-token > deploy/codespaces/runtime/secrets/bootstrap-token
export GIT_SHA="$(git rev-parse --short HEAD)"
export ACR_LOGIN_SERVER="${CACO_ACR_LOGIN_SERVER:?set ACR login server}"
export CACO_CODESPACE_IMAGE="${ACR_LOGIN_SERVER}/cacophony:${GIT_SHA}"
export CACO_CODESPACE_CONFIG="$HOME/.cacophony/config.yaml"
just codespaces-container-validate
docker compose -f deploy/codespaces/docker-compose.yml pull
docker compose -f deploy/codespaces/docker-compose.yml up -d

Secrets today

Codespaces start with no application secrets. caco codespace new only attempts to project bootstrap values used for enrollment: CACO_BOOTSTRAP_TOKEN, CACO_BOOTSTRAP_URL, CACO_NODE, plus legacy aliases CACO_ENROLL_TOKEN and CACO_RENDEZVOUS_URL. The bootstrap path uses per-Codespace user secrets through gh codespace user-secret set NAME --codespace "$CODESPACE_NAME" and passes values through GH_USER_SECRET_VALUE instead of command-line arguments. The repo bootstrap immediately materializes known token variables into files under ~/.cacophony/secrets/codespace/ and unsets plaintext variables before running caco up.

Per-task provider-secret commands are not available yet. For now, use these supported paths and prefer *_FILE variables wherever possible:

• Native devcontainer: set provider secrets with gh secret set --user --app codespaces or the GitHub UI, supplying values from stdin or a file. Example: PROVIDER_API_KEY="${PROVIDER_API_KEY:?set provider API key}"; printf '%s' "$PROVIDER_API_KEY" | gh secret set PROVIDER_API_KEY --user --app codespaces --repos "$REPO". Before deleting a Codespaces user secret, run gh secret list --user --app codespaces and confirm no other Codespace or repository still depends on that user-level value.
• Repo-owned container flow: mount config from CACO_CODESPACE_CONFIG and project secret files under deploy/codespaces/runtime/secrets/.

Troubleshooting

• gh codespace create reports no terminal: use a current caco codespace new build, which passes --machine and --default-permissions; if your org has a different machine allow-list, pass MACHINE="allowed-machine"; caco codespace new --repo "$REPO" --machine "$MACHINE".
• Bootstrap token missing or expired: push fresh CACO_BOOTSTRAP_TOKEN and CACO_BOOTSTRAP_URL values from the operator host, then run caco codespace enroll and caco up inside the Codespace.
• Stale peer pin / rebuild: run caco codespace rekey "cs-${CS_HASH}" on the operator host, then caco codespace enroll --reinit inside the Codespace.
• Bootstrap authority unreachable: current caco codespace new builds reject private literal bootstrap IPs before provisioning. Verify the configured bootstrap authority URL and public reachability from GitHub-hosted Codespaces; if you bypassed the guard with CACO_CODESPACE_ALLOW_PRIVATE_BOOTSTRAP=1, confirm the operator-managed tunnel is active before retrying enrollment.
• caco ssh cs-$CS_HASH / caco @cs-$CS_HASH tui: Codespace nodes are dynamic nodes, but they have a first-party SSH-family exception because GitHub owns the transport. The CLI resolves the derived node ID through caco codespace ls and delegates to gh codespace ssh -c "$CODESPACE_NAME" -- ..., so no dynamic_nodes.codespace.ssh host metadata is required.
• Suspended node looks offline: use caco codespace ls --refresh; suspended Codespaces should render as expected offline rather than generic peer failure.

For design-stage background, use the internal architecture and key-distribution notes in the repository tree. Treat examples for not-yet-shipped subcommands as design notes, not operator commands.