System

Delivery

A phone should not compile a desktop environment, and a laptop should not spend an hour fetching nixpkgs it already has. Collective builds machines on dedicated cache nodes from immutable revisions, serves the results from its own binary caches, and activates them in a way that never moves a machine backwards.

Switching a machine

cltv switch wraps the platform's rebuild command (nixos-rebuild, darwin-rebuild or nix-on-droid) and does the parts that make it reliable around a private, input-heavy flake.

1. Pick healthy caches

   Every self-hosted cache is probed concurrently for nix-cache-info, with a 2-second connect and 4-second total limit. The substituter list is replaced by the ones that answered, so a dead peer never costs a minute of retries per path. Failed peers stay out even when --retry reruns the rebuild.

2. Fetch private inputs as you

   The rebuild evaluates as root, with no SSH agent. So cltv first asks Nix which input source trees are missing (flake archive --dry-run --json) and fetches only those as the user. The lock has about 140 source trees, including 18 nixpkgs revisions.

3. Pin what was fetched

   Under lazy trees nothing else roots those inputs, so the next garbage collection would delete them and the next switch would copy gigabytes again. They are pinned with indirect GC roots in ~/.config/collective/gcroots.

4. Rebuild, logged

   With --option accept-flake-config true, so caches and keys the new configuration declares are usable during the switch that installs them, and with line-oriented logs that survive SSH.

5. Drain around activation

   Activation restarts the agent daemon, its terminal multiplexer and the self-hosted CI runners. Where the agent orchestrator, Cacophony, offers caco node drain, the new generation is built first, then the node is drained: new work is held, each runner listener stops once its job finishes, and agents are checkpointed and parked. Only then is it activated, and caco node undrain resumes exactly what was parked, even if activation failed. A node that never becomes quiet is released again and not activated (exit 75). Cached activation drains the same way.

The full list, including the fleet options below, is in the CLI reference. cltv nix revert rebuilds from the store copy of the flake, which undoes a specialisation switch without picking up new changes.

Prebuilds on cache nodes

Six WSL devboxes run Attic binary caches. Each registry machine may name one of them in prebuild; that node's collective-cache-prebuild service is enabled automatically and builds its assigned machines on a two-hour timer. At this revision twelve machines are prebuilt, including every phone and the Mac's Linux VM.

Each run writes a status record, and each successful machine a publication. A running build or an old failure is never mistaken for a new publication.

{ "schema": 1, "node": "lantern", "state": "success", "message": "prebuild completed",  "startedAt": 1760651220, "finishedAt": 1760655983, "exitCode": 0 }

The service materialises only its own node's push token from SOPS, sets GIT_LFS_SKIP_SMUDGE=1 so identical revisions hash identically on every builder, and opts out of restart-on-change so an ordinary switch of the builder does not discard hours of work. Phones and the aarch64 VM are built under emulation on x86 nodes with two jobs of two cores and a 24-hour budget.

Why it looks like this

Most of these properties were added after a specific failure:

Self-hosted binary caches

The caches are Attic servers on six devboxes, reached over the tailnet. Machines prefer them to the public caches, and their signing keys are part of every machine's Nix configuration.

Peer caches are declared per machine, never in the flake's nixConfig: accepted flake configuration would otherwise bring offline peers back after the switch-time health filter removed them.

Activation that only moves forward

On a machine with a prebuild assignment, cltv switch --cached needs no evaluation at all, which matters on a phone. It runs scripts/activate-cached, whose cache map is a literal list so it works before any flake is evaluated.

The rule is ancestry, not timestamps. The installed revision comes from the record of the last cached activation, or from the system's configuration revision. The published revision must descend from it, so a stale publication, a rebuilt older branch or a divergent experiment is refused. Generations from before revisions were recorded are migrated once: a publication newer than the installed profile is accepted, and ancestry applies from then on.

cltv switch --cachedactivate-cached: quarry via lantern  revision: 40-hex revision  built:    build time  path:     /nix/store/…-nixos-system-quarry-version

Switching many machines

The everyday route is the justfile. Each machine gets its own tmux window over SSH, pulls the checkout and switches, using --cached only where a prebuild assignment exists:

just collective-switch all --cachedjust collective-switch quarry orchard

There is also an optional control plane, cltv-daemon: a small Rust service with GET /status (identity, current generation, last switch) and POST /switch, which runs cltv switch and streams its output. cltv switch --all or --machines drives it from a ratatui dashboard. It is off by default; its access control is the tailnet ACL, so it should only be enabled on machines reachable by trusted peers.

FlakeHub, alongside

FlakeHub is being evaluated next to Attic, not instead of it. A workflow on every push to main builds the same twelve closures on the same self-hosted builders as the prebuild service and pushes them to FlakeHub Cache, which only accepts uploads from CI. Machines read it last, and an expired token produces a warning, never a failed build. Publishing the flake itself waits on its size: the source archive is larger than FlakeHub's limit, almost all of it wallpapers.