Collectivenix fleet

30 machines · 4 platform classes · 1 registry

In use

Operations

Running thirty machines is mostly about knowing what is true without disturbing it: which secret a service can read, which machine has gone quiet, which binary is really on the path. These are the tools and rules for that.

Secrets

From a machine's key to a service's secret fileA line from left to right. Each machine has its own SSH key. Its age identity is derived from that key with ssh-to-age, precomputed in collective-lib or computed during evaluation. The age identities are the recipients in a generated .sops.yaml, and one secrets.yaml file is encrypted to all thirty recipients. At activation, sops-nix decrypts only the secrets a module declares, with the owner and mode it asks for, into one of two trees: the system tree under /run/secrets for system services, or the home tree under ~/.config/sops-nix/secrets for user services. machine key per-machine SSH age identity ssh-to-age recipients generated .sops.yaml secrets.yaml 30 recipients sops-nix at activation system tree /run/secrets home tree ~/.config/sops-nix system services user services Only declared secrets are materialised, each with its own owner and mode. The prebuild service, for example, gets just its node's cache push token.
Secrets.One encrypted file, many recipients, and only the declared secrets decrypted on each machine.
CommandDoes
cltv secret editEdit the encrypted secrets file with SOPS
cltv secret reinstallRe-run sops-nix activation to install new or changed secrets
cltv secret getPrint one secret, for the rare case a human needs it

On Android there is no activation daemon, so a supervised watcher checks every minute whether secrets need installing.

Diagnosing without reading

When a service reports a missing secret, the rule is to prove what is wrong without ever printing a value. Start from the exact path the failing service is configured with, in the right tree: system services read /run/secrets, user services read ~/.config/sops-nix/secrets, and the health of one says nothing about the other.

secret=/run/secrets/keys/example/tokentest -r "$secret" && stat -c 'mode=%a owner=%U' "$secret" && test "$(wc -c < "$secret")" -gt 0

Fleet liveness

Every managed machine reports that it is alive every five minutes to a receiver run by the agent orchestrator, which keeps a private last-seen record and decides whom to notify. There is no automatic remediation.

PlatformReporter
NixOS, including WSLA user oneshot service on a five-minute timer, bounded to 30 seconds
macOSA launchd agent, at load and every 300 seconds
AndroidA supervisord program sleeping five minutes between bounded attempts, with no offline queue
Windows host of a devboxA SYSTEM scheduled task running a resident PowerShell loop, independent of any login, WSL, Nix or tailnet client

Reports carry only a hostname and a layer, over verified TLS, refusing redirects, with a token read from SOPS at run time. The Windows reporter is the interesting one. When WSL is wedged, the machine still says whether its host is up, because the reporter uses Windows' own HTTPS stack. Its files are staged from WSL during an ordinary switch, and the operator approves one UAC prompt to install the task.

Trust modelThe token is shared, so any holder could report as another host. This is availability telemetry, not attestation, and a live report means the managed user service responds, not that every component is healthy.

What is really on the path

Home Manager owns installed binaries, but a cargo install or a copied binary in ~/.local/bin can shadow them and survive every switch. sinbin takes such copies out of the path across machines, after checking that doing so is not a downgrade, and can put them back:

sinbin APP --dry-run               # show what would move, everywheresinbin APP                         # all standard and Android machinessinbin APP --nodes local           # only this machine, no SSHsinbin --undo APP

Copies go to ~/.local/sinbin and ~/.cargo/sinbin, never deleted.

Safe quick commits

Agents and people commit constantly, from many machines, to the same branches. Two helpers make the common cases safe:

CommandDoes
gic [message]Commit tracked changes and staged files locally, generating a message from the change if none is given; works offline
gii messageCommit, sync with the origin branch, then push
gic --amendReplace a local commit, refusing if any remote branch or tag already contains it

Diverged histories are merged, not rebased. That may add a merge commit, but a plain rebase can drop changes that exist only in a merge and then push the deletion silently.

Maintenance

CommandDoes
cltv nix gc -t 14dCollect garbage older than a period, after confirmation
cltv nix repair storeVerify and repair the Nix store
cltv nix gc reap-zombies-nodReap orphaned supervisord children that block garbage collection on Android
cltv nix specialise -s NAME, cltv nix revertSwitch to a specialisation, and back without new changes
cltv kv get|set|rmA small local key-value store for scripts' persistent settings
cltv debugTurn on verbose logging for every cltv command, persisted in that store

Runbooks

Incidents end with a runbook in the repository's docs/runbooks/: what failed, the evidence, what changed and how to validate it. The 35 there cover adding machines and caches, cached switches, six prebuild failures, login recovery, secrets diagnostics, liveness, Tailscale and Hyprland on Android, GPU toolchains, trackpads and displays. The design described on this site is largely their outcome, which is why so many sections here explain a rule by the failure behind it.