Collectivenix fleet

30 machines · 4 platform classes · 1 registry

System

Fleet

Every machine is one entry in machines/registry.nix. The flake builds configurations from it, modules read it as constants, and scripts read a plain-data copy. No second list has to be kept in step unless it must work before Nix evaluates.

Public siteThis site never names machines or addresses. Examples use invented names such as quarry and lantern; counts are from the registry as of this revision.

One registry

One entry, many consumersOn the left, a registry entry for a NixOS laptop named quarry: its typed system, groups, whether it is live yet and the cache node that prebuilds it. In the middle, the flake turns the registry into three things: the machine's configuration, constants.registry for modules, and the plain .#registry output. Modules on every machine read constants.registry. The plain output is read by scripts and just recipes as fleet lists, by the cache node's prebuild service as its mappings, by credential recovery as the token broker list, and written to a registry.json file in every home. machines/registry.nix quarry = { system = System.NixOS.X86; groups = [Group.Personal]; pending = false; prebuild = "lantern"; }; typed, checked at evaluation nixosConfigurations.quarry the machine constants.registry for modules .#registry plain data modules on every machine scripts, just recipes fleet lists prebuild service cache mappings login recovery token brokers Also written to ~/.config/collective/ registry.json
One entry, many consumers.Adding a machine to the registry creates its configuration and updates every derived list in the same evaluation.

Only system is required. Everything else has a default, and the type of system chooses the builder, the module tree and the defaults for the rest.

FieldDefaultMeaning
systemrequiredA typed system from collective-lib's data.nix, optionally tagged: Tagged [VM WSL] System.NixOS.X86. Home-only entries give an architecture instead.
homefalseA standalone Home Manager configuration with no system layer.
usernullLogin user; null means the default for the system type.
sshPort22Android's sshd listens on 8022.
groups[]Members of the Group enum: Work, Personal, Android. They select fleet lists.
pendingfalseDefined but not reachable yet; excluded from every fleet list.
prebuildnullThe cache node that builds and publishes this machine's activation closure.
atticnullSet on a machine that serves an Attic cache: { publicKey; }. URL and cache name are derived.
azureTokenBrokerfalseThis machine's Azure login mints tokens for machines that cannot sign in.
vmnullA guest run by another registry machine: host, runner, cpus, memoryMiB, diskGiB, mac, shareHome, autostart, nested, rosetta.

An entry for a new laptop, prebuilt by a cache node:

quarry = {  system = System.NixOS.X86;  groups = [Group.Personal];  pending = true;         # out of fleet lists until it is live  prebuild = "lantern";   # the cache node that builds its closure};

Platform classes

A system's type picks one of four builders. Tags change assumptions within a class rather than creating new ones.

ClassBuilderArchitecturesMachines
NixOSlib.nixosSystemx86_64-linux, aarch64-linux22
nix-darwinnix-darwin.lib.darwinSystemaarch64-darwin1
Nix-on-Droidnix-on-droid.lib.nixOnDroidConfigurationaarch64-linux5
Home ManagerhomeManagerConfigurationx86_64-linux2

The 22 NixOS machines are 11 bare-metal x86_64 machines, 9 WSL devboxes, one aarch64 board and one aarch64 VM guest of the Mac. Their roles span desktops, laptops, handheld PCs, servers and a microVM.

Typed systems and tags

Systems are values from collective-lib's data.nix, not strings. Each enum value carries a switch that dispatches on it, so code that forgets a platform fails to evaluate instead of quietly doing nothing.

rebuildCmd = system.systemType.switch {  NixOS = "sudo --preserve-env=SSH_AUTH_SOCK env NIX_REMOTE=daemon nixos-rebuild";  NixOnDroid = "nix-on-droid";  NixDarwin = "sudo --preserve-env=SSH_AUTH_SOCK /nix/var/nix/profiles/system/sw/bin/darwin-rebuild";};
TagEffect
WSLAdds the NixOS-WSL module with wsl.enable and a WSL state version. The devbox module then converges the Windows host too.
VMA virtual guest: hardware-specific configuration is skipped.
HomeOnlyNo system layer. Exported to scripts as __COLLECTIVE_HOME_ONLY=1.

Architecture is typed the same way. Upstream flakes that only work on x86 are masked rather than failing: on aarch64 the gaming module becomes an option-only stub, so unrelated settings such as game.enable = false still resolve.

Groups and fleet lists

The registry exports plain lists for anything that is not a Nix module. The sizes at this revision:

nix eval --json .#registry --apply 'r: builtins.mapAttrs (_: builtins.length) r.lists'{"android":5,"androidNodes":5,"atticPublicKeys":6,"atticSubstituters":6,"azureTokenBrokerSsh":13,"azureTokenBrokers":13,"cacheBuildMap":11,"cacheNodes":6,"linuxNodes":13,"nodes":20,"personal":6,"work":14}
ListHoldsRead by
nodesSSH targets of the live work and personal machines, by MagicDNS name (no addresses are recorded)Fleet just recipes, tmux fan-out
linuxNodes, androidNodesSubsets by platformPlatform-specific recipes
cacheNodes, cacheBuildMapCache servers, and which machine each buildsPrebuild service, cache status
atticSubstituters, atticPublicKeysCache URLs and signing keysNix configuration on every machine
azureTokenBrokersMachines whose Azure login can serve othersCredential recovery
work, personal, androidGroup membersWorkspaces and fleet recipes

A copy of the plain registry is also written to ~/.config/collective/registry.json on every machine.

Lists that stay explicit

A few lists must work before the flake can be evaluated, such as the cache table a fresh machine needs to download its first closure, or the cache a phone activates from. They stay as literal lists in shell scripts and the justfile, between markers that make them easy to find:

# COLLECTIVE_MACHINE_LIST_START prebuild-mappings: keep in sync with machines/registry.nix `prebuild`case "$machine" in  quarry) cache=lantern ;;  …esac# COLLECTIVE_MACHINE_LIST_END prebuild-mappings
rg -n 'COLLECTIVE_MACHINE_LIST_(START|END)' ~/collective

Adding a machine

  1. Copy the nearest machine directory

    machines/name/ holds only what is that machine's own: nixos/configuration.nix and its hardware file, nix-darwin/ or nix-on-droid/configuration.nix, and home-manager/home.nix.

  2. Add the registry entry

    With pending = true until the machine is reachable. This alone creates its configuration and updates the derived lists, prebuild mappings and broker lists.

  3. Keys and secrets

    Add its SSH public key and age key to collective-lib's key lists and re-encrypt the secrets it needs.

  4. Evaluate

    nix eval --json .#registry.listsnix eval --raw .#nixosConfigurations.name.config.system.build.toplevel.drvPath
  5. Build on its cache node first

    So the first switch downloads instead of compiling.

  6. Switch, join the tailnet, go live

    Run cltv switch, then tailscale up, set pending = false, update the explicit lists, and switch the machines whose lists changed.

VM guests

A registry machine with a vm attribute set is run by its host. On the Mac, every guest whose vm.host is the Mac gets:

The guest assumes only virtio and EFI, so linux-vm runner name krunkit boots the same disk under another runner. linux-vm stop holds a guest down across switches and reboots until it is started again.

Working across the fleet

The justfile turns the fleet lists into operations. Each fan-out opens a tmux window per machine over SSH, so long commands stay visible and interruptible.

RecipeDoes
just nodes, android-nodes, linux-nodesPrint SSH targets: explicit lists kept in step with the registry
just cluster-tmux-run alias cmd [nodes]Run a command on each machine in its own tmux window
just collective-pull [nodes]Pull the checkout everywhere
just collective-switch all|nodes [--cached]Pull, then switch each machine; --cached activates published generations where a machine has a prebuild assignment
just cache-build-mapWhich cache node builds which machine
just herdr-init-work|personal|androidOpen a terminal workspace with a tab per machine in a group
just bootstrap-ssh-allOn every machine, connect once to every other by short name, address and full name so each records the others' host keys
just distribute-pi-authCopy this machine's coding-agent (Pi) credentials to the remote machines