System
CI runners
Collective's own repositories build on self-hosted GitHub Actions runners that Collective declares. The devboxes and the Mac each run a pool of ephemeral runners for the owner's repositories, and a handful of hard-won mechanisms keep the pool from quietly disappearing.
The default pool
A machine opts in with one line, and gets an ephemeral runner for each repository in the default pool, 17 at this revision: Collective itself and its first-party projects.
git.hub.actions.enableDefaultRunners = true;On one devbox that evaluates to 27 runner services, all ephemeral. Every runner carries self-hosted, the OS and architecture, nix, the platform and the machine's hostname, so a job can ask for any Nix builder or pin itself to one machine. Runners labelled kvm also get /dev/kvm for VM tests.
runs-on: [self-hosted, nix, nixos, x86_64-linux] Project sites, this one included, are published by a shared reusable workflow that runs on these runners and pushes static files to gh-pages, so publishing spends no hosted minutes and no artifact storage.
Ephemeral by default
Each runner registers, takes exactly one job and exits; its supervisor starts it again. Persistent runners can still be declared per runner, but the pool-wide default is ephemeral.
The registration gate
Restarting dozens of ephemeral runners at once would race for the GitHub API and can exhaust the account's rate limit. Every registration on a host passes one gate:
- A durable first-in first-out queue issues one permit every
registrationMinIntervalSec(60 seconds), consumed before GitHub is called, so failed token exchanges are paced too and no repository's runners starve another's. - A runner that has to wait keeps its place for up to
registrationMaxWaitSec(30 minutes) instead of exiting and retrying. On macOS, a job that exits too quickly is put in launchd's penalty box and refused spawns entirely. On systemd, a large pool of fast-exiting losers livelocks, oscillating instead of draining. - Persistent runners skip the gate.
Failures that look healthy
The two worst runner failures leave every process apparently fine. A health timer reads both the service journal and the runner's own diagnostic log, since some errors only reach the latter.
| Symptom | Cause | Handling |
|---|---|---|
| The listener runs and logs "Connected to GitHub", but no jobs arrive | GitHub rejects its session: a name collision or invalidated credentials | The health unit fails, visible in systemctl --failed within six minutes |
The whole pool is gone after a rebuild; units show Result=success, zero restarts | A switch stops changed units and starts them at the end; an interrupted switch never does. Restart= only covers units that exit on their own | The timer starts any stopped runner unit again |
On macOS: last exit code = 78: EX_CONFIG, "service is in penalty box" | launchd throttled a runner that kept exiting faster than its minimum runtime | A minimum-lifetime guard, and recovery by bootout and bootstrap from the plist. kickstart would wait forever for a spawn launchd has decided not to perform. A running listener is never touched |
On macOS: actions/checkout hangs on git version | Git's locale lookup blocked inside the system preferences service | A bounded Git preflight before the runner starts |
Every one of these helpers is a shell or Python file that the module includes verbatim and a test runs directly, so the deployed behaviour and the tested behaviour cannot drift. On macOS, the runner logs that tell these failures apart are preserved across restarts, because an outage investigation once found every one of them already gone.
Isolation
A flake check evaluates a real devbox's runner services and asserts their sandbox, so a refactor cannot loosen it silently:
ProtectSystem=strictand a private/tmp;- the access token and the runner's current registration token are
InaccessiblePaths, out of reach of the jobs it runs; - each runner has its own state and a unique work directory, bind-mounted, so an ephemeral runner's start-up cleanup cannot delete a sibling's active job;
- work directories are disk-backed, not tmpfs, so large builds do not exhaust memory;
/dev/kvmonly for runners labelledkvm.
Windows runners from WSL
A runner marked wslTunnel on a WSL devbox is not a Linux service at all. It is installed natively on the Windows host underneath, from inside WSL:
Mint a registration token
From the stored access token, as the WSL user who can drive Windows interop. The system runner user cannot.
Hand over through a private file
The token goes into a temporary file that PowerShell reads through the
\\wsl$share; the file is deleted on exit.Install idempotently
Download the runner only if absent and register only if there is no local state.
--replaceis used only for a stale record of the same name on GitHub.Start without a console
A windowless launcher in the user's Startup folder keeps the runner and its children off the desktop, and no service identity change is needed.
Windows binaries are referenced by absolute path, since NixOS-WSL does not put them on PATH for services.