In use
Operations
Running thirty machines is mostly about knowing what is true without disturbing it: which secret a service can read, which machine has gone quiet, which binary is really on the path. These are the tools and rules for that.
Secrets
| Command | Does |
|---|---|
cltv secret edit | Edit the encrypted secrets file with SOPS |
cltv secret reinstall | Re-run sops-nix activation to install new or changed secrets |
cltv secret get | Print one secret, for the rare case a human needs it |
On Android there is no activation daemon, so a supervised watcher checks every minute whether secrets need installing.
Diagnosing without reading
When a service reports a missing secret, the rule is to prove what is wrong without ever printing a value. Start from the exact path the failing service is configured with, in the right tree: system services read /run/secrets, user services read ~/.config/sops-nix/secrets, and the health of one says nothing about the other.
secret=/run/secrets/keys/example/tokentest -r "$secret" && stat -c 'mode=%a owner=%U' "$secret" && test "$(wc -c < "$secret")" -gt 0
- Reveal path, metadata, readability and non-emptiness only: never
cat,headorset -x. - On macOS the secrets directory may deliberately be unlistable. An empty
lsproves nothing; probe the exact file. - Readable in your shell does not mean readable by a launchd daemon or another user. Compare the service's identity with the file's owner and mode.
Fleet liveness
Every managed machine reports that it is alive every five minutes to a receiver run by the agent orchestrator, which keeps a private last-seen record and decides whom to notify. There is no automatic remediation.
| Platform | Reporter |
|---|---|
| NixOS, including WSL | A user oneshot service on a five-minute timer, bounded to 30 seconds |
| macOS | A launchd agent, at load and every 300 seconds |
| Android | A supervisord program sleeping five minutes between bounded attempts, with no offline queue |
| Windows host of a devbox | A SYSTEM scheduled task running a resident PowerShell loop, independent of any login, WSL, Nix or tailnet client |
Reports carry only a hostname and a layer, over verified TLS, refusing redirects, with a token read from SOPS at run time. The Windows reporter is the interesting one. When WSL is wedged, the machine still says whether its host is up, because the reporter uses Windows' own HTTPS stack. Its files are staged from WSL during an ordinary switch, and the operator approves one UAC prompt to install the task.
What is really on the path
Home Manager owns installed binaries, but a cargo install or a copied binary in ~/.local/bin can shadow them and survive every switch. sinbin takes such copies out of the path across machines, after checking that doing so is not a downgrade, and can put them back:
sinbin APP --dry-run # show what would move, everywhere sinbin APP # all standard and Android machinessinbin APP --nodes local # only this machine, no SSH sinbin --undo APP
Copies go to ~/.local/sinbin and ~/.cargo/sinbin, never deleted.
Safe quick commits
Agents and people commit constantly, from many machines, to the same branches. Two helpers make the common cases safe:
| Command | Does |
|---|---|
gic [message] | Commit tracked changes and staged files locally, generating a message from the change if none is given; works offline |
gii message | Commit, sync with the origin branch, then push |
gic --amend | Replace a local commit, refusing if any remote branch or tag already contains it |
Diverged histories are merged, not rebased. That may add a merge commit, but a plain rebase can drop changes that exist only in a merge and then push the deletion silently.
Maintenance
| Command | Does |
|---|---|
cltv nix gc -t 14d | Collect garbage older than a period, after confirmation |
cltv nix repair store | Verify and repair the Nix store |
cltv nix gc reap-zombies-nod | Reap orphaned supervisord children that block garbage collection on Android |
cltv nix specialise -s NAME, cltv nix revert | Switch to a specialisation, and back without new changes |
cltv kv get|set|rm | A small local key-value store for scripts' persistent settings |
cltv debug | Turn on verbose logging for every cltv command, persisted in that store |
Runbooks
Incidents end with a runbook in the repository's docs/runbooks/: what failed, the evidence, what changed and how to validate it. The 35 there cover adding machines and caches, cached switches, six prebuild failures, login recovery, secrets diagnostics, liveness, Tailscale and Hyprland on Android, GPU toolchains, trackpads and displays. The design described on this site is largely their outcome, which is why so many sections here explain a rule by the failure behind it.