runners-vm

Hetzner machines that exist for as long as Forgejo has CI work for them.

The Kubernetes runner in ../kubenix-cluster/services/forgejo-runner-k8s.nix caps a job at 2 vCPU and 2 GiB, because the three cluster workers have 4 vCPU and 8 GiB each and a runaway Rust build once took Cilium with it. This answers a second label, nixos-big, on a machine with eight dedicated cores and a Nix store that survives the machine.

Nothing here is a Kubernetes node. The machine registers itself with Forgejo from its own user-data and runs job steps on the host.

The pieces

machines/burst/	the NixOS that boots from a Hetzner snapshot
scaler/	watches the Forgejo queue, creates and destroys machines
nix/	flake-parts modules: image, packaging, devshell

A push to a repository whose workflow says runs-on: nixos-big leaves a job waiting. The scaler sees it, creates a server from the snapshot with the slot's volume attached, and the machine's runner picks the job up. When the queue empties, the scaler holds the machine to the end of the Hetzner billing hour it has already paid for, then shuts it down and deletes it.

/nix is the volume

The root disk carries an ordinarily bootable NixOS, store and all. Stage 1 finds the attached volume, mounts it over /sysroot/nix, and seeds it from the root disk when it is blank. So the copy on the root disk is both the seed and the fallback: a machine whose volume is empty still boots, it just pays for one cold build.

A volume warm for a different image is reformatted rather than merged into. Copying a new system into an existing store would leave its paths absent from that store's database, and nix treats a path that is present but unregistered as one to substitute over, which would rewrite the running system from underneath itself. The cost is one cold build per image rebuild.

Measured: the system closure is 1.6 GiB, the compressed image 541 MB, and one nix flake check of the qr workspace pulls 5.3 GiB across 1339 store paths. 50 GB of volume is €2.86 a month and grows.

Setting it up

This repository needs its own RUNNERS_PACKAGE_UPLOAD_KEY Actions secret before CI can publish the scaler image. Set it with the plaintext, through the API or the web UI. It cannot be copied from infra/runners as a database row: Forgejo binds each secret to its own row, and a copy that fails to decrypt panics task fetching for the whole instance. See ../kubenix-cluster/incidents/2026-10-12-copied-actions-secret-panics-task-fetch.md.

Then, one time, in order:

sops secrets/hetzner.yaml          # a single hcloud_token key
just upload-image                  # build, compress, snapshot
just volume-create 0 50            # 50 GB, hel1, grows only

Then register the slot's runner identity against Forgejo and put it in Vault, the same way ../kubenix-cluster/justfile does for the Kubernetes runner:

forgejo forgejo-cli actions generate-secret
forgejo forgejo-cli actions register --name ci-burst-0 --labels nixos-big --secret-file -
vault kv put secret/forgejo/runner-burst-0 uuid=... token=...

Registration is idempotent on the first 16 characters of the secret, so a replaced machine re-attaches to its own row rather than leaving an offline one behind.

Running one by hand

Before wiring the scaler up, the machine is worth proving on its own:

just burst-up 0        # create it
just burst-down 0      # shut down cleanly, then delete

Never skip the shutdown. The store on the volume is a sqlite database, and cutting power mid-write is what leaves it needing nix-store --verify --repair.

Known costs

• A Hetzner volume attaches to one server and does not cross locations, so each slot is pinned to a location and cannot fail over when ccx33 is sold out there.
• Job steps run on the host rather than in a container. The machine is single-tenant and about to be destroyed, which is stronger isolation than the privileged container on a shared worker that the nixos label still uses, but the warm store does persist between jobs from different repositories.
• A warm store saves fetching, not compiling. rustPlatform.buildRustPackage compiles dependencies inside the same derivation as the workspace, so a source change still rebuilds every crate. Splitting that out is what turns this into skipped rustc runs.