adm — Arch Declarative Markdown

There are two questions you should be able to answer about any machine you run: how do I rebuild it, and why is it configured this way? On an Arch Linux box that has been alive for a year, both answers are usually "I don't remember." State accumulates imperatively, one pacman -S at a time, and the reasoning behind it was never written down anywhere.

The answer, always, is to put something declarative over a system that is not one. The two established ways differ in how far they go.

Ansible wraps. The box stays imperative underneath; a playbook describes the state you want and converges the machine toward it, run after run. That answers the first question and not the second, because a playbook records what to converge to and never why you chose it.

NixOS replaces. There is no imperative path left to drift down — the build sandbox makes the configuration the only channel through which the system can change, so one file answers both questions. That enforcement is exactly what adm does not have, and if you can use Nix it is worth more than anything here. It was considered and rejected for this box; DECISIONS.md #001 records the reasoning and what would reverse it.

adm wraps. Arch stays imperative underneath and nothing stops you reaching past it. The bet is that the wrapper can be prose: a box whose setup is contained entirely within a set of runbooks is effectively declarative anyway — the runbooks stop being a record of what happened and become the definition of what the box is. Replay them and the machine comes back.

A runbook is ordinary Markdown — this is a literate administration repo, and runbooks/ is all of it. The prose explains why; the commands sit beneath it in fenced blocks, and those are the ones that actually run:

`btop` replaces top or htop, with no runtime dependencies beyond libuv and
ncurses. The agent harness needs to watch system resources during long runs,
and `btop` makes that obvious rather than parsing text output.

```bash id=btop
sudo pacman -S --needed --noconfirm btop
```

```bash id=btop verify
pacman -Qi btop >/dev/null || { echo "btop not installed"; exit 1; }
command -v btop >/dev/null || { echo "btop not on PATH"; exit 1; }
```

That is 065 — the run block verbatim, the verify block with one of its three assertions dropped for length.

The explanation and the commands are one artifact, so they cannot drift apart. And every change carries a verify block asserting the property it was supposed to establish. That is the part that matters: the documentation can fail. adm verify runs every assertion in the repo, so a document that has quietly become fiction announces itself instead of waiting to be discovered.

That falsifiability is also what makes the box reasonable to hand to an agent. You describe an outcome; the agent reads the conventions, writes the runbook, shows you its plan, and stops. One approval, then it runs. It is not permitted to configure the machine directly — the only channel it has is the one that documents itself, so the write-up is a side effect of the work rather than a chore after it. See 070 for the harness and the approval gate.

The honest limit: containment is a convention here, not an invariant. Nothing errors when you break it — the repo just stops being a rebuild procedure and becomes a diary, which is why it has to be held deliberately; see AGENTS.md. adm facts is the counterweight, snapshotting packages, enabled units, listening sockets and /etc so that a git diff surfaces drift nobody wrote down.

The top-level claim is falsifiable too: replay onto a bare VM, finish with a clean adm verify. That test is owed, not passed — DECISIONS #001 records it as this project's acceptance test. Until it passes, this is a premise with good evidence behind it.

The project and the box

This repo is not scaffolding around some other purpose; it is the purpose. The runbooks and adm are the system under development, and rea-dev is the first box they describe — both the workbench and the first subject. The acceptance test above is owed by the thing being built, to itself.

The box name is a nod to REA — "Reverse Engineer Anything" — which is intended to run there later inside an agent harness. That is not set up yet, and no runbook describes it. The project took its own name afterwards, which is why the box and the repo no longer match — rea-dev is a machine, adm is what describes it.

The box

Usage

adm list                  # every runbook and the blocks it will run
adm run 000-bootstrap.md  # run that runbook's blocks, verifying as it goes
adm verify                # run every verify block in every runbook
adm replay                # run every runbook in order — i.e. rebuild
adm facts                 # snapshot system state into facts/, commit if changed
adm log                   # the run journal

adm run and adm replay stop at the first failure. adm verify runs everything and reports all failures, so it is safe to use as a health check.

Add -n / --dry-run to see what would execute without running it. To work around a block that is failing for a reason you already understand — a pending reboot, say — use adm run <runbook> --only <block-id> (repeatable) to run a subset, or -k / --keep-going to continue past failures.

Which flags each of those three takes, exactly:

adm run     -n --only -k --no-snapshot
adm replay  -n -k --no-snapshot
adm verify  -n

That is a fence rather than a sentence because 000 checks it against the parser. The prose version said -k could be added "to any of them" while adm replay -k was an error, and nothing noticed: a claim about a program, written beside the program, in a form the program cannot read.

Layout

The repo splits in two: the generic half is the product, and each box gets a namespace directory for its own configuration and state. One repo can describe several machines; adm picks the box directory automatically when there is only one, and by $ADM_BOX or hostname when there are more.

Getting started

Two different jobs, with different prerequisites. Read the one you are doing.

Both share one non-obvious step. adm finds a box by looking for <dir>/<dir>.env, picking it automatically when there is only one and otherwise matching $ADM_BOX or the hostname. A freshly provisioned box still carries the image's hostname — arch, typically — so once this repo describes more than one machine, selection by hostname fails until 000 sets the real name. Until then, pass it:

ADM_BOX=<box> adm replay

A — a new machine from barebones Arch

For standing up a box this repo does not yet describe.

Prerequisites: an Arch VM you can reach over SSH as root or as a sudo-capable user. Prefer a btrfs root: 040 requires it and refuses to run otherwise, and Arch's own arch-boxes images are btrfs already. If the box was provisioned by cloud-init, let it finish before touching anything — it is what installs root's authorized_keys, which 010 copies to the admin user, and 030 explains why it must not be disabled before then.

1. Get the repo onto the box at /opt/admin. On a brand new machine use rsync from a workstation that already has a clone — cloning on the box itself needs a deploy key that does not exist yet:

   rsync -a --exclude '.git' ./ root@<newbox>:/opt/admin/
   

2. Adopt the box.

   adm adopt <box>
   adm adopt <box> --from workstation   # and opt into a purpose
   

   A profile is an env preset, not a different set of runbooks: it carries the opt-in list for a purpose and records ADOPTED_FROM. templates/ holds them as <name>.preset, and a preset may contain only keys the encryption policy exempts — so it can turn 070 on but cannot carry its credentials.

   That creates hosts/<box>/, writes hosts/<box>/<box>.env with the keys that have to be readable in the clear, appends the anchored negation to hosts/.gitignore, and stages both. It then prints the keys the later runbooks will require, and which runbook requires each.

   It does not restate that key list anywhere: it reads the runbooks for the assertions they already fail on. This paragraph used to carry the list by hand and was wrong for several hours after 010 began requiring a third key, which is why the command exists and why 000's adopt-roundtrip now fails if the two ever disagree again.

   The keys it writes are readable in the clear by necessity: 010 creates the admin account before 015 generates this box's age key, so at that moment nothing on the machine could decrypt a value. .sops.yaml exempts them for that reason.

   what makes the box yours; a git add -f is not a substitute.

3. Bootstrap and create the admin user, as root or with sudo:

   ADM_BOX=<box> python3 /opt/admin/bin/adm run 000 010
   

   000 brings the system current, sets the hostname from BOX_NAME and puts adm on PATH; 010 creates the admin account, grants passwordless sudo and hands it ownership of the repo. After this, connect as ssh admin@<box> and stop using root.

4. Reboot if 000 says the kernel changed, then carry on. It tells you explicitly rather than rebooting for you.

5. Set up secrets — 015 generates this box's age identity, adds it to .sops.yaml, and arms the git hooks:

   adm run 015
   

   The box key it generates is disposable — it only ever reads this box's own env file, and a rebuild can replace it. What must already exist is an operator recipient listed in operator.pub, whose private half lives on your workstation at ~/.config/sops/age/keys.txt and in your password manager. That is the one key whose loss is unrecoverable; see 015 for the two tiers.

6. Add any secret values this box needs to its env file, then encrypt:

   sops hosts/<box>/<box>.env          # edit decrypted, saves encrypted
   adm run 015 --only encrypt-env
   

7. Give it a remote — 020 generates a deploy key and prints it. Add it to the forge with write access, then adm run 020 finishes and pushes.

8. Finish the rest, then check:

   adm replay && adm verify && adm facts
   

A second box is also the first real test of whether the generic/per-box split holds. Expect to find things that are still accidentally specific to rea-dev; that is useful information, and fixing it belongs in the runbooks rather than in the new box's env file.

B — reproducing a machine this repo already documents

For rebuilding a box after loss, or recreating it to prove the documentation is true. Doing this successfully is this project's acceptance test — see DECISIONS.md #001, where it is recorded as owed.

Create the replacement VM to match. Nothing inside the guest can check these until it is too late to change them cheaply, and hosts/<box>/facts/ in this repo records what the original had:

That last one is the real hazard: it is the only failure here that locks you out rather than failing loudly.

Prerequisites — get these in hand first. Both are discoverable halfway through a rebuild, which is the worst possible moment:

• The operator age key, on the workstation at ~/.config/sops/age/keys.txt. A rebuilt box generates a new identity which is not yet a recipient of its own encrypted env file, so it cannot decrypt its own configuration until someone re-keys the file — and only an operator can, because only an operator can read it. This is the prerequisite that makes a rebuild possible at all; without it the configuration is permanently unreadable, which is what encryption means. The old box key is not needed. That is the point of the two-tier design — box keys are disposable, see 015.
• Forge access for a deploy key. The rebuilt box generates a new one, so be ready to paste its public key into the forge with write access.

0. Do not delete the original yet. Shut it down and build the replacement alongside it. A stopped VM costs nothing, and it turns an irreversible test into a reversible one — if the rebuild stalls, start the original and you have lost an afternoon rather than a machine. Delete it only once the replacement passes step 7.

1. Provision barebones Arch to the requirements above, matching the original's filesystem — hosts/<box>/facts/storage.txt records what it was.

2. Get the repo onto the box at /opt/admin, by rsync from a workstation clone. Cloning on the box needs a deploy key that does not exist yet, so rsync first and let 020 sort out the remote later.

3. Bootstrap and create the admin user. Stop here — do not replay further yet, because the next step needs the admin user's home to exist:

   ADM_BOX=<box> python3 /opt/admin/bin/adm run 000 010
   

4. (Optional shortcut.) If you kept the old box key, restore it before 015 runs and the rebuild proceeds without the operator step below:

   install -d -m 0700 ~/.config/sops/age
   install -m 0600 /path/to/restored/keys.txt ~/.config/sops/age/keys.txt
   

   015's age-key block only generates a key when none exists, so a restored one is left alone. This is a convenience, not a requirement.

5. Replay, and expect it to stop twice. Both stops are the runbooks reporting something untrue, not defects:

   adm replay
   

   • At 015's sync-env-recipients, because the rebuilt box's new key is not yet a recipient of its own env file. The block prints the fix: from the operator workstation, in a clone of this repo, git pull then sops updatekeys -y hosts/<box>/<box>.env, commit and push. Then git pull on the box and re-run. Do not copy the operator key onto the box.
   • At 020's forge-authorized, until the new deploy key is authorised on the forge with write access.

6. Reboot when 000 reports the kernel changed, then adm replay again. Every block is idempotent, so this is safe and expected.

7. The actual test:

   adm verify
   

   It must come back completely clean. If it does not, the documentation is wrong, and that is a bug in this repo — fix the runbook, not the box.

8. Compare against the recorded state. adm facts writes a fresh snapshot over the committed one, so git diff hosts/<box>/facts/ is a direct comparison between the rebuilt machine and the one this repo described. Expect noise only in genuinely variable things; anything structural is a reproduction gap worth a runbook change.

Honest limitations

• Runbooks only know what they were told. A change made by hand at the shell is invisible to them. adm facts is the counterweight: it snapshots package lists, enabled units, listening sockets, users, /etc file manifests and pacman-modified-file checks, so a git diff in hosts/<box>/facts/ reveals drift even when nobody documented it. Run it after any session.
• Idempotence is the author's job. Nothing enforces it. Use pacman -S --needed, install -m, systemctl enable --now, ln -sfn, mkdir -p, and guard edits so re-running is a no-op.
• Encrypted, but not a secret manager. Secrets do live in here, encrypted with SOPS and age — see 015. Values are ciphertext; keys stay deliberately readable, so the shape of the configuration is public even when its contents are not. Nothing rotates, expires or audits access. Credentials still do not belong in runbook prose — reference where they come from instead. And hooks/pre-commit is the only thing between a plaintext value and the remote: it scans staged content, so commit --no-verify walks straight past it, and anything once committed in the clear stays in history.
• Backup is not this repo's job. It happens at the hypervisor layer, which nothing inside the guest can verify — so no verify block covers it, and none should pretend to. The btrfs snapshots set up by 040 are convenience only. See DECISIONS.md #003.

Prior art and lineage

Read this before assuming the conventions here are ones you already know. The ideas are established; the syntax is local to this repo.

What is standard:

• "Runbook" is long-standing operations vocabulary — data-centre run books, ITIL operational procedures, SRE playbooks, incident runbooks. "Runbook automation" is a product category.
• Literate programming is Knuth, 1984 (noweb, cweb).
• Literate DevOps — applying it to systems administration — was popularised by Howard Abrams around 2014–16 using Emacs org-babel to execute shell blocks against hosts. That is the closest named ancestor of this repo. Note what that approach does differently: it runs from the workstation over TRAMP, so nothing is installed on the target. We traded that away so the box can replay its own runbooks unaided.
• Documentation that fails is Python's doctest, Rust doc tests, Go examples. The verify block is doctest-shaped.
• DECISIONS.md follows MADR, the Markdown form of Nygard's 2011 Architecture Decision Records.
• Executable Markdown is a real tool category, with several maintained implementations: Runme (CLI and VS Code, the most mature), runmd, mdrb, runbookmd.

What is not standard — and this is the important part — is the fence syntax. CommonMark specifies nothing about execution attributes on code fences, so every tool in that list invented its own convention; see the syntax comparison across projects. So id=, verify and skip are ours. No other tool reads them, and nobody arrives already knowing them. The full rule set is five lines in AGENTS.md.

Why a bespoke runner instead of Runme

adm needs only python3, which ships in Arch's base. The stock image that became this box had no git, no make, no jq — on a bare rebuild target adm replay runs with zero bootstrapping. Runme is a Go binary you would have to get onto the box before you could follow the runbook that installs things, which is a chicken-and-egg problem on exactly the machine where it matters most. Runme also has no verify-block equivalent, which is the feature doing the real work here.

One accidental virtue: the attributes degrade gracefully. Renderers key syntax highlighting off the first info-string token, so ```bash id=foo verify still renders as a normal highlighted bash block on GitHub and elsewhere. These files stay readable as plain documentation with no tooling present at all.

Provisioning the machine

Everything else in this repository runs on a box. This runs on the hypervisor, before the box exists, and adm never touches it — there is no runbook here because there is nothing for adm to verify yet.

Written for Proxmox, because that is what this was built on. The requirements below are what the runbooks check for; any hypervisor or bare metal that meets them will do.

What the runbooks require of the machine

Each of these is enforced by a check, named so you can see what refuses and where:

The Arch cloud image satisfies the first two as it ships. It is not an ext4 image that needs converting: the layout it produces is

sda1    1M          BIOS boot partition, for GRUB on GPT
sda2  300M vfat     ESP, automounted at /efi
sda3  ...G btrfs    root, with compress-force=zstd

which is the layout 040 and 050 were written against — and the layout rea-dev has, because it came from this image. DECISIONS.md #002 records discovering that: a migration was planned on the belief that root was ext4, and root was already btrfs.

The third requirement is the one you must add by hand. See below.

Create the template

The script is from the Proxmox chapter of book.rymcg.tech and builds templates for several distributions; only Arch is relevant here.

On the Proxmox host:

wget https://raw.githubusercontent.com/EnigmaCurry/blog.rymcg.tech/master/src/proxmox/proxmox_kvm.sh
chmod a+x proxmox_kvm.sh
DISTRO=arch TEMPLATE_ID=9000 STORAGE_TYPE=local STORAGE=local-lvm ./proxmox_kvm.sh template

It downloads Arch-Linux-x86_64-cloudimg.qcow2 from mirror.pkgbuild.com, imports it, sets rootflags=compress-force=zstd and a serial console in GRUB, installs qemu-guest-agent, re-initialises the pacman keyring, upgrades, and converts the VM to a template.

Read the variables at the top of the script before running it: STORAGE_TYPE and STORAGE for a different backend, SSH_KEYS for the public key cloud-init will authorise, FILESYSTEM_SIZE, NUM_CORES, MEMORY. On NFS storage chattr +i fails during template creation and that can be ignored.

Set the CPU type, which the script does not

The script configures fourteen qm flags and --cpu is not among them, so the template inherits Proxmox's default — kvm64, which is roughly x86-64-v1: no AVX2, no AVX, not even SSE4.2 or AES-NI.

qm set 9000 --cpu host

Set it on the template so clones inherit it. Without it, 000#cpu-capability stops the run before anything is installed, which is the check working — 070's agent harness is a Bun binary that needs AVX2, and on a CPU without it the failure is not an error but a process spinning at 100% for ever. This box shipped as kvm64 and 070 was blocked until the VM's CPU type was changed.

host passes the physical CPU's features through, which also means a migration to a host with an older CPU can take features away. On a cluster, set a named model that every node supports instead.

A template ages

The script upgrades the image while building the template, so a template is current on the day it is made and not afterwards. A clone of a template built months ago is a stale Arch install with current package databases, and that combination is the one Arch handles worst: installing a single package against synced databases is a partial upgrade, which is unsupported and fails.

It is not hypothetical. The first machine built from these instructions was cloned from a nine-month-old template, and the first command in the creation directive — installing git — aborted on conflicting files, because today's git depends on libgcc and libstdc++, which have since been split out of gcc-libs, and the old gcc-libs still owned those files. Nothing was damaged; pacman refused, which is what it is for.

So either rebuild the template before cloning from it:

qm destroy 9000
DISTRO=arch TEMPLATE_ID=9000 STORAGE_TYPE=local STORAGE=local-lvm ./proxmox_kvm.sh template
qm set 9000 --cpu host

or expect the first step on the clone to be a full system upgrade, which is what the creation directive now does: archlinux-keyring first, then pacman -Syu with the packages, in one transaction. On an image that has sat for a long time that can be a hundred or more packages including the kernel, and a reboot before continuing.

Rebuilding is cheaper and gives a clone that boots ready. Upgrading is the only option for a template you cannot rebuild.

Clone a machine from it

TEMPLATE_ID=9000 VM_ID=100 VM_HOSTNAME=my-box ./proxmox_kvm.sh clone
qm start 100
VM_ID=100 ./proxmox_kvm.sh get_ip

Then continue in 000-bootstrap.md, or follow the creation directive at the top of the published document, which is the same sequence written for someone who has only that page.

Cloning for the cold-boot test

The point of a disposable clone is a box that has never been bootstrapped. Several checks in this repository can only fail there: a verify block that asserts state a later block creates passes forever on a box that was set up once, because the state is there from last time. Seven defects of that shape were found by doing this by hand, and nothing in the repo can catch the next one.

So a clone made for testing should be destroyed afterwards rather than kept and re-run — a second run on the same box is a poor test of the first.

Conventions for this repo

Read README.md first for what this repo is and how adm works.

This file is the contract for adding to it. It applies to Claude and to humans equally.

The rule

Nothing changes on this box except through a runbook. If you are about to run a command that alters the system, it goes in a runbook first, and you run it with adm run. Reaching for a bare shell to "just quickly" install or edit something is how the documentation becomes fiction.

Read-only investigation at the shell is fine and encouraged. Mutation is not.

This is not bureaucracy — it is the single load-bearing assumption of the whole repo. Arch is an imperative system, but setup contained entirely within the runbooks is effectively declarative: the runbooks become the definition of the box, and the box becomes a replayable consequence of them. That is the same property NixOS provides, obtained by discipline instead of by a build sandbox, and it is why this repo stays plain enough to read as prose. See DECISIONS.md #001.

The difference is that Nix enforces the single channel and this repo only assumes it. Break the assumption and nothing errors — the repo quietly stops being a rebuild procedure and becomes a diary. adm facts is the only detector, which is why it runs at the end of every session.

Writing a runbook

Name it runbooks/NNN-topic.md, where NNN leaves room to insert later: step by 10 within a theme, by 100 between themes. Filename order is execution order, so a runbook may depend on anything numbered below it and must not depend on anything above.

Structure:

1. # NNN — Title
2. A short paragraph on why this exists — the decision, the constraint, the thing that will be non-obvious in six months. This is the part that no other tool (etckeeper, Ansible) preserves. Write it for a reader who has forgotten everything.
3. Then, per change: prose explaining it, the bash block that does it, and a verify block that proves it.

Block syntax

This syntax is local to this repo — not a standard, and not read by any other tool. Do not assume familiarity from elsewhere; see README.md for what here is established practice and what is invented.

A bash block is executed if and only if it has an id= attribute:

```bash id=install-nginx
pacman -S --needed --noconfirm nginx
```

```bash id=install-nginx verify
pacman -Qi nginx >/dev/null
systemctl is-enabled nginx
```

• id=<slug> — makes the block executable; unique within its runbook.
• verify — assertion block. Runs immediately after the block with the same id, and again on every adm verify. Non-zero exit = failure.
• skip — parsed, logged, not executed. For blocks retired but kept for the historical record.

One block is not bash at all. Every runbook declares, directly under its purpose line, whether every box runs it:

```adm
required
```

required runs everywhere. optional runs only where the host env lists its number in OPTIONAL_RUNBOOKS. A runbook declaring neither is unclassified and adm will not run or verify it — not "required by default", because a new runbook silently running everywhere and one silently running nowhere are both wrong, and only one of them is visible. 000's runbook-classification fails on an unclassified runbook, and on an opt-in naming something that does not exist or is not optional.

An optional runbook nobody enabled is silent during a run — no warning, no skip notice — and still listed by adm list, with how to enable it. Silence is for things deliberately not in use; a misspelled opt-in is a different matter and fails loudly.

A required runbook must never depend on an optional one. The optional one may not have run, and the bug would appear only on boxes that did not opt in. Nothing enforces this: required runbooks already link to optional ones nine times, every one a cross reference rather than a dependency, so a scan would report nine false positives and teach everyone to ignore it. Telling "refers to" from "relies on" needs a person.

A bash block with no id= is illustrative and never runs — use that for "here is what the broken output looked like". Blocks in any other language (ini, toml, text, …) are never executed, so config file contents can be shown inline for readability and then written by a bash block.

Every block runs under bash with set -euo pipefail, with the repo root as cwd, and with $ADM_ROOT, $ADM_RUNBOOK, $ADM_ID in the environment. Use $ADM_ROOT rather than hardcoding /opt/admin.

Every key in <box>.env is also exported — $BOX_NAME, $ADMIN_USER, $GIT_REMOTE. Runbooks must be generic: anything true of one machine rather than of the procedure belongs in <box>.env, and anything that is history rather than procedure belongs in AS-FOUND.md. A runbook that accumulates a specific box's story stops being runnable on the next box, which is most of its value gone.

Two parts of that are checked, by 080's generic-runbooks. A box's name may not appear in an executable block — that is code, and a name in it makes the procedure wrong on the next machine; use $BOX_NAME, which adm substitutes before the block runs. Nor in a link target: a link to hosts/<box>/AS-FOUND.md resolves on one machine and is dead in every fork, so per-box files are referred to by name without a link.

Prose and illustrative blocks may name a box, and should where a concrete example reads better. Substitution reaches executable blocks only, so a placeholder written into an example reaches the published document as a placeholder: sops <box>/<box>.env is something nobody can paste. The generic source is what the export's templates carry; the rendered page is allowed to be about the machine that produced it.

Run adm list after editing to confirm the runner sees what you intended.

Idempotence is mandatory

adm replay must be safe to run on an already-configured box, because that is what makes this repo a rebuild procedure rather than a diary. Concretely:

• pacman -S --needed --noconfirm, never bare -S
• mkdir -p, ln -sfn, install -D -m 0644
• systemctl enable --now, which is already idempotent
• useradd guarded: id -u svc >/dev/null 2>&1 || useradd -r svc
• For config files, write the whole file from a heredoc rather than appending or sed-patching in place. Append-based blocks duplicate their own output on the second run.
• Never pacman -Sy without -u. It is how this box got into a partial-upgrade state in the first place.

Verify blocks should assert, not narrate

Check the property you actually care about, and exit non-zero when it does not hold. systemctl is-active foo is a good verify; systemctl status foo is not, because it succeeds regardless. Prefer checking observable behaviour (a port answers, a file has the expected mode) over checking that you ran the command (a package is installed).

Verify blocks must be read-only and fast. They run on every adm verify, including when someone is debugging a broken box.

A verify block must be able to see what it asserts. Runbooks run as admin, so an unprivileged check against a root-owned path can fail for the wrong reason — or, far worse, succeed for the wrong reason. test ! -e /root/.ssh/config passes whenever /root is unreadable, reporting absence when it has merely been denied a look. Absence assertions in particular must acquire privilege (sudo test -e ...) and should confirm the containing directory is readable first, so "not there" is distinguishable from "not visible". This exact bug shipped here and survived until a replay as admin caught it; running the runbooks as root hides the entire class.

A verify block must fail only for the reason it names. Distinguish "the property does not hold" from "I could not observe it" — these are different findings, and only one of them is a bug in the box. Empty output is not a negative result; that is the rule above, applied past the filesystem. A probe that shells out to something which starts a server, waits on a timer or touches the network will sometimes come back with nothing, and that is a retry, not a verdict. Nor should you discard stderr from the command whose failure you are diagnosing: 2>/dev/null is how a failing block produces a log holding one line of conclusion and no cause. This shipped here as well — agent-gate reported the agent's permission gate missing, twice, because opencode debug agents returns nothing while another opencode is mid-start, which during adm verify is the normal path rather than an edge case. The determinism demanded of fact collectors below applies here too, and for a sharper reason: a noisy collector produces a diff you learn to ignore, while a flaky verify raises a false alarm about the one property you built it to watch.

Let a service settle before asserting it runs. systemctl is-active immediately after systemctl enable --now can return active for a unit that has already exited, because systemd has not reaped it yet. This shipped here: grub-btrfsd died after 11 ms from a missing optional dependency and the run block reported success. Pause, then re-check, and assert the main PID is actually alive — not merely that the unit was once started.

Fact collectors must be deterministic

hosts/<box>/facts/ earns its keep only if a diff means something changed. A collector whose output varies while the system does not produces fake diffs, and a few reboots' worth of those teaches you to skim past the one file you should be reading — which is strictly worse than not collecting it.

So: run a collector twice in a row and require byte-identical output. Known offenders, all of which bit this repo already:

• PIDs and FDs — ss -tulpn renumbers every boot. Stripped with sed, keeping the process name, because a new or differently-owned listener is real drift and must still appear.
• Timings — systemd-analyze reported 38.7s and 5.9s for two identical boots. Removed outright.
• Idle-timer automounts — /efi is an automount with TimeoutIdleSec=120, so its presence in df depends on snapshot timing. Excluded from the live mount view, with the target list recorded separately since that list is stable configuration.
• Unsorted output — sort anything whose order is not guaranteed.

When you must drop something transient, record the stable part of it rather than dropping the subject entirely. And prefer normalising to excluding: an omitted collector hides real drift just as effectively as a noisy one buries it.

Finishing a session

adm verify          # everything still true?
adm facts           # snapshot state; commits only if something changed
git -C /opt/admin status

If adm facts shows drift you did not make through a runbook, do not just commit it — find out what changed, then either write the runbook that explains it or revert it. Committing unexplained drift silently turns this repo back into an audit log.

Secrets

Do not put credentials in runbooks. Document where a secret comes from and how to install it ("the deploy key lives in <password manager>; install to /root/.ssh/id_ed25519 mode 0600"), and let the verify block check the file's mode and fingerprint rather than its contents.

Decisions

Append-only. One entry per decision that shaped this box, newest last.

A runbook records how something was done. This file records why that approach — the forks in the road, what was rejected, and the conditions under which a decision should be reopened.

Format. Entries use the field names of an Architecture Decision Record (Nygard, 2011) in its Markdown form, MADR: Status, Context and problem statement, Decision drivers, Considered options, Decision outcome, Consequences, Confirmation. Standard names are used so the structure is recognisable rather than invented here — see README.md.

One local addition: Reopen this if. MADR's Status field can be set to superseded after the fact, but nothing in the template says in advance what should trigger that. Writing the trigger down while the reasoning is fresh is the difference between a decision record and a diary entry.

001 — Arch Linux, not NixOS

• Status: accepted
• Date: 2026-10-18
• Deciders: ryan

Context and problem statement

This box is a stock Arch image (KVM guest, legacy BIOS boot, btrfs root) being adopted as a service host. The workstation administering it runs NixOS with a declarative disko disk layout, so NixOS was the obvious default and was seriously considered, including the in-place conversion paths.

The question: should the box be converted to NixOS to get declarative, reproducible configuration, or stay on Arch and obtain that property another way?

Decision drivers

• Reproducibility: a documented path to recreating the box from scratch.
• Administration assumes no hypervisor console access. Console in fact exists, but the box is deliberately managed as though it does not, so no procedure may depend on it.
• Legibility: the configuration should be readable by anyone who can read a shell command.
• Incident ergonomics on a long-lived service host.
• Scale: one host, one operator, 195 packages, 12 of them explicit.

Considered options

1. Stay on Arch, contain all setup in literate runbooks.
2. nixos-anywhere + disko — kexec into an installer, wipe, declaratively install.
3. nixos-infect — install Nix on the running system, swap the bootloader in place.
4. Side-by-side NixOS install, Arch kept bootable as fallback.
5. Provision a fresh NixOS VM and migrate the service onto it.

Decision outcome

Option 1 — stay on Arch, contain all setup in the runbooks.

The governing insight, in the operator's words:

NixOS is declarative, but it fights our runbook plainness. Arch is imperative, but if we contain setup to the runbook, it becomes effectively declarative, yet plain.

That is the whole design. Declarativeness is not a property of the OS — it is a property of the discipline of a single channel for change. Nix enforces that channel with a type system and a build sandbox, and charges real complexity for it. A runbook enforces it by convention and charges almost nothing. Convention is weaker, but it is legible.

Supporting reasons, roughly in order of weight:

1. The conversion was the risk, not the OS. An in-place conversion is a bootloader, initrd and root-filesystem transplant performed simultaneously — and this box boots legacy BIOS, so there is one MBR stage and no efibootmgr escape hatch. Steady-state comparison favours NixOS; the comparison that mattered was steady-state NixOS minus an unrecoverable transplant.
2. Scale mismatch. Nix's value scales with configuration complexity and with the number of hosts and people needing identical environments. Here: one host, one operator, a system whose entire state fits on a page.
3. Nix's guarantee doesn't cover what breaks service hosts. It makes the system closure reproducible. It does not manage service data, disk layout (without re-running disko destructively), or secrets. Outages come from the database, the certs and the full disk, not from package drift.
4. Rollback is already better here. Booting an older NixOS generation reverts the closure, not on-disk state — old code against a migrated database is arguably worse than no rollback. This box already runs btrfs, where snapshots can revert system and data together.
5. Incident ergonomics. pacman -S strace during an outage, standard FHS paths, and third-party prebuilt binaries that run without patchelf or an FHS wrapper.

Pros and cons of the other options

Consequences

Good

• No risky operation is required; the box is already running and current.
• The configuration stays readable as prose, and renders as ordinary Markdown with no tooling installed.
• adm depends only on python3 from Arch's base, so a bare rebuild target can replay the runbooks with zero bootstrapping.
• Room for the why — this decision record exists because the format has somewhere to put it. A configuration.nix does not.

Bad

• Containment is a convention, not an enforced invariant. Break it and nothing errors; the repo silently degrades from a rebuild procedure into an audit log. adm facts is the only detector.
• Idempotence is the author's responsibility, with no checker.
• The fence syntax is bespoke, so no other tool reads these runbooks.
• Two mental models to maintain: NixOS on the administering workstation, Arch here.

Confirmation

This decision holds only while all three of these are true. They are separate properties and should not be conflated — the second is cheap and says little, the third is the one that matters.

1. Containment — every change to the box flows through a runbook. Checked by adm facts reporting no drift that isn't attributable to a runbook, at the end of every session. Ongoing.
2. Idempotence — adm replay on an already-configured box is a no-op. Checked by a full replay finishing green with no duplicated state. Demonstrated 2026-10-18T06:19:04–06:19:10Z: all eight blocks of 000-bootstrap.md ran in order against a configured box, every block and verify green, no spurious commit from repo-init. See hosts/<box>/log/journal.md.
3. Reproduction — adm replay onto a fresh Arch VM builds this box, finishing with a clean adm verify. Never run. Still owed.

Check 2 proves the runbooks are re-runnable; only check 3 proves they are sufficient. Passing 2 is weak evidence for 3, because a block can silently depend on state the stock image doesn't have and still succeed here. Until 3 passes, reproducibility is a claim rather than a demonstrated property — NixOS would have made it true by construction, and we chose to owe a test instead. Pay the debt.

Reopen this if

• The box becomes several boxes. Nix wins outright at rea-dev-01..05 and none of the reasoning above survives that change.
• Bit-identical rebuilds on demand become a requirement rather than a nicety.
• A from-scratch rebuild is needed anyway and console access is available — at which point take option 5, not an in-place conversion.

002 — btrfs snapshots, not ZFS

• Status: accepted
• Date: 2026-10-18
• Deciders: ryan

Context and problem statement

We wanted snapshot capability: cheap rollback for development workflows, and a way to undo catastrophes like a mistaken rm. The initial proposal was to migrate the root filesystem to ZFS, in place, on a single disk, from what was believed to be an ext4 root — using a RAM-based SSH-accessible rescue environment to shrink the filesystem offline.

The premise was mistaken: root was already btrfs, with compress-force=zstd:3, not ext4:

/dev/sda3 btrfs rw,compress-force=zstd:3,discard=async,space_cache=v2,subvolid=5

So the requested capability was already present and unused. That reframed the question from "how do we migrate to ZFS" to "is ZFS worth migrating to, given snapshots are already available".

Decision drivers

• Snapshots with cheap rollback, and file-level recovery.
• Administration assumes no hypervisor console, so any procedure that can leave the box unbootable is expensive.
• Upgrades must keep working unattended on a rolling release.
• One disk, and no second disk assumed available.

Considered options

1. Use btrfs snapshots, already available.
2. ZFS root, single disk: shrink in place, kexec rescue environment, migrate.
3. ZFS root on a second virtual disk.
4. ZFS for data only, keeping btrfs root.

Decision outcome

Option 1. Implemented in runbooks/040-btrfs-snapshots.md: snapper with a tuned retention policy and timers, plus snap-pac for automatic pre/post snapshots around every pacman transaction.

The decisive argument is not that btrfs is better than ZFS — it is not, feature for feature. It is that ZFS is not in Arch's repositories. It ships via AUR or the third-party archzfs repo as an out-of-tree DKMS module, and OpenZFS supports new kernels on a lag. On a rolling release that makes every pacman -Syu a chance to install a kernel ZFS cannot build against — and with ZFS as root, that means an unbootable box.

The migration risk was explicitly accepted, and would have been acceptable; this is a different thing. It is a recurring risk, on every upgrade, forever, on a machine with no console. Avoiding it would require pinning to linux-lts with IgnorePkg, reintroducing precisely the staleness that 000 exists to prevent.

Two further points settled it:

• Space reclamation is a dead end. ZFS cannot expand a vdev backward into freed space. Adding the old partition as a second top-level vdev yields a stripe across two partitions on one disk: no redundancy gain, and losing either partition destroys the pool. Worse than the status quo, permanently.
• It would reverse #001. That record cites btrfs snapshots covering system and data rollback as a reason Arch was sufficient. Migrating to ZFS to obtain snapshots re-solves a solved problem, and trades an in-tree filesystem for a kernel-coupled out-of-tree one.

Pros and cons of the other options

Consequences

Good

• Zero migration risk: nothing was repartitioned, and no reboot was required.
• In-tree filesystem, official packages, no DKMS, no kernel coupling.
• Snapshots are near-free: copy-on-write, on an already-compressed filesystem.
• snap-pac makes a bad upgrade a readable diff rather than an archaeology project.

Bad

• No whole-system rollback. Root is mounted from the btrfs top level (FS_TREE, subvolid=5) rather than an @ subvolume, so the snapper rollback idiom — make a snapshot the default subvolume — has nothing to swap. File-level restore works; "roll the whole box back to 09:00" does not. Fixing it means moving root into @, which deserves its own decision.
• Snapshots are not backups. Same partition, same filesystem, same disk. No off-box target exists yet, so this is the largest remaining gap. Superseded on this point by #003: backup is handled at the hypervisor layer, so this is a division of labour rather than a gap.
• Forgoing ZFS features that btrfs genuinely lacks: zvols, per-dataset properties and quotas, ARC, and better send/recv ergonomics.

Confirmation

• adm verify 040 passes, including a precondition asserting root is btrfs.
• 040's snapshot-roundtrip block proves the capability end to end rather than asserting configuration: it writes a canary, snapshots, deletes it, restores from the snapshot, compares content, and cleans up. It runs on every replay, so the feature cannot silently rot.
• snap-pac observed producing a pre/post pair for a real pacman transaction, with the command line recorded as the description.

On converting an ext4 root to btrfs

Not needed here, and not implemented, but assessed because the repo aims to be generic. It is materially safer than ext4 → ZFS would have been: btrfs-convert works in place with no partition surgery, keeps the same PARTUUID, and has a real undo in btrfs-convert --rollback (the original metadata is preserved in an ext2_saved subvolume). And btrfs is in-tree, so the recurring kernel-coupling risk disappears entirely.

What remains genuinely hard:

• Root must be unmounted, so an SSH-reachable pre-root environment is still required. Everything needed is in official repos — mkinitcpio-netconf, mkinitcpio-dropbear, mkinitcpio-tinyssh, mkinitcpio-systemd-tool, kexec-tools — but note this box uses the systemd initramfs hook, so the common netconf + dropbear recipe (written for the busybox encrypt flow) does not apply unmodified.
• The result is untidy: chunk layout inherited from ext4, no @ subvolume layout, no retroactive compression, and ext2_saved holding space until deleted — which forfeits the rollback.

Preferred alternatives, in order: provision btrfs in the first place (Arch's own arch-boxes images are btrfs, which is where this box came from); or copy-based migration — create btrfs on free space or a second disk, rsync across, switch root — which avoids btrfs-convert altogether and yields a clean @ layout that in-place conversion cannot produce.

Any conversion procedure added later must be labelled untested until it has been exercised against a throwaway ext4 VM. We have no ext4 box here, so shipping it unlabelled would mean the first person to run it is the test.

Reopen this if

• zvols, per-dataset properties, or zfs send ergonomics become load-bearing — then take option 4, data-only, on a second disk.
• A second virtual disk is attached, which makes option 3 reasonable and removes every geometry objection above.
• Whole-system rollback becomes a requirement. That is a btrfs subvolume layout problem, not a filesystem choice problem, and does not need ZFS.

003 — Backups live at the hypervisor layer

• Status: accepted
• Date: 2026-10-18
• Deciders: ryan

Context and problem statement

#002 set up btrfs snapshots and listed the absence of off-box replication as the largest remaining gap, with btrbk or btrfs send as the implied future work. That framing was wrong: backup is already handled, one layer down.

Decision

Backups are taken at the hypervisor layer. In-guest btrfs snapshots are for convenience only — fast, fine-grained undo — and carry no durability guarantee.

Decision drivers

• Backup belongs wherever it can be verified and restored independently of the thing being backed up. A guest cannot sensibly guarantee its own recovery from guest loss.
• Avoid two half-solutions to one problem.

Consequences

Good

• The two tiers are complementary, with no overlap to maintain:

  tier	covers	does not cover
  hypervisor	whole disk, ESP, partition table, host loss, off-box	fine-grained or instant recovery; invisible from inside the guest
  btrfs snapshots	instant file-level undo, pre-pacman state, in-guest	disk failure, guest loss, anything outside the root filesystem

• The hypervisor tier covers what btrfs structurally cannot. /efi is a separate vfat partition that no btrfs snapshot can reach, and the partition table itself is outside any filesystem. Both are inside a whole-disk image.

• Two items are now explicitly off the roadmap, which is worth more than it sounds — each was a non-trivial piece of work implied by #002:

  • Off-box btrfs send. Would duplicate the hypervisor's job, less well.
  • Moving root into an @ subvolume. Its only purpose was whole-system rollback, and hypervisor rollback is strictly more thorough — it reverts the disk, not one filesystem. Restructuring the subvolume layout to obtain a weaker version of an existing capability is effort for a worse result. The FS_TREE layout therefore stays, and #002's main "bad" consequence is largely neutralised rather than outstanding.

• Retention in 040 was retuned to match the actual purpose: monthly and yearly timelines dropped to zero, weekly cut to 2. A six-month btrfs timeline would imply a durability guarantee this tier does not provide.

Bad / to keep in mind

• Hypervisor snapshots of a running VM are crash-consistent, not application-consistent. The disk is captured at an instant with no filesystem quiesce. btrfs tolerates this well — it is copy-on-write with atomic metadata commits, so a restored image looks like an unclean shutdown, which it handles. But if a database ever runs on this box, its own consistency is its own problem: it will need either a quiesce hook or its own dump, and a restored image may contain a torn transaction.
• The backup tier is not verifiable from inside the guest, so nothing in this repo can assert it exists or works. adm verify cannot cover it, and no verify block should pretend to. Restore testing lives with whoever operates the hypervisor.
• This repo therefore documents a dependency it cannot check — worth stating plainly, since every other claim here is backed by a verify block.

Confirmation

Not confirmable from the guest, by construction. The honest check is a periodic restore test at the hypervisor layer — a backup that has never been restored is a hypothesis. That check is out of scope for this repo and is recorded here so the gap in coverage is visible rather than assumed away.

Reopen this if

• The hypervisor's backup or retention policy changes materially, or restore testing reveals it does not cover what is assumed here.
• This box starts hosting something with its own consistency requirements — a database, a queue — at which point crash-consistent whole-disk images stop being sufficient on their own.
• The box is ever migrated somewhere without a backup tier underneath it, in which case off-box btrfs send returns to the table immediately.

004 — A shareable repo with secrets kept in it

• Status: accepted; its publication caveat amended 2026-10-20, see the amendment and #007
• Date: 2026-10-18
• Deciders: ryan

Context and problem statement

The purpose of this box became explicit: it is a development workstation for building an LLM-bootstrapped, self-documenting workflow that produces a declarative configuration system for Arch Linux. The repo is not scaffolding around the purpose — it is the product, and the box is both workbench and first subject. (rea-dev is named for REA, "Reverse Engineer Anything", intended to run here later in an agent harness; not set up, and no runbook describes it.)

A product is meant to be shared. That collides with the fact that configuring a real machine involves real secrets, and with the earlier design in which every runbook simply hardcoded this box's values.

Decision

Three changes, together:

1. Split generic from per-box. The product — runbooks/, bin/adm, AGENTS.md, DECISIONS.md, .sops.yaml — lives at the root. Each machine gets a namespace directory holding only its own configuration and state: hosts/<box>/<box>.env, hosts/<box>/AS-FOUND.md, hosts/<box>/facts/, hosts/<box>/log/. One repo can describe several boxes; adm selects the directory automatically when there is one, by $ADM_BOX or hostname when there are more.
2. Nothing host-specific in a runbook. Every value that is true of one machine rather than of the procedure comes from <box>.env. Host and port are derived from $GIT_REMOTE rather than stored again, so there is one source of truth.
3. Secrets in the repo, encrypted with SOPS + age. The key lives on the box, outside the repo, at the location SOPS looks by default. Superseded by #005: there are now two tiers, and the key that can read everything deliberately does not live on any managed box.

Pros and cons of the encryption options

The deciding factor was reviewable diffs, not cryptographic strength — all four are adequate there.

Consequences

Good

• Runbooks are now genuinely generic; standing up another box means adding one directory with one env file.
• Encrypted values never reach the remote in clear, and adm refuses to export a value that is still ENC[...] rather than passing ciphertext into a block as configuration.
• 015 re-encrypts a plaintext env file on every replay, so an accidental plaintext edit is repaired rather than pushed.

Bad

• SOPS encrypts dotenv comments. A self-documenting env file becomes a wall of #ENC[...] — worse than no comments, since the noise looks like data. The env file is therefore bare KEY=VALUE and all explanation lives in 015 and the README.
• Two keys must stay plaintext (BOX_NAME, ADMIN_USER) or the repo cannot bootstrap itself: adm must read them on a box where sops is not installed yet, which is every box at the start of 000. Neither is secret.
• The age key is plaintext on the box by necessity, since adm replay runs unattended. Encryption changes the repo's exposure, not the box's.
• Losing the box and sharing the repo loses the secrets. The key must also live in the password manager; nothing here can verify that it does.

Confirmation

• 015's secrets-roundtrip asserts that sops -d and adm's own loader agree on a decrypted value — not merely that decryption succeeds, but that what blocks see matches what the file holds.
• encrypt-env verify asserts the file is encrypted, that the bootstrap keys are still readable, and that at least one value really is ciphertext — so an over-broad unencrypted_regex cannot silently turn encryption into theatre.
• adm replay passes end to end in the new layout.

The caveat: this repo is not yet publishable

Encrypting <box>.env was necessary and is nowhere near sufficient.

1. Git history already contains the plaintext. GIT_REMOTE was committed in clear earlier the same day, and the forge URL appeared in prose in several files before being parameterised. Encrypting a file does not retract what is already in history.
2. hosts/<box>/facts/ leaks the network. facts/network.txt records the box's address, gateway, DNS server and search domain; facts/users.txt records accounts; facts/etc-manifest.txt lists every file in /etc. These are committed in plain text by design, because their value is the reviewable diff — encrypting them would destroy the drift detection they exist for.

So the honest status: encryption protects against accidental exposure and makes the repo shareable with people who are already trusted. It does not make this repo publishable. Publication needs one of:

• extracting the generic half into a fresh repo with no history — clean, and natural now that the namespace split exists; or
• rewriting history and accepting that facts/ must then be excluded, encrypted, or scrubbed, each of which costs something real.

Not decided. Recorded so that "the secrets are encrypted" is never mistaken for "this can be made public".

Amendment 2026-10-20: publication is now a real goal

The reopen condition below has fired. #007 makes the generic half a public repository, which is what "a product is meant to be shared" was always reaching for. What follows corrects this entry where later work changed the facts, and leaves the rest standing.

The premise holds; only its status changes. The three decisions above are unaffected. The generic/per-box split and "nothing host-specific in a runbook" come out strengthened — the second turned out to have no enforcement at all, and a block in 015 was carrying one box's blob shas inside a generic runbook for weeks before anyone measured it (21529bd). That is an argument for the rule, not against it.

The route is still open, and this entry's advice is still the safer half of it. Taking the fresh-repo route "rather than trying to launder this history" was written before hooks/scan-history existed. That tool now proves no secret survives a filtered history, which weakens one objection — but not the other. Measured today: the forge's hostname is absent from the working tree and present in 4 commits touching generic paths, in prose written before the remote was parameterised. A path-filter keeps those commits, and scan-history will not flag them, because it checks env files for unencrypted values and a hostname in a paragraph is neither. So the choice is a real one:

• fresh repo, no history — clean, loses the commit messages, which in this repo carry most of the reasoning; or
• path-filtered history plus a text replacement for the forge host — keeps the reasoning, and the replacement is the part that must be checked by reading rather than by a tool.

Deliberately still not decided here; it belongs to #12's execution, where the filtered result can be inspected before anything is pushed.

The second objection is gone. hosts/<box>/facts/ cannot leak into a public repo under either route: a path filter keeps only generic paths, and hosts/ is not one of them. Nothing needs excluding, encrypting or scrubbing — the concern was correct for rewriting this repo in place, which is no longer the plan.

Reopen this if

• Publication becomes a real goal, at which point take the fresh-repo route rather than trying to launder this history. Fired 2026-10-20 — see the amendment above. The fresh-repo-versus-filter choice is narrower than this line assumed, and is #12's to make.
• A second box is added, which is the first genuine test of whether the generic/per-box split actually holds.
• Secrets outgrow a dotenv — more than a handful of values, or anything needing rotation — at which point a real secret store beats a file.

005 — Two tiers of key: one operator, one per box

• Status: accepted
• Date: 2026-10-18
• Deciders: ryan

Context and problem statement

#004 put secrets in the repo encrypted with SOPS + age, with each box holding its own key. That left a question it did not answer: can one stored key decrypt every machine's configuration, including machines that do not exist yet?

The answer was "accidentally yes, for now". 015 took the union of every recipient in a single creation_rules entry, and SOPS consults the recipient list only at encryption time and never re-keys existing files. The result was chronology-dependent access: each box could read the env of every box created after it, and none of the ones created before. Neither isolation nor a master key — an artefact of the order the machines happened to be built in.

Worse, the only universal recipient was rea-dev's own box key. The thing that must never be lost was sitting in plaintext on a disposable VM, and a compromise of that VM would hand over the key to every future machine's secrets too.

Decision

Two kinds of identity, with different lifetimes and different reach:

.sops.yaml is generated from operator.pub plus each hosts/<box>/<box>.age.pub, with one creation_rules entry per box naming the operator recipients and that box's key. There is deliberately no catch-all: an unmatched .env makes SOPS refuse rather than encrypt to the wrong set.

Decision drivers

• One key, stored once, must open everything — including future machines.
• The recovery key must not sit on any disposable machine.
• A box has no business reading another machine's secrets.
• adm replay must decrypt that box's own configuration unattended.

Considered options

1. Union of all recipients in one rule. The status quo being replaced.
2. Per-path rules, operator plus that box. Chosen.
3. Per-path rules, operator only — boxes hold no key at all.
4. An external secret manager.

Pros and cons of the other options

Consequences

Good

• One operator key, stored once, decrypts every box's configuration — demonstrated, not assumed: the workstation decrypted rea-dev.env holding no box key at all.
• Per-box isolation is real, not nominal. Verified in a scratch repo: with one rule per box, each box key opened its own file and was denied on the other, while the operator opened both.
• Box keys are now disposable. A rebuild no longer needs the old box key restored, which removes a prerequisite from the rebuild procedure and a thing to back up.
• The recipient topology is declarative — the set of committed public keys — and any box regenerates .sops.yaml byte-identically, so no box fights another over a tracked file.

Bad

• path_regex cannot be templated, so every box needs its own rule. Mitigated by generating the file, but it means adding a box touches a shared tracked file.
• A rebuilt box cannot decrypt its own configuration until an operator re-keys it. The new key is not a recipient, and only someone who can read the file can add one. 015's sync-env-recipients detects this and prints the operator-side remedy, because it will happen mid-rebuild.
• Adding a recipient never re-keys existing files; sops updatekeys is required and must run somewhere that can already decrypt.
• Losing the operator key is now the single unrecoverable failure. That is an improvement — it is one identified thing to protect — but it is absolute.

Confirmation

• sops-rules verify asserts every rule grants an operator recipient, that this box has a rule, and that no catch-all exists. A box becoming unrecoverable is precisely what the first of those prevents.
• sync-env-recipients verify counts the recipients baked into the env file and compares them against the rule, so a file left un-re-keyed fails rather than lurking.
• box-recipient verify asserts the published public key matches the private key actually on the box.
• End-to-end: the operator key on the workstation decrypted rea-dev.env after sops updatekeys retrofitted it.

Reopen this if

• A second operator identity is wanted — a hardware key, or a colleague. Add a line to operator.pub, regenerate, and sops updatekeys every existing env file; nothing else changes.
• Boxes must stop being able to read their own secrets at rest, in which case take option 3 and accept interactive replays.
• The number of secrets or the need to rotate them outgrows a dotenv, in which case option 4 becomes worth its weight.

006 — The root filesystem stays top-level; no @, no true rollback

• Status: accepted
• Date: 2026-10-19
• Deciders: ryan

Context and problem statement

040 now brackets every adm invocation in a snapshot pair, and adm checkpoint / adm restore give named, permanent restore points. All of it is file-level: snapper undochange computes a diff between two snapshots and copies files back.

What that cannot be is a rollback in the sense people mean when they say the word — the VM-style "put the machine back as it was". That is snapper rollback, which creates two snapshots, points btrfs's default subvolume at a writable copy of the target, and reboots into it. It requires the root filesystem to be a nested subvolume, conventionally @, so that something outside it can be redirected. This box's root is subvolid=5 — FS_TREE, the filesystem itself. There is nothing to point elsewhere.

So: migrate to @ and gain true rollback, or stay as we are?

Decision

Root stays top-level. No @ migration. Rollback on this box means file-level restore, and whole-machine recovery stays at the hypervisor layer where #003 already put it.

Decision drivers

• The migration is one-way, and its failure mode is a box that does not boot. It is normally done from rescue media, not from the running system it is rearranging.
• @-rollback is worse than the tier we already have. A hypervisor snapshot covers the ESP, the partition table, and a box that will not POST; @ covers one subvolume of one filesystem.
• The only thing @-rollback buys that file-level restore cannot is repairing an unbootable box from inside, because undochange needs a running system to execute in. On a VM, that case is already covered one layer down.
• The image this box came from — Arch's own arch-boxes — lays out root without @. Migrating means the rebuild procedure in README.md no longer matches what a fresh image gives you.

Consequences

Good

• Nothing risky happens, and the rebuild path keeps matching the stock image.
• Filters remain the exclusion mechanism, which is what makes exclusions cheap. Every computed diff honours /etc/snapper/filters/*.txt, so keeping /home or a container volume out of a restore is a line in a file. Under @-rollback there is no diff to filter, so the same exclusions would have to exist as real subvolumes.

Bad

• No atomic rollback, and no boot-into-a-snapshot-and-commit. 050 installs grub-btrfs and there are snapshot boot entries, so a snapshot can be inspected from the boot menu — but it cannot be made permanent. Half that machinery is inert, by this decision rather than by accident.
• Consistency after a restore is not guaranteed. Snapper says so itself; its example is undoing a user creation and leaving that user's files behind.

Confirmation

• btrfs subvolume get-default / reports ID 5 (FS_TREE) and findmnt shows subvol=/, so snapper rollback is structurally unavailable. Read off the layout rather than tested: the failure mode of testing it is an unbootable box, and there is nothing to learn that the layout does not already say.
• adm undo and adm restore exist, are guarded, and have been exercised on a real pair.

Reopen this if

• This repo describes a box with no hypervisor beneath it. This is the case the whole argument turns on. A native install has no lower tier, so recovering an unbootable system from inside stops being redundant and becomes the only option available.
• A box is being provisioned from scratch anyway. The migration is nearly free at install time and expensive afterwards. If a fresh machine is being laid down, create @ and @home then and this decision costs nothing to reverse.

If either happens, the ordering is not negotiable: /home, and anything like /var/lib/docker, must become their own subvolumes before the @ migration, not after. Rolling back @ takes everything inside @ with it, and a snapshot cannot exclude a directory — only a separate subvolume is excluded, because snapshots do not recurse into child subvolumes. Do these in the wrong order and the first rollback reverts a user's home directory. That is why distributions shipping rollback hand you @ and @home together from the installer; it is not a stylistic choice.

007 — The generic half becomes a public repo; machines are private forks

• Status: done 2026-10-20 (issue #12). Upstream is adm on GitHub; this repo is a fork of it, and its first commit is upstream's.
• Date: 2026-10-20
• Deciders: ryan

Context and problem statement

#004 split generic from per-box but left publication undecided, and its premise — "a product is meant to be shared" — stayed aspirational. The collision became concrete once this repo went private: anything under hosts/<box>/ is then readable only by repo members, which is right, but if every machine repo is private the generic half has no home. N copies, quietly diverging, and the runbooks stop being the definition and become per-machine forks of one. That is the single failure this project cannot absorb.

The export changed one half of the problem in the meantime. A published document now carries its own sources and bin/adm, verified byte-for-byte on every render, so the generic half can travel in one file to anyone holding a link. That solved distribution. It did nothing for canonicity — being able to hand someone a copy is not the same as there being an original.

Decision

The generic half becomes a public repository. Each machine repo is a private fork of it, and pulls fixes with git merge upstream/master.

Considered and rejected

Consequences

Good

• "The runbooks are the definition" becomes true rather than aspirational.
• Pulling upstream is plain git, and adm verify is a real acceptance test afterwards: 76 verify blocks name exactly which adm run a merge has left owing.
• A fork relationship is what makes merging work at all. A forge template starts an unrelated history and cannot merge later.

Bad

• Sending a fix upstream needs care. A commit made on a machine may touch both halves, and pushing it as-is would put one box's data in a public repo. The safe shape is to cherry-pick a generic-only commit onto a branch from upstream/master.
• operator.pub holds this operator's public recipients. Not secret, not generic; the public repo needs an empty or example one.
• The split is only as reliable as its enforcement, and there was none until issue #11.

Confirmation

• #11's verify — a runbook naming a specific box fails. It is the precondition for this decision being safe, not an improvement on it.
• hooks/scan-history over whatever history the public repo starts with. It scans every blob reachable from every ref and is the existing tool for proving nothing per-box survived.
• The per-box allowance list moved out of the generic half in 21529bd; a public repo correctly starts with none.

Reopen this if

• Forks diverge enough that "one definition" stops describing reality — at which point the honest move is to say so rather than to keep merging.
• The generic/per-box boundary moves. Each move is a re-test of whether the split holds, and #11 is what makes a move visible.

What publication actually cost

The generic half was published as one commit, not as filtered history, and the reason is worth recording because the first attempt was wrong and public for ten minutes.

Filtering seemed safe: drop hosts/, operator.pub, *.env, confirm no per-box path remains, confirm scan-history is clean. All of that passed, and the result still carried one machine's network configuration, its account list and a 24 KB inventory of its /etc. Per-box files had lived at three spellings over the project's life — the repo root, then rea-dev/, then hosts/rea-dev/ — and a filter written against the current layout keeps the oldest. scan-history did not object because those files hold no secret by its definition: they are facts, which is exactly why they are committed.

The verification was wrong in the same way the filter was, which is the part worth remembering. The paths checked for were the paths excluded, so the check confirmed the filter had done what it was told rather than that the result was clean. The question that would have caught it is which top-level paths have ever existed in this history — answerable in one command, and not asked.

A single commit has no such surface. The cost is the commit messages, which carried most of the reasoning; this file is where that reasoning lives instead, which is an argument for keeping it current rather than for keeping the history.

• Nobody ever forks it. A public repo with one user is a private repo with extra steps, and the reasoning above would not have been wrong, only moot.

000 — Bootstrap

Required — every box runs this.

Take a freshly provisioned Arch box to a documented baseline with this repo installed and running.

This runbook is generic: it should work on any stock Arch machine, and nothing in it is specific to the box it happens to be running on. Values that differ per machine live in <box>.env. For what one particular box looked like when it was adopted — provenance, hardware, loose ends — see AS-FOUND.md; that archaeology deliberately does not live here.

Examples below use rea-dev for concreteness. Substitute your own.

Assumes

• A stock Arch install, reachable over SSH, with python installed. bin/adm is Python, so it cannot install its own interpreter: pacman -S python is the one command that has to be run before anything here can be. Arch's base does not include it.
• A login that is root or can sudo. Every privileged command is written with an explicit sudo, so either works — which matters because admin does not exist until 010.
• Working network and a usable mirror list.
• This repo already present at /opt/admin.

That last point makes the runbook mildly self-referential: it sets up the repo that contains it. On a real rebuild you get the repo onto the box first — clone it from the remote configured in 020, or rsync it from a workstation — and only then replay. Every block is idempotent, so replaying against an already-bootstrapped box is a no-op.

Per-box values

adm exports every key in <box>.env into the environment of every block, so runbooks refer to $BOX_NAME rather than hardcoding a name. Standing up another machine means editing that one file.

test -n "${BOX_NAME:-}" || { echo "BOX_NAME unset -- is <box>.env present?"; exit 1; }
echo "BOX_NAME=$BOX_NAME  ADMIN_USER=${ADMIN_USER:-unset}"

Which runbooks this box runs

Every runbook declares itself, in a fenced adm block directly under its purpose line:

```adm
required
```

Required runs on every box. Optional runs only where the host env opts in by number, through OPTIONAL_RUNBOOKS:

OPTIONAL_RUNBOOKS=050 070 080

A runbook declaring neither is unclassified, and adm will not run or verify it. That is deliberate rather than defaulting either way: a new runbook that silently ran everywhere and one that silently ran nowhere are both failures, and only one of them is visible. adm list names it in red.

The flag is in the file rather than the filename. Filenames are referenced fifty-nine times across this repo and anchor every link in the export, so a rename to flip one runbook would rewrite its inbound links each time.

Silence is asymmetric, and that is the point

An optional runbook nobody opted into produces no output at all during a run — not a warning, not a skip notice. With enough of them, per-run notices about things deliberately not in use train people to stop reading adm run.

An opt-in naming a runbook that does not exist must fail loudly. Silence there would hide a typo that disables a runbook the operator believes is running, which is the dangerous direction. That check lives in a verify block rather than only in adm run, so a misspelling fails on every adm verify instead of waiting for a replay.

Not opting in also means not verified. 080's export-renderer asserts cmark-gfm is installed; on a box that never enabled 080, that is a failure reported for something deliberately absent. So adm verify skips them too, which sharpens what DECISIONS #001's acceptance test means: a clean verify covers every required runbook plus the optional ones this box chose.

One invariant this adds, which nothing enforces

A required runbook must never depend on an optional one, because the optional one may not have run — and the bug would appear only on the boxes that did not opt in, which is the worst possible distribution.

Nothing checks it, and the obvious check does not work. Nine required runbooks already link to optional ones, every link a legitimate cross reference rather than a dependency, so scanning for references would report nine false positives and teach everyone to ignore it. Distinguishing "refers to" from "relies on" needs a human.

cd "$ADM_ROOT"
state() {
  awk '/^```adm$/{f=1;next} f&&/^```$/{exit} f{print}' "$1" | tr -d '[:space:]'
}

bad=0 req=0 opt=0
for f in runbooks/*.md; do
  case "$(state "$f")" in
    required) req=$((req+1)) ;;
    optional) opt=$((opt+1)) ;;
    "")  echo "$f declares neither required nor optional"; bad=1 ;;
    *)   echo "$f declares an unknown state: $(state "$f")"; bad=1 ;;
  esac
done
test "$bad" = 0 || { echo "adm will not run or verify an unclassified runbook"; exit 1; }

# The asymmetry: an opt-in naming something absent is a typo that silently
# disables a runbook the operator believes is running.
missing=""
for n in ${OPTIONAL_RUNBOOKS:-}; do
  hit="$(ls runbooks/ | grep -c "^$n-" || true)"
  if [ "$hit" = 0 ]; then
    missing="$missing $n(no such runbook)"
  elif [ "$(state "runbooks/$(ls runbooks/ | grep "^$n-" | head -1)")" != optional ]; then
    missing="$missing $n(not optional)"
  fi
done
test -z "$missing" \
  || { echo "OPTIONAL_RUNBOOKS names:$missing"
       echo "an opt-in must name a runbook that exists and declares itself optional"
       exit 1; }

echo "$req required, $opt optional, ${OPTIONAL_RUNBOOKS:-none} opted in"

Every block a runbook declares has to be a block adm can see

A fence is closed by the first line that looks like a closing fence, and parsing then resumes after it. Several blocks here write whole documents from heredocs, and those documents contain fenced code of their own — so a stray fence inside one truncates the block it is in and can swallow the blocks that follow. Nothing complains. adm run and adm verify simply have fewer blocks to work with than the file appears to declare, and both report success over what is left.

That is not hypothetical and it was not a stranger's mistake. A section was once inserted into 070 by anchoring on a heading that occurs twice — once in that runbook's prose, once inside the skill document the runbook writes. The fenced block went into the heredoc, agent-skill lost its closing fence and its own verify, and adm verify 070 reported all blocks ok while running eleven of thirteen.

cd "$ADM_ROOT"
PYTHONDONTWRITEBYTECODE=1 python3 - <<'PY'
import importlib.machinery as M, importlib.util as U
import io, os, re, sys
sys.argv = ["adm"]
loader = M.SourceFileLoader("a", "bin/adm")
mod = U.module_from_spec(U.spec_from_loader("a", loader))
loader.exec_module(mod)

# Column 0 only. An indented fence is an illustration inside prose, which is
# not a declaration and is not meant to run; a buried one is at column 0,
# because it was written to be a block and ended up inside another.
fence = re.compile(r"^```(?:bash|sh) id=(\S+)", re.M)
bad = []
for path in sorted(mod.all_runbook_paths()):
    declared = fence.findall(io.open(path, encoding="utf-8").read())
    parsed = [b.attrs["id"] for b in mod.parse_runbook(path)]
    for ident in declared:
        if ident not in parsed:
            bad.append("%s#%s" % (os.path.basename(path), ident))
if bad:
    print("declared at the start of a line, but not visible to adm:")
    for b in bad:
        print("  " + b)
    print()
    print("Each is inside another block's body -- almost certainly a heredoc")
    print("that writes markdown. adm would skip it silently.")
    raise SystemExit(1)
print("every declared block parses, across %d runbooks"
      % len(mod.all_runbook_paths()))
PY

Adopting a box

A box is adopted by adm adopt <box>, which creates hosts/<box>/, writes the env file with the keys that must be readable before anything can decrypt, and appends the anchored negation to hosts/.gitignore.

It was prose until now, and prose is the one form this repo cannot check. It drifted exactly as you would expect: the README said two keys were enough to bootstrap for several hours after this runbook began requiring a third, so the documented path for a new machine was broken and nothing could notice.

The command does not restate the key list. It reads the runbooks for ${KEY:?...} assertions — the same text the runbooks themselves fail on — and writes the ones it can supply, then reports the rest with the runbook that will demand each. The instructions cannot disagree with the runbooks because there is only one list.

Two details are load-bearing, both found by getting them wrong:

• The negation is committed, not worked around with git add -f. Once hosts/* ignores a directory, files created inside it later are invisible rather than untracked, so adm facts would write snapshots git status never mentions.
• The pattern is anchored — /* then !/<box>/. An unanchored * matches at every depth, so re-including the directory lets git descend into it and ignore its contents all over again.

Adopting as something

adm adopt <box> --from workstation

A profile is an env preset, not a separate set of runbooks. It carries the opt-in list for a purpose plus whatever non-secret defaults those runbooks want, and adm adopt instantiates it into hosts/<box>/<box>.env, recording ADOPTED_FROM so a later reader can tell what was chosen. Separate runbook sets per profile would break "the repo is one document in order" and bring back the dependency problem; a preset selects, it does not fork.

The opt-in list for a purpose is itself knowledge. "A workstation wants user home subvolumes, boot-into-snapshots, the agent harness and the export" is a judgement someone made once. Without presets every new box re-derives it by reading the runbook directory and guessing.

Presets are named <name>.preset, not <name>.env, deliberately. 015's commit hook refuses any *.env that is not SOPS-encrypted — the rule that stops a machine's secrets reaching a remote in clear — and a preset must be plaintext, because a fork has no key for it. Exempting templates/ from the scanner would have been the easy fix and the wrong one: the file is not a box env, so it should not take the name. The scanner's key-material check still applies to it, as it does to every blob.

A preset may contain only keys the encryption policy exempts. That rule is not invented for presets: .sops.yaml's unencrypted_regex is this repo's definition of what is not a secret, and a preset is a plaintext file shipped to every fork. So 070's LLM credentials and 080's publish keys cannot be in one — the preset turns those runbooks on, and the operator supplies the values with sops afterwards. A key-shaped value in a shipped preset is a trap adm env-check could never catch, because the name would be perfectly well formed.

Profiles are exclusive for now. --from workstation,docker-host raises ordering and conflict questions immediately, and there is no evidence yet about what they should be.

cd "$ADM_ROOT"
test -d templates || { echo "no templates/ directory"; exit 0; }
shopt -s nullglob
presets=(templates/*.preset)
test "${#presets[@]}" -gt 0 || { echo "templates/ holds no presets"; exit 0; }

unenc="$(awk -F\' '/^[[:space:]]*unencrypted_regex:/ {print $2; exit}' .sops.yaml)"
test -n "$unenc" || { echo "no unencrypted_regex in .sops.yaml"; exit 1; }

state() {
  awk '/^```adm$/{f=1;next} f&&/^```$/{exit} f{print}' "$1" | tr -d '[:space:]'
}

bad=0
for p in "${presets[@]}"; do
  while IFS= read -r line; do
    case "$line" in ''|'#'*) continue ;; esac
    [[ "$line" == *=* ]] || continue
    k="${line%%=*}"
    # The whole rule: a preset is plaintext and shipped, so it may carry only
    # what the policy already says is not a secret.
    grep -qE "$unenc" <<<"$k" \
      || { echo "$p sets $k, which the policy does not exempt"; bad=1; }
    if [ "$k" = OPTIONAL_RUNBOOKS ]; then
      for n in ${line#*=}; do
        f="$(ls runbooks/ 2>/dev/null | grep "^$n-" | head -1)"
        if [ -z "$f" ]; then
          echo "$p opts into $n, which is not a runbook here"; bad=1
        elif [ "$(state "runbooks/$f")" != optional ]; then
          echo "$p opts into $n, which is not optional"; bad=1
        fi
      done
    fi
  done < "$p"
done
test "$bad" = 0 || exit 1
echo "${#presets[@]} preset(s), none carrying a secret, all opt-ins real"

The README's flag table, against the parser

README.md lists which flags adm run, adm replay and adm verify take. It said -k worked on "any of them" while adm replay -k was an error — a claim about a program, written beside the program, in a form the program could not read. It is a fence now, and this compares it with --help.

Both directions fail. A flag the README claims and the parser rejects is the original defect; a flag the parser accepts and the README omits is how the first one started.

cd "$ADM_ROOT"
PYTHONDONTWRITEBYTECODE=1 python3 - <<'PY'
import io, re, subprocess

text = io.open("README.md", encoding="utf-8").read()
# Located by content, not position: every line is "adm <cmd> <flags...>", so
# adding a paragraph above the fence is not a silent breakage.
claims = {}
for block in re.findall(r"^```\n(.*?)^```\n", text, re.M | re.S):
    rows = [l.split() for l in block.strip().split("\n") if l.strip()]
    if (rows and all(len(r) >= 3 and r[0] == "adm" for r in rows)
            and all(f.startswith("-") for r in rows for f in r[2:])):
        claims = {r[1]: set(r[2:]) for r in rows}
        break
if not claims:
    raise SystemExit("README has no fenced flag table. It documented these in "
                     "prose once\nand the prose was wrong, which is why the "
                     "fence exists.")

# The usage line, not the options list. argparse prints every flag there in
# brackets, one spelling each -- the same spelling the README uses -- so it
# needs no regex over help text. Reading the options list instead is what the
# first version of this did, and a mis-escaped character class turned "-n,
# --dry-run" into a flag called --d.
bad = []
for cmd, want in sorted(claims.items()):
    out = subprocess.run(["./bin/adm", cmd, "--help"],
                         capture_output=True, text=True).stdout
    usage = out.split("options:")[0]
    have = set()
    for tok in re.findall(r"\[([^\]]+)\]", usage):
        first = tok.split()[0]
        if first.startswith("-"):
            have.add(first)
    have.discard("-h")
    for f in sorted(want - have):
        bad.append("%s does not accept %s, but README lists it" % (cmd, f))
    for f in sorted(have - want):
        bad.append("%s accepts %s, which README does not list" % (cmd, f))

if bad:
    for b in bad:
        print("  " + b)
    raise SystemExit(1)
print("README's flag table matches the parser for %d command(s): %s"
      % (len(claims), ", ".join(sorted(claims))))
PY

Requirement: vCPU capability

Checked first, because the cost of discovering it late is the whole run. The agent harness in 070 is a Bun-compiled binary that needs AVX2, and on a CPU without it the failure is not a clean error — it spins at 100% for ever producing no output, which reads like a hang or a network fault. Finding that out after a full system upgrade, a user, a remote and six more runbooks is a waste of an hour and a reboot.

Hypervisors make this easy to get wrong. Proxmox defaults new VMs to the kvm64 model, which is roughly x86-64-v1 — no AVX, no AVX2, not even SSE4.2 or AES-NI. This box shipped that way, and 070 was blocked until the VM's CPU type was changed to host.

The check reports the x86-64 microarchitecture level rather than listing flags, because that is the vocabulary hypervisors and toolchains use. REQUIRE_CPU_LEVEL defaults to 3 and can be lowered in <box>.env for a box that genuinely does not need an agent — a deliberate exception rather than a silent one.

It is exempt from encryption, which it has to be. This block runs before 015 has generated anything that could decrypt a value, so an encrypted REQUIRE_CPU_LEVEL is an escape hatch that cannot be opened at the only moment anyone reaches for it: a fresh box, refusing to start, on a CPU the operator has already decided to accept. It was encrypted until a bootstrap on such a box went looking for the hatch and found it locked.

have() { grep -qw "$1" /proc/cpuinfo; }
# Every flag must be present. Note: a bare `for ... || break` cannot express
# this -- a loop's exit status is the status of `break`, which is zero, so the
# first version of this check reported v2 on a CPU with no SSE4.2 at all.
all() { local f; for f in "$@"; do have "$f" || return 1; done; return 0; }

level=1
if all cx16 lahf_lm popcnt sse4_1 sse4_2 ssse3; then level=2; fi
if (( level == 2 )) && all avx avx2 bmi1 bmi2 f16c fma movbe xsave; then level=3; fi
if (( level == 3 )) && all avx512f avx512bw avx512cd avx512dq avx512vl; then level=4; fi

want="${REQUIRE_CPU_LEVEL:-3}"
model="$(grep -m1 'model name' /proc/cpuinfo | cut -d: -f2- | sed 's/^ //')"
echo "cpu: $model -> x86-64-v$level (require v$want)"

if (( level < want )); then
  echo
  echo "This vCPU is below the level this stack requires."
  echo "In Proxmox: set the VM's CPU type to 'host' (or x86-64-v$want) and reboot."
  echo "'kvm64', the default, is v1 and lacks AVX2, AES-NI and much else."
  echo "A CPU type change needs a full stop/start, not a warm reboot from"
  echo "inside the guest -- the vCPU model is set when the VM process starts."
  echo "If this box genuinely needs no agent harness, set REQUIRE_CPU_LEVEL"
  echo "in <box>.env to record that as a decision."
  exit 1
fi

Boot environment

Reported early so a rebuild on different firmware is visible before it causes confusion, and because 050 has a real requirement here.

Firmware is not the constraint. BIOS and UEFI both work; what matters is which bootloader and where /boot lives. 050 drives GRUB, so a box using systemd-boot gets everything except boot-into-snapshot. And 040/050 both depend on /boot being inside the root filesystem, so a snapshot carries the kernel and initramfs that matched it — a UEFI install that mounts the ESP at /boot instead of /efi breaks that, on otherwise identical hardware.

This block reports rather than dictates; the hard requirements live in the runbooks that actually need them.

if [[ -d /sys/firmware/efi ]]; then fw="UEFI"; else fw="BIOS (legacy)"; fi

loader="unknown"
if command -v grub-mkconfig >/dev/null && [[ -d /boot/grub ]]; then
  loader="grub"
elif sudo bootctl is-installed >/dev/null 2>&1; then
  loader="systemd-boot"
fi

# /boot inside the root filesystem is what lets a snapshot carry its kernel.
if boot_src="$(findmnt -no SOURCE /boot 2>/dev/null)"; then
  root_src="$(findmnt -no SOURCE /)"
  if [[ "$boot_src" == "$root_src" ]]; then boot_loc="same filesystem as /"
  else boot_loc="SEPARATE ($boot_src)"; fi
else
  boot_loc="inside the root filesystem"
fi

echo "firmware: $fw   bootloader: $loader   /boot: $boot_loc"
test "$loader" != "unknown" || {
  echo "no recognised bootloader found; 050 cannot manage this box"; exit 1; }

Order matters: keyring, then full upgrade, then everything else

This is the one sequencing rule in the runbook, and getting it wrong on a stale image breaks the system.

Arch is a rolling release with no supported partial-upgrade path. Syncing the package databases and then installing a single package (pacman -Sy foo) links that package against libraries newer than the ones on disk. Never -Sy without -u.

The keyring is the one documented exception. On an image that has sat for months, some packages are now signed by keys it has never seen, and a full upgrade aborts on signature errors before it can fix itself. Pulling archlinux-keyring ahead of the upgrade is the sanctioned way out.

sudo pacman -Sy --needed --noconfirm archlinux-keyring

pacman -Qi archlinux-keyring | awk -F': +' '/^Version/ {print "keyring", $2}'
sudo pacman-key --list-keys >/dev/null

Now the full upgrade. On a long-stale image this is the step most likely to need a human: read any .pacnew notices and intervention warnings it prints rather than assuming --noconfirm handled everything.

sudo pacman -Syu --noconfirm

if pacman -Qu; then
  echo "ERROR: upgrades still pending"
  exit 1
fi
echo "no pending upgrades"

Reboot if the kernel was replaced

When an upgrade replaces the kernel, the running kernel's module tree is deleted from disk, and loading any module not already resident fails until reboot. The symptom is confusing and arbitrary, so check for it explicitly.

Deliberately not paired with an automatic reboot: rebooting is a decision, not a side effect of a verify. Expect this to fail after any kernel upgrade and to pass once rebooted.

running="$(uname -r)"
installed="$(pacman -Q linux | awk '{print $2}')"
if [[ -d "/usr/lib/modules/$running" ]]; then
  echo "running kernel $running still has its module tree; no reboot required"
else
  echo "REBOOT REQUIRED: running $running, installed $installed,"
  echo "and /usr/lib/modules/$running is gone."
  exit 1
fi

To reboot, deliberately: sudo systemctl reboot.

Host identity

Stock Arch images ship a placeholder hostname — arch, typically — and an SSH client alias on your workstation does not change what the box calls itself. The mismatch is invisible until something that reports its own hostname confuses you: logs, mail, certificates, monitoring, prompts.

Setting it is cheap, but it is a rename, so it is a separate decision from the rest of the bootstrap. If the box is already named correctly this block is a no-op.

if [[ "$(hostnamectl --static)" != "$BOX_NAME" ]]; then
  sudo hostnamectl set-hostname "$BOX_NAME"
fi
hostnamectl --static

test "$(hostnamectl --static)" = "$BOX_NAME" \
  || { echo "static hostname is '$(hostnamectl --static)', expected '$BOX_NAME'"; exit 1; }
echo "hostname is $BOX_NAME"

Admin tooling

Kept deliberately small — just what this repo needs to function. Editors, language toolchains and services each earn their own runbook, so the reason for a package being present is written down next to it rather than accumulating in one anonymous list.

python is in that list and used to be explained as absent, with the reason that it comes with Arch's base. It does not. base is a package of 38 dependencies and none of them is Python; there is no base group at all:

$ pacman -Qi base | grep -c python
0

Every box this repo has ever run on had Python because a Proxmox cloud-init image installs it as a dependency — of the service 030 exists to switch off. On this box it was never an explicitly installed package, which means pacman -Rns cloud-init, a reasonable thing to do once it has been disabled, was a plausible route to removing the interpreter the whole repository runs on.

Installing it here does not make it available — it has to be there already, since adm is what runs this block. It makes it owned: an explicit install reason, so nothing takes it away as an unused dependency, and a line in a table saying why it is on the box.

The conclusion the old note drew is still right, for a better reason. adm is written against the standard library, so it needs the interpreter and nothing else — no virtualenv, no packages, no bootstrapping. That is why the bootstrap-critical keys in <box>.env are left unencrypted: adm has to read them before this block has run.

sudo pacman -S --needed --noconfirm git rsync jq sops age python
# --needed skips a package already present, but asking for it by name is what
# marks it explicitly installed. On a box where Python arrived as a
# dependency, that is the whole point of naming it here, so make the install
# reason certain rather than hoping -S set it.
sudo pacman -D --asexplicit python >/dev/null

for p in git rsync jq sops age-keygen python3; do
  command -v "$p" >/dev/null || { echo "missing: $p"; exit 1; }
done
# Explicitly installed, not merely present. `command -v python3` succeeded on
# this box for a year while the package was only a transitive dependency of
# cloud-init, which is exactly the state that makes it removable without
# anything objecting. Presence is what adm needs today; ownership is what
# keeps it here.
pacman -Qe python >/dev/null 2>&1 \
  || { echo "python is installed but not owned: it is a dependency of"
       echo "something else, so removing that something takes bin/adm's"
       echo "interpreter with it. Fix: sudo pacman -D --asexplicit python"
       exit 1; }
echo "git $(git --version | awk '{print $3}'), rsync, jq, sops $(sops --version --disable-version-check 2>/dev/null | awk '{print $2}'), age, python $(python3 -V | awk '{print $2}') present and owned"

Initialise the repo

The HEAD guard matters: this block makes the initial commit only. Without it, every replay would sweep whatever happened to be uncommitted into a commit labelled "initial commit" — which is precisely what it did on its first revision here, mislabelling an unrelated change. Committing ongoing work is a human decision with a human message.

cd "$ADM_ROOT"
if [[ ! -d .git ]]; then
  git init -q -b master
fi
# The commit identity, set on the repository rather than on a user. See below
# for why; it is here because the commit this block makes is the first thing
# that needs it, and because an identity for a repository cannot be written
# before the repository exists.
git config --local user.name  "$BOX_NAME admin"
git config --local user.email "admin@$BOX_NAME"
# This one is genuinely per-user: it decides the branch name of repos not yet
# created, so it has nowhere else to live. Belt and braces either way -- the
# init above names master explicitly -- but an operator typing `git init` by
# hand gets the same answer.
git config --global init.defaultBranch master
if ! git rev-parse --verify -q HEAD >/dev/null; then
  git add -A
  git commit -q -m "admin: initial commit of runbooks and adm runner"
fi
git log --oneline -1

cd "$ADM_ROOT"
test -d .git || { echo "no git repo at $ADM_ROOT"; exit 1; }
# That master is the trunk, not that you are standing on it. Work happens on
# branches -- `adm land` requires one -- and a check that fails for being
# somewhere else is testing where you stand rather than what it claims to.
git rev-parse --verify --quiet master >/dev/null \
  || { echo "no master branch; master is this repo's trunk"; exit 1; }
echo "master: $(git log --oneline -1 master)  (HEAD on $(git rev-parse --abbrev-ref HEAD))"

Checking that adoption held

adm adopt ran before this runbook did, and this is where its work is checked rather than where it happens. The check lives here, among the system steps, because it asks git whether the box directory is visible -- which needs the repository the block above just created. It sat in the preflight group until a fresh box proved it could not: on a machine bootstrapping for the first time, it ran fourteen blocks before anything had made a repository for it to ask. Every box after the first one hid the problem, because the repo was already there from last time.

cd "$ADM_ROOT"
box=admadopttest
test ! -e "hosts/$box" || { echo "hosts/$box exists; clean it up first"; exit 1; }

cleanup() {
  rm -rf "hosts/$box" "hosts/${box}2"
  if [ -f hosts/.gitignore ]; then
    grep -v -E -x -- "!/${box}2?/" hosts/.gitignore > hosts/.gitignore.tmp \
      && mv hosts/.gitignore.tmp hosts/.gitignore
  fi
  git rm -q --cached -r --ignore-unmatch "hosts/$box" "hosts/${box}2" >/dev/null 2>&1 || true
  git restore --staged hosts/.gitignore >/dev/null 2>&1 || true
}
trap cleanup EXIT

adm adopt "$box" >/dev/null || { echo "adm adopt failed"; exit 1; }
# Two box directories now exist, so adm must be told which one it is running
# as. Naming it rather than relying on the hostname also keeps this honest on
# a machine whose hostname does not match its box.
export ADM_BOX="$BOX_NAME"
env="hosts/$box/$box.env"
test -f "$env" || { echo "$env not created"; exit 1; }
test "$(grep -c "^BOX_NAME=$box\$" "$env")" = 1 \
  || { echo "BOX_NAME is not $box"; exit 1; }

# Required keys, read from inside bash blocks only. Grepping the raw markdown
# instead found KEY, from the paragraph above that describes the pattern:
# prose is not code, and adm does not treat it as code either.
keys() {
  python3 - "$@" <<'PY'
import io, re, sys
for path in sys.argv[1:]:
    fence = False
    for line in io.open(path, encoding="utf-8"):
        if line.startswith("```"):
            fence = not fence
            continue
        if fence:
            # Comment lines skipped, exactly as _required_env_keys skips them.
            # This check compares adopt against that derivation, so the two
            # have to agree about what counts -- and when they agreed by
            # sharing a defect, a key invented by a comment passed both.
            if line.lstrip().startswith("#"):
                continue
            for k in re.findall(r"\$\{([A-Z][A-Z0-9_]*):\?", line):
                if not k.startswith("ADM_"):
                    print(k, path)
PY
}

# A key required before anything on the box can decrypt has to be in this
# file: there is no other way to supply it, because the age key does not
# exist yet. Which runbook that boundary falls at is derived, not named -- the
# one defining the sops-rules block.
#
# Excluding this runbook is not a detail. A check that greps the runbooks is
# itself a runbook, so whatever it searches for, it finds in its own text.
# Three drafts of this block failed that way in a row: on ".sops.yaml", on the
# word KEY in the paragraph above, and on the very block id below. $ADM_RUNBOOK
# is what adm exports for exactly this -- the file the running block lives in.
sops_rb="$(grep -l 'id=sops-rules' runbooks/*.md \
           | grep -vF -- "$ADM_RUNBOOK" | sort | head -1)"
test -n "$sops_rb" \
  || { echo "no runbook defines sops-rules, so the point where encryption"
       echo "becomes available cannot be located"; exit 1; }

early=""
for rb in $(ls runbooks/*.md | sort); do
  [ "$rb" \< "$sops_rb" ] || break
  early="$early $rb"
done
test -n "$early" || { echo "no runbooks sort before $sops_rb"; exit 1; }

# An earlier draft of this listed ADMIN_USER, ADMIN_UID and BOX_NAME, which
# restates the list it exists to derive -- the exact drift adopt was written
# to make impossible, reproduced inside its own test. It passed while 010
# required a key adopt had never heard of.
missing=""
while read -r k rb; do
  grep -q "^$k=" "$env" || missing="$missing $k($(basename "$rb"))"
done < <(keys $early | sort -u)
test -z "$missing" \
  || { echo "adopt wrote no value for keys needed before encryption exists:"
       echo " $missing"; exit 1; }

# Every other required key must at least be accounted for: written, or named
# in what adopt tells the operator to add later. One it has never heard of is
# the drift this exists to catch.
report="$(adm adopt "${box}2" 2>&1)" || { echo "$report"; exit 1; }
unknown=""
while read -r k _; do
  grep -q "^$k=" "$env" && continue
  grep -q "^  $k " <<<"$report" || unknown="$unknown $k"
done < <(keys $(ls runbooks/*.md | grep -vF -- "$ADM_RUNBOOK") | sort -u -k1,1)
test -z "$unknown" || { echo "adopt neither wrote nor reported:$unknown"; exit 1; }

# The trap: the directory must be visible to git, or facts written into it
# later are invisible rather than merely untracked.
! git check-ignore -q "hosts/$box/" \
  || { echo "hosts/$box is ignored; the negation did not take"; exit 1; }
printf 'probe\n' > "hosts/$box/probe.txt"
git status --porcelain -- "hosts/$box/probe.txt" | grep -q . \
  || { echo "a file inside hosts/$box is invisible to git status"; exit 1; }

grep -q -x -- "!/$box/" hosts/.gitignore \
  || { echo "hosts/.gitignore has no anchored negation for $box"; exit 1; }

adm adopt "$box" >/dev/null 2>&1 \
  && { echo "a second adopt of the same box succeeded"; exit 1; }

cleanup
trap - EXIT
test ! -e "hosts/$box" || { echo "cleanup left hosts/$box behind"; exit 1; }
grep -q -x -- "!/$box/" hosts/.gitignore \
  && { echo "cleanup left the negation in hosts/.gitignore"; exit 1; }
echo "adopt supplies every key the pre-encryption runbooks require, leaves the box visible to git, and refuses a second time"

Git commit identity

Commits made on the box need an author. This is the box's identity, not a person's: a commit authored here means "done on this machine", which is exactly what you want to see when reading the history later.

So it is set on the repository, not on a user, and the difference is not cosmetic. It used to be git config --global, with this runbook claiming that root got it during bootstrap and admin got it afterwards and both ended up with the same values. Nothing made that true. admin does not exist until 010, so on a fresh box only root was ever configured, and the first commit attempted after the pivot failed with Author identity unknown — on a box whose entire record is its commits.

Repo-scoped is also the only answer that stays right as the box gains users. MACHINE_USERS may be real people with their own git identities; a --system setting would rename their commits, and a --global one has to be repeated for each of them and remembered for the next. The repository is the thing the identity describes, so the repository is where it goes.

What repo-scoped gives up is that .git/config is not cloned, so a rebuild that clones from the remote arrives without an identity. That is why repo-init sets it on every run rather than only when it creates the repo: a second run on a cloned repo does nothing else, and sets this.

cd "$ADM_ROOT"
# --local only, deliberately. `git config user.name` would answer from the
# global config too, so it would pass for any user who happened to have one
# set -- which is exactly the state that looked fine for root and failed for
# admin. Reading only the repository's own config asks the real question.
test "$(git config --local user.name)"  = "$BOX_NAME admin" \
  || { echo "the repo has no commit identity of its own; run repo-init"; exit 1; }
test "$(git config --local user.email)" = "admin@$BOX_NAME" \
  || { echo "the repo's commit email is not admin@$BOX_NAME"; exit 1; }
test "$(git config --global init.defaultBranch)" = "master"
# Whoever is running this must actually be able to commit. The identity being
# present and the identity being usable are different claims, and it was the
# second one that broke.
git -C "$ADM_ROOT" var GIT_AUTHOR_IDENT >/dev/null \
  || { echo "git cannot resolve an author for $(id -un) here"; exit 1; }
echo "commit identity: $(git config --local user.name) <$(git config --local user.email)>  (as $(id -un))"

Put adm on PATH

sudo ln -sfn "$ADM_ROOT/bin/adm" /usr/local/bin/adm

test "$(readlink -f /usr/local/bin/adm)" = "$ADM_ROOT/bin/adm"
command -v adm

Record the resulting state

Snapshot the box into hosts/<box>/facts/, establishing the baseline that every later adm facts diffs against. From here on, anything that changes outside a runbook shows up as a reviewable diff.

adm facts

Next

Filename order is replay order, and a runbook may depend only on what is numbered below it. Numbering steps by 10 leaves room to insert.

030 must come after 020 for a non-obvious reason worth knowing before a rebuild: cloud-init is what installs root's authorized_keys, which 010 copies to admin. Disable it too early and a fresh box comes up with no way in. 040 must come after 010 because it grants an ACL to the admin user, and setfacl refuses an account that does not exist.

Still undocumented: what this box is actually for. Everything above is scaffolding — the account, the remote, the snapshots. No runbook yet describes the service this machine exists to run, which is the one that will matter most to whoever reads this repo next.

010 — The admin user

Required — every box runs this.

Create the unprivileged account that owns this repo and runs the runbooks.

Why not just use root

The box was adopted as root, which worked but left two problems.

Explicitness. When everything runs as root, nothing in a runbook says which commands actually need privilege — the reader cannot tell pacman -S nginx (needs root) from git log (does not). Running as admin and writing sudo in front of the privileged commands makes the privilege boundary part of the document. That is worth more here than it would be elsewhere, because this repo's whole claim is that reading it tells you what the box is.

Repo ownership. If admin owns /opt/admin but adm runs as root, git refuses to operate on the tree (detected dubious ownership) and any file a block writes lands root-owned inside an otherwise user-owned repo. One owner for the repo, and privilege acquired per-command, avoids the whole class of problem.

A useful consequence: because root may always sudo, every block written with explicit sudo works whether root or admin runs it. The runbooks are user-agnostic, which matters on a rebuild where admin does not exist yet — see the pivot note below.

Identity and privilege

admin gets no password at all (authentication is by SSH key) and passwordless sudo. Those two facts are the same decision: a password on an account whose entire purpose is unattended administration is a secret to manage with no corresponding security gain, since the SSH key already grants full access and sudo would only re-prove the same thing.

Privilege is granted by one explicit file in /etc/sudoers.d/, not by membership in wheel. Arch's convention is wheel plus an uncommented line in /etc/sudoers, but that splits the grant across two places and makes it invisible unless you think to check group membership. One named file is greppable and says exactly what it does.

The uid is pinned by $ADMIN_UID rather than left to useradd, which would take the next number above UID_MIN. On this box that reliably produced 1000 — but only because AS-FOUND.md records that the image shipped no non-root accounts, so the admin user was always the first. That is a dependency on the base image, not a decision, and it was nowhere written down as the reason.

It matters because this account owns $ADM_ROOT, its own home, the age key and the deploy key. A rebuild that numbered it differently would leave a restored archive's files owned by nobody in particular, and hosts/<box>/facts/ would report the change as drift without explaining it. The group is pinned to the same number, so uid == gid and neither depends on creation order.

ADMIN_UID is readable in the clear in <box>.env for the same reason ADMIN_USER is, and the reason is specific: this runbook runs before 015 generates the box's age key, so at this moment nothing on the machine can decrypt anything. Whatever this block needs has to be plaintext or it cannot be read at all.

: "${ADMIN_USER:?ADMIN_USER unset -- is <box>.env present?}"
: "${ADMIN_UID:?ADMIN_UID unset -- is <box>.env present?}"

# Refuse rather than renumber. Changing the uid of the account that owns
# $ADM_ROOT, the age key and the deploy key means rewriting every owner on
# disk, which is a manual job and not something a replay should start.
if id -u "$ADMIN_USER" >/dev/null 2>&1 \
   && [[ "$(id -u "$ADMIN_USER")" != "$ADMIN_UID" ]]; then
  echo "$ADMIN_USER exists with uid $(id -u "$ADMIN_USER") but <box>.env"
  echo "declares ADMIN_UID=$ADMIN_UID."
  exit 1
fi
owner="$(getent passwd "$ADMIN_UID" 2>/dev/null | cut -d: -f1 || true)"
if [[ -n "$owner" && "$owner" != "$ADMIN_USER" ]]; then
  echo "uid $ADMIN_UID already belongs to '$owner', not '$ADMIN_USER'"
  exit 1
fi

if ! id -u "$ADMIN_USER" >/dev/null 2>&1; then
  getent group "$ADMIN_USER" >/dev/null 2>&1 \
    || sudo groupadd --gid "$ADMIN_UID" "$ADMIN_USER"
  sudo useradd --uid "$ADMIN_UID" --gid "$ADMIN_USER" \
       --create-home --shell /bin/bash \
       --comment "$ADMIN_USER -- owns $ADM_ROOT, runs the runbooks" "$ADMIN_USER"
fi
# No password, ever. Key auth only; '!' is a locked hash, not an empty one.
sudo passwd -l "$ADMIN_USER" >/dev/null
home="$(getent passwd "$ADMIN_USER" | cut -d: -f6)"
sudo install -d -m 0755 -o "$ADMIN_USER" -g "$ADMIN_USER" "$home"
id "$ADMIN_USER"

id "$ADMIN_USER"
test "$(id -u "$ADMIN_USER")" = "$ADMIN_UID" \
  || { echo "uid is $(id -u "$ADMIN_USER"), <box>.env declares $ADMIN_UID"; exit 1; }
test "$(id -g "$ADMIN_USER")" = "$ADMIN_UID" \
  || { echo "gid is $(id -g "$ADMIN_USER"); uid and gid are pinned equal"; exit 1; }
test "$(getent passwd "$ADMIN_USER" | cut -d: -f7)" = "/bin/bash"
# Locked password: field is '!' or starts with '!'
sudo awk -F: -v u="$ADMIN_USER" '$1==u {print $2}' /etc/shadow | grep -q '^!' \
  || { echo "$ADMIN_USER password is not locked"; exit 1; }
echo "$ADMIN_USER exists, shell ok, password locked"

Passwordless sudo

Validated with visudo -c before installation. A syntax error in /etc/sudoers.d/ makes sudo refuse to run at all, which on a box managed without console access is how you lose administrative access permanently. Never write this file in place.

tmp="$(mktemp)"
cat > "$tmp" <<EOF
# $ADMIN_USER owns $ADM_ROOT and runs its runbooks; see $ADM_ROOT/AGENTS.md
# Passwordless because administration is unattended and SSH key auth
# already grants equivalent access -- see runbooks/010-admin-user.md
$ADMIN_USER ALL=(ALL:ALL) NOPASSWD: ALL
EOF
sudo visudo -cqf "$tmp" || { echo "sudoers syntax check FAILED, not installing"; rm -f "$tmp"; exit 1; }
sudo install -m 0440 -o root -g root "$tmp" "/etc/sudoers.d/$ADMIN_USER"
rm -f "$tmp"

f="/etc/sudoers.d/$ADMIN_USER"
test "$(sudo stat -c '%a %U:%G' "$f")" = "440 root:root"
sudo visudo -cqf "$f"
sudo -n -u "$ADMIN_USER" sudo -n true \
  && echo "$ADMIN_USER has passwordless sudo" \
  || { echo "$ADMIN_USER cannot sudo without a password"; exit 1; }

Inbound SSH keys

admin inherits the same authorized keys root was adopted with, copied rather than re-listed so no key material lives in this repo. On a rebuild root's authorized_keys arrives from the image's cloud-init, so this still works.

The copy direction is deliberate: edit root's keys (or cloud-init) and replay, rather than editing admin's by hand — a replay would overwrite hand edits here.

home="$(getent passwd "$ADMIN_USER" | cut -d: -f6)"
sudo install -d -m 0700 -o "$ADMIN_USER" -g "$ADMIN_USER" "$home/.ssh"
sudo install -m 0600 -o "$ADMIN_USER" -g "$ADMIN_USER" \
     /root/.ssh/authorized_keys "$home/.ssh/authorized_keys"
sudo ssh-keygen -lf "$home/.ssh/authorized_keys"

home="$(getent passwd "$ADMIN_USER" | cut -d: -f6)"
want="700 $ADMIN_USER:$ADMIN_USER"
test "$(stat -c '%a %U:%G' "$home/.ssh")" = "$want" \
  || { echo "$home/.ssh is $(stat -c '%a %U:%G' "$home/.ssh"), expected $want"; exit 1; }
test "$(stat -c '%a %U:%G' "$home/.ssh/authorized_keys")" = "600 $ADMIN_USER:$ADMIN_USER"
n=$(sudo ssh-keygen -lf "$home/.ssh/authorized_keys" | wc -l)
test "$n" -ge 1 || { echo "no authorized keys for $ADMIN_USER"; exit 1; }
echo "$ADMIN_USER has $n authorized key(s)"

Root keeps no SSH identity of its own

Root has no ~/.ssh/config and no private key, and must never acquire either: outbound SSH from this box (the git remote, in 020) belongs to admin. Root's authorized_keys is left alone — that is inbound access granted by the image, not an identity root holds.

This verify block is an assertion about something that should stay absent, which is the only kind of guard that catches a future runbook quietly doing the convenient thing.

Every test here goes through sudo, and that is not incidental. /root is mode 0750, so an unprivileged test ! -e /root/.ssh/config succeeds because the path is unreadable, not because it is absent — the guard would report "root has no ssh config" while root happily had one. An absence assertion is only meaningful if the asserting process can actually see the thing it claims is missing. This block read as passing for exactly that wrong reason until a replay as admin exposed it.

sudo test -d /root/.ssh || { echo "/root/.ssh missing entirely -- unexpected"; exit 1; }
! sudo test -e /root/.ssh/config || { echo "root has an ssh config; it should not"; exit 1; }
found=$(sudo find /root/.ssh -maxdepth 1 \( -name 'id_*' -o -name '*.pem' \) -print)
test -z "$found" || { echo "root holds private key material: $found"; exit 1; }
echo "root has no ssh config and no private keys (checked with privilege)"

Hand the repo to admin

sudo chown -R "$ADMIN_USER:$ADMIN_USER" "$ADM_ROOT"
sudo git config --system --add safe.directory "$ADM_ROOT"
ls -ld "$ADM_ROOT"

safe.directory is set system-wide so that if anyone does run git here as root — during recovery, say — it works rather than failing confusingly. It is a safety net, not an invitation.

The check below covers the repo root, .git, and every tracked file. It deliberately excludes hosts/<box>/log/runs/, which is gitignored scratch output owned by whoever ran adm — when this runbook is executed by root during a rebuild, that block's own log files are necessarily root-owned, and failing on them would make the check unpassable on the one path where it matters.

cd "$ADM_ROOT"
want="$ADMIN_USER:$ADMIN_USER"
test "$(stat -c '%U:%G' .)"    = "$want" || { echo "repo root not owned by $ADMIN_USER"; exit 1; }
test "$(stat -c '%U:%G' .git)" = "$want" || { echo ".git not owned by $ADMIN_USER"; exit 1; }
bad=$(git ls-files -z | xargs -0 stat -c '%U %n' \
      | awk -v u="$ADMIN_USER" '$1 != u { print $2 }' | head -5)
test -z "$bad" || { echo "tracked files not owned by $ADMIN_USER:"; echo "$bad"; exit 1; }
echo "$ADM_ROOT: root, .git and $(git ls-files | wc -l) tracked files owned by $ADMIN_USER"

The pivot

This runbook is the point where administration stops being root's job. Everything numbered above it was written to run either way; everything below it assumes admin.

On a rebuild that means: run 000 and 010 as root (or via any account that can sudo), then switch to admin and run the rest. Connect as ssh admin@rea-dev from here on.

Disabling root SSH login entirely is the obvious follow-up, and is deliberately not done here — it should not happen until admin logins have been confirmed working, and it deserves its own runbook so the lockout risk is considered on its own terms.

echo "runbooks run as: $(id -un)"
sudo -n true && echo "sudo: passwordless ok"

015 — Secrets in the repo, encrypted

Required — every box runs this.

Let this repo be shared without leaking what it knows.

The point of this repo is to be a shareable declarative configuration system, which collides with the fact that configuring a real machine involves real secrets. The usual answers are both bad: keep secrets out of the repo and the repo no longer describes the box, or keep them in and the repo can never leave the building.

So: secrets live in the repo, encrypted at rest, with the key living only on the machines entitled to read them.

Why SOPS and age

That diff property is the deciding factor, not cryptographic strength — all four are fine on that axis.

What is and is not encrypted

.sops.yaml matches \.env$ at any depth, and sets unencrypted_regex so a few keys stay plaintext. The list is in the sops-rules block below, as unenc, and it used to be repeated here so a reader could see it without scrolling — until the two needed changing together, which is the moment a second copy stops being a convenience.

ADMIN_UID is exempt for the same bootstrap reason as ADMIN_USER, and the reason is sharper than it looks: 010 runs before this runbook generates the box's age key, so at the moment the admin account is created there is nothing on the machine that could decrypt an encrypted value. Anything 010 needs has to be readable in the clear or it cannot be read at all.

REQUIRE_CPU_LEVEL is exempt for a sharper version of the same reason. 000's cpu-capability block names it as the way to override a CPU check that is refusing to let the box start — and 000 runs before this runbook exists to decrypt anything. Encrypted, it was an escape hatch that could not be opened at the only moment anyone would reach for it.

This is not laziness. adm must read BOX_NAME and ADMIN_USER on a box where sops is not installed yet — which is every box at the start of 000, since sops is installed by 000. Encrypting them would make the repo unable to bootstrap itself. Neither is secret: one is a hostname, the other a conventional account name.

PUBLISH_S3_ENDPOINT, PUBLISH_S3_BUCKET and PUBLISH_S3_PREFIX (080) are plaintext because none of them is a secret: a regional DigitalOcean hostname, the bucket name, and the box name. Only the access key and the secret key are. Encrypting the rest was the "encrypted means secret" confusion again, and it had a concrete cost -- the export's leak gate fired on the bucket name, because heading slugs derive arch-declarative-markdown from the title Arch Declarative Markdown, so a value that was encrypted appeared in the output and not in the source. The gate was right; the classification was wrong.

FORGE_HOST_FINGERPRINT stays plaintext because a public host key fingerprint is not a secret, and is useless without knowing the host — which is encrypted.

MACHINE_USERS (045) is plaintext for the same reason and one more. Account names are not secret, and hosts/<box>/facts/users.txt already commits every one of them in the clear — see DECISIONS.md #004 — so encrypting the list here would be theatre rather than protection. The additional reason is structural: the correct value on a box with no people on it is the empty string, and SOPS does not encrypt an empty dotenv value. It writes MACHINE_USERS= and leaves it, which env-fully-encrypted then reports as an unencrypted key — correctly, having no way to tell an exempt empty value from a leaked one. That check found this the first time the key was added.

Everything else is encrypted, and adm refuses to export a value that is still ENC[...] rather than passing ciphertext into a block as if it were configuration.

No comments in the env file

SOPS encrypts dotenv comments along with values, so a commented, self-documenting env file turns into a wall of #ENC[...] the moment it is encrypted — strictly worse than no comments, because the noise looks like data. The first version of this box's <box>.env had a careful header explaining every key; all of it became ciphertext.

So the env file holds nothing but KEY=VALUE, and every explanation lives here and in README.md where it stays readable. Keys are descriptive enough to carry their own meaning, which is the only self-documentation that survives encryption.

The key

One age identity, at the location SOPS looks by default, in the admin user's home — deliberately outside the repo, so cloning the repo conveys nothing.

key="$HOME/.config/sops/age/keys.txt"
install -d -m 0700 "$HOME/.config/sops" "$HOME/.config/sops/age"
if [[ ! -f "$key" ]]; then
  age-keygen -o "$key" 2>/dev/null
fi
chmod 0600 "$key"
echo "recipient: $(age-keygen -y "$key")"

key="$HOME/.config/sops/age/keys.txt"
test -f "$key" || { echo "no age key at $key -- run 015"; exit 1; }
test "$(stat -c '%a %U' "$key")" = "600 $ADMIN_USER" \
  || { echo "age key has wrong mode/owner: $(stat -c '%a %U' "$key")"; exit 1; }
age-keygen -y "$key" >/dev/null || { echo "age key is unreadable"; exit 1; }
echo "age identity present, recipient $(age-keygen -y "$key")"

Back this key up somewhere other than this box

If the box dies and the repo has been shared, the encrypted values are unrecoverable without this key. Proxmox backs up /home/admin as part of the whole disk (DECISIONS.md #003), so this is not a single point of failure in practice — but a hypervisor restore is a heavy way to recover one line of text.

Put the recipient and the private key in the password manager. Nothing in this repo can check that you did, which is exactly why it is written down here.

Two tiers of key: operator and per-box

There are two kinds of identity, and the distinction is the whole security model:

The operator key is the one thing that must be backed up, and the one thing whose loss is unrecoverable. Keeping it off every managed box means a compromised or discarded VM never hands it over.

A per-box key exists so adm replay can decrypt that box's own configuration unattended. It deliberately grants nothing else: a box has no business reading another machine's secrets.

Why .sops.yaml is generated, not edited

An earlier version of this runbook kept a single rule listing the union of every recipient. That was wrong in a way worth recording, because it looked fine: SOPS consults the recipient list only at encryption time and never re-keys existing files, so the union produced chronology-dependent access — each box could read the env of every box created after it, and nothing could read the ones created before. Neither isolation nor a master key; just an artefact of the order the machines happened to be built in.

Per-path rules fix it, but path_regex cannot be templated, so every box needs its own rule — and a rule naming a box's key requires that key's public half, which that box alone knows. So each box commits its own recipient to hosts/<box>/<box>.age.pub, and .sops.yaml is derived from operator.pub plus those files.

That makes the recipient topology declarative: it is the set of committed public keys, and any box regenerates byte-identical output from them. No churn, and no box fighting another over a tracked file.

There is deliberately no catch-all rule. An .env with no matching rule makes sops refuse rather than silently encrypt to the wrong recipients — loud and recoverable, instead of quiet and not.

cd "${ADM_BOX_DIR:?ADM_BOX_DIR unset}"
# head -1, because a keyring can hold more than one key and age-keygen -y
# prints a recipient for each. `adm unlock` puts a restored box key there
# alongside the one this runbook generated, and without this the box's
# .age.pub was written with two recipients on two lines -- a file whose own
# comment says "Public key" and which sops-rules reads one line from.
recipient="$(age-keygen -y "$HOME/.config/sops/age/keys.txt" | head -1)"
install -m 0644 /dev/stdin "${BOX_NAME}.age.pub" <<EOF
# Public age recipient for this box, written by runbooks/015-secrets.md.
# Public key: safe to commit. The private half never leaves this machine.
$recipient
EOF
cat "${BOX_NAME}.age.pub"

cd "${ADM_BOX_DIR:?ADM_BOX_DIR unset}"
f="${BOX_NAME}.age.pub"
test -f "$f" || { echo "$f missing"; exit 1; }
want="$(age-keygen -y "$HOME/.config/sops/age/keys.txt")"
grep -qx "$want" "$f" \
  || { echo "$f does not contain this box's recipient"; exit 1; }
echo "$BOX_NAME recipient published: ${want:0:28}..."

Generate .sops.yaml

cd "$ADM_ROOT"
test -s operator.pub || { echo "operator.pub missing -- see the runbook"; exit 1; }
# An operator.pub extracted from a published document carries a marker, and
# this refuses while it is there. The recipient in such a file belongs to the
# operator of the machine that document described, and an operator recipient
# is added to every box's rule -- so keeping it silently grants that person
# the ability to decrypt this box. The box cannot tell whose key it is; the
# person unpacking can, which is why this stops rather than warns.
if grep -q "ADM-OPERATOR-UNVERIFIED" operator.pub; then
  echo "operator.pub is marked UNVERIFIED and has not been checked."
  echo
  echo "It came from a published document. The recipient in it belongs to"
  echo "that document's operator, and an operator recipient can decrypt every"
  echo "box it is added to -- including this one, and including secrets that"
  echo "do not exist yet."
  echo
  echo "  if it is yours:     remove the marker comments from operator.pub"
  echo "  if it is not:       replace the recipient with your own first"
  exit 1
fi
recips() { grep -vE '^[[:space:]]*(#|$)' "$1"; }
ops="$(recips operator.pub | paste -sd,)"
test -n "$ops" || { echo "operator.pub lists no recipients"; exit 1; }
unenc='^(BOX_NAME|ADMIN_USER|ADMIN_UID|FORGE_HOST_FINGERPRINT|MACHINE_USERS|OPTIONAL_RUNBOOKS|ADOPTED_FROM|REQUIRE_CPU_LEVEL|PUBLISH_S3_ENDPOINT|PUBLISH_S3_BUCKET|PUBLISH_S3_PREFIX)$'

tmp="$(mktemp)"
{
  echo "# GENERATED by runbooks/015-secrets.md -- do not hand-edit."
  echo "# Derived from operator.pub and each hosts/<box>/<box>.age.pub, so any box"
  echo "# regenerates this file identically. One rule per box: the operator"
  echo "# recipients can read every box, a box key reads only its own."
  echo "# No catch-all: an unmatched .env makes sops refuse rather than"
  echo "# encrypt to the wrong recipients."
  echo "creation_rules:"
  for pub in $(ls -1 hosts/*/*.age.pub 2>/dev/null | sort); do
    box="$(basename "$(dirname "$pub")")"
    key="$(recips "$pub" | head -1)"
    test -n "$key" || continue
    printf '  - path_regex: ^hosts/%s/%s\\.env$\n' "$box" "$box"
    printf '    input_type: dotenv\n'
    printf "    unencrypted_regex: '%s'\n" "$unenc"
    printf '    age: %s,%s\n' "$ops" "$key"
  done
} > "$tmp"

if cmp -s "$tmp" .sops.yaml; then
  echo ".sops.yaml already current"
else
  install -m 0644 "$tmp" .sops.yaml
  echo "regenerated .sops.yaml"
fi
rm -f "$tmp"
grep -cE '^  - path_regex:' .sops.yaml | sed 's/^/rules: /'

# A marked operator.pub must never survive into a working box: the whole
# point is that it stops the bootstrap rather than being carried past it.
grep -q "ADM-OPERATOR-UNVERIFIED" operator.pub 2>/dev/null \
  && { echo "operator.pub still carries the UNVERIFIED marker"; exit 1; }
cd "$ADM_ROOT"
grep -q "AGE_RECIPIENT_PLACEHOLDER" .sops.yaml \
  && { echo ".sops.yaml still contains a placeholder recipient"; exit 1; }

n_rules=$(grep -cE '^  - path_regex:' .sops.yaml)
n_ages=$(grep -cE '^    age:' .sops.yaml)
test "$n_rules" -ge 1 || { echo "no creation_rules in .sops.yaml"; exit 1; }
test "$n_rules" = "$n_ages" || { echo "$n_rules rules but $n_ages age lines"; exit 1; }

# Every rule must grant an operator recipient, or a box becomes unrecoverable.
while IFS= read -r op; do
  missing=$(grep -cE '^    age:' .sops.yaml)
  have=$(grep -E '^    age:' .sops.yaml | grep -cF "$op")
  test "$have" = "$missing" \
    || { echo "operator recipient ${op:0:20}... missing from $((missing-have)) of $missing rules"; exit 1; }
done < <(grep -vE '^[[:space:]]*(#|$)' operator.pub)

# This box must have a rule of its own.
grep -qE "^  - path_regex: \^hosts/${BOX_NAME}/${BOX_NAME}\\\\\.env\\\$" .sops.yaml \
  || { echo "no rule for hosts/$BOX_NAME in .sops.yaml"; exit 1; }

# And no catch-all that would quietly swallow an unknown box.
grep -qE '^  - path_regex: \\\.env\$$' .sops.yaml \
  && { echo "a catch-all rule exists; an unknown .env would be mis-encrypted"; exit 1; }

echo "$n_rules rule(s), every one granting the operator, including $BOX_NAME"

Bring existing files up to date

Adding a recipient to .sops.yaml does not re-key files that are already encrypted — SOPS consults the rules only when encrypting. An env file written before the operator key existed stays readable only by its box, which is precisely the failure this whole arrangement is meant to prevent.

sops updatekeys retrofits it, and must run somewhere that can already decrypt the file. It is a no-op once the recipients match, so this is safe on every replay.

On a rebuild this block is where the design pays off, and where it needs a hand. The rebuilt box generates a fresh identity, which is not yet a recipient of its own encrypted env file — so it cannot decrypt it, and therefore cannot re-key it either. The remedy is deliberately not to copy the operator key onto the box; it is to re-key from the operator workstation, which can read the file. The failure message below says exactly that, because this will happen mid-rebuild when nobody wants to reason it out.

cd "$ADM_ROOT"
f="$ADM_BOX_DIR/${BOX_NAME}.env"
test -f "$f" || { echo "no $f yet"; exit 0; }
if ! grep -q '^sops_version=' "$f"; then
  echo "$f is not encrypted yet; encrypt-env will handle it"
  exit 0
fi
if out="$(sops updatekeys -y "$f" 2>&1)"; then
  printf '%s\n' "$out" | tail -3
else
  printf '%s\n' "$out" | tail -4
  echo
  echo "This box cannot decrypt $f, so it cannot re-key it."
  echo "Expected on a rebuild: the old box key is gone and the new one is not"
  echo "yet a recipient."
  echo
  echo "Run this yourself, at a terminal on the box:"
  echo "    adm unlock"
  echo
  echo "It names the two private keys that can open the file and what each"
  echo "one costs, then takes whichever you have. The operator key is used"
  echo "for one command and never written to disk."
  echo
  echo "The old instruction here was to re-key from the operator workstation"
  echo "instead, which works and needs a second machine with a clone and the"
  echo "operator key -- an undocumented prerequisite in the middle of a"
  echo "restore that the published document claims to be sufficient for."
  exit 1
fi

cd "$ADM_ROOT"
f="$ADM_BOX_DIR/${BOX_NAME}.env"
# The block above exits 0 on a plaintext env, saying encrypt-env will handle
# it; this used to exit 1 on the same file, so the two disagreed about the
# same state and the verify won. On a first run the env is still plaintext
# here -- encrypt-env is four blocks further down -- so there are no
# recipients to compare and nothing this check can say.
#
# Skipping is only honest because the invariant is asserted elsewhere and not
# optionally: encrypt-env's own verify and env-fully-encrypted both fail if
# the file is still plaintext when they run. A skip with no such backstop
# would be this check quietly deciding not to do its job.
if ! grep -q '^sops_version=' "$f"; then
  echo "$f is still plaintext; encrypt-env is what encrypts it, and its own"
  echo "verify is what insists that it did. Nothing to compare yet."
  exit 0
fi
# Count age recipients baked into the file; must match the rule for this box.
in_file=$(grep -cE '^sops_age__list_[0-9]+__map_recipient=' "$f")
in_rule=$(grep -A3 -E "^  - path_regex: \^hosts/${BOX_NAME}/" .sops.yaml \
          | awk -F': ' '/^    age:/ {print $2}' | tr ',' '\n' | grep -c .)
test "$in_file" = "$in_rule" \
  || { echo "$f has $in_file recipients but its rule names $in_rule -- run sops updatekeys"; exit 1; }
echo "$f encrypted to all $in_file recipients named by its rule"

Keep the env file encrypted

This block is the safety net that makes the arrangement survive contact with real use. If someone edits <box>.env in plaintext — entirely natural, since the keys are readable and it looks like an ordinary dotenv file — the next replay re-encrypts it rather than letting the plaintext reach a shared remote.

Guarded on sops_version, the metadata SOPS appends, so an already-encrypted file is left alone rather than double-encrypted.

cd "${ADM_BOX_DIR:?ADM_BOX_DIR unset -- no hosts/<box>/<box>.env found}"
env_file="${BOX_NAME}.env"
test -f "$env_file" || { echo "no $env_file in $ADM_BOX_DIR"; exit 1; }
if ! grep -q "^sops_version=" "$env_file"; then
  echo "$env_file is plaintext; encrypting with sops"
  sops -e -i "$env_file"
else
  echo "$env_file already encrypted"
fi
# `grep -c` exits 1 when the count is zero, and these blocks run under
# pipefail, so on a freshly adopted box this line failed for reporting the
# number 0. Nothing was wrong with the file: adopt writes only the values adm
# must read before sops exists, and all of those are exempt by policy, so
# there was nothing to encrypt and encrypt-env said so by failing.
# [A-Z0-9_], not [A-Z_]. Without the digits this counted 5 of the 7
# encrypted values on the first box it ran on, silently omitting
# PUBLISH_S3_ACCESS_KEY and PUBLISH_S3_SECRET_KEY -- the two most sensitive
# entries in the file, left out because their names contain a 3. A count that
# under-reports exactly the values someone is auditing for is worse than no
# count. Nothing asserted it, which is why it went unnoticed: the assertion
# that every non-exempt key is encrypted belongs to env-fully-encrypted and
# is made per key, by name, so it was never fooled.
n=$(grep -cE "^[A-Z0-9_]+=ENC\[" "$env_file" || true)
echo "encrypted values: $n"
if [[ "$n" == 0 ]]; then
  echo "none yet -- every key in this file is exempt by policy. Expected on a"
  echo "box that has just been adopted and has no credentials yet; the first"
  echo "'sops set' will change it."
fi

cd "${ADM_BOX_DIR:?ADM_BOX_DIR unset}"
env_file="${BOX_NAME}.env"
grep -q "^sops_version=" "$env_file" \
  || { echo "$env_file is NOT encrypted -- secrets would reach the remote in clear"; exit 1; }
# The bootstrap keys must remain readable, or a fresh box cannot start.
grep -qE "^BOX_NAME=${BOX_NAME}$" "$env_file" \
  || { echo "BOX_NAME is not readable in $env_file; bootstrap would break"; exit 1; }
grep -qE "^ADMIN_USER=" "$env_file" \
  || { echo "ADMIN_USER is not readable in $env_file"; exit 1; }
# There is deliberately no "and at least one value must be encrypted" here.
# A box that has just been adopted holds nothing but exempt keys, so such a
# check fails for a file that is exactly as the policy requires.
#
# There used to be one, reading `grep -qE "=ENC\["`, commented "or this is
# theatre". It passed on every file ever written, including an all-exempt one,
# because sops stores its own integrity hash as sops_mac=ENC[AES256_GCM,...].
# The check matched that line and never once looked at a key. Worth keeping as
# a comment: it read as the strictest assertion in the block.
#
# What it was reaching for -- no secret sitting in the clear -- is
# env-fully-encrypted's job, per key, using the same check_env the commit hook
# uses. That one cannot be satisfied by the MAC line.
echo "$env_file: encrypted, bootstrap keys still readable"
echo "encrypted values: $(grep -cE "^[A-Z0-9_]+=ENC\[" "$env_file" || true)"

Unlocking a rebuilt box: adm unlock

A rebuilt machine gets its env from the published document, encrypted, and cannot read a word of it. The file is encrypted to the box key — which went with the old machine — and to the operator key, which this runbook used to forbid copying onto a box. So the document arrived complete and unreadable, and finishing the restore meant a second machine.

adm unlock takes either key, from a person at a terminal. It refuses without one, for the reason env-fill refuses: a key on a command line is in the agent's context, its transcript, shell history and /proc. There is no flag to supply it.

The difference between the two keys is explained where it is asked, not here, because here is not where anyone will be reading it:

• the box key opens this box alone. If you kept a backup of this machine's own key, this is the one to use: the box becomes the box it was, nothing is re-keyed, and no fleet-wide secret touches the disk. It is appended to the keyring rather than replacing what 015 generated, because that generated key is already a recipient of .sops.yaml and of anything written since.
• the operator key opens every box there is or will be. It is passed to one sops updatekeys in that process's environment — never a file — which re-keys the env to this box's current key. When the command returns the operator key is gone and the box reads its own secrets with its own key.

A key that is neither is refused by name, with nothing written. The public half is derived through age-keygen -y - on stdin, so establishing which key you pasted does not require writing it anywhere.

Adding a value: through sops, never to the file on disk

The obvious way to add a key is the wrong one, and it earns a section because the failure is loud in the wrong place and silent in the right one.

Opening hosts/<box>/<box>.env in an editor and typing NEW_KEY=value does not produce a file with one plaintext entry among the ciphertext. It produces a file sops will no longer open at all: on load it tries to decrypt every key not matched by unencrypted_regex, reaches the plaintext one, and gives up.

Error decrypting tree: Error walking tree: Could not decrypt value: Input
string ... does not match sops' data format

So the editor never appears — and the value is sitting in the working tree in the clear, with nothing complaining. That is the wrong moment to find out. The hooks below are the net at commit and push time, but exposure starts when the value is written: anything that reads the working tree in between has already had it, and the only remedy left is to revoke the secret and issue a new one.

Add values through sops, which decrypts to a temporary file, lets you edit that, and re-encrypts on save:

sops hosts/<box>/<box>.env

Or without an editor, keeping the value out of shell history and out of the process list — sops set wants a JSON-encoded value, which is what the printf is for:

read -rs TOK && printf '"%s"' "$TOK" | sops set --value-stdin \
  --input-type dotenv --output-type dotenv \
  hosts/<box>/<box>.env '["NEW_KEY"]'

unset TOK

encrypt-env above would repair such a file on the next replay, and the hooks would refuse to commit it. Neither notices it now, which is what this block is for — it is the only check in this repo that looks at the env file as it currently sits on disk.

Changing the exemption list: adm reencrypt

.sops.yaml's creation_rules apply when a file is created. An existing encrypted file carries its own copy of the policy in sops_unencrypted_regex, and that copy is the one SOPS consults — so editing .sops.yaml changes nothing about a file that already exists, and the two drift apart in silence.

That cost a real bug. ADMIN_UID was added to the list above and then set, and it came back encrypted, because sops set read the stale embedded policy; an encrypted ADMIN_UID is unreadable by 010, which runs before the age key exists. It would have failed on the next rebuild and nowhere before it. What hid it for several commits was that the only key exercising the drift until then was MACHINE_USERS, whose value was the empty string — and SOPS does not encrypt an empty value.

adm reencrypt          # no-op when the policy already matches
adm reencrypt -n       # show what would change
adm reencrypt --force  # rotate the data key regardless

It is a subcommand rather than a recipe in this prose because the manual form has three checks worth doing, and nobody does three checks by hand twice. It decrypts and re-encrypts through a pipe so no plaintext reaches disk, then refuses to replace the file unless the decrypted content is byte-identical, the recipient count is unchanged, and the new file actually records the policy it was meant to adopt. env-fully-encrypted below detects the drift and names this command as the fix, which is the pairing worth having: the check that finds a problem should say what repairs it.

Recipients are a different question and not this command's. That is sops updatekeys, driven by sync-env-recipients above.

cd "$ADM_ROOT"
# Reuse the hooks' own check rather than writing a second one. A divergent
# copy would eventually disagree with the code that actually blocks commits,
# and the weaker of the two would be the one you were trusting.
. hooks/secret-scan.sh
env_file="${ADM_BOX_DIR:?ADM_BOX_DIR unset}/${BOX_NAME}.env"
test -f "$env_file" || { echo "no $env_file"; exit 1; }

check_env "${BOX_NAME}.env" "$(unencrypted_regex)" "$(cat "$env_file")" \
  || { echo "a value in ${BOX_NAME}.env is in the clear on disk right now"
       echo "it was almost certainly added by editing the file instead of"
       echo "going through sops -- see 'Adding a value' in this runbook."
       echo "treat the value as exposed: rotate it, do not just encrypt it."
       exit 1; }
# The file carries its own copy of the exemption policy, and that copy is the
# one sops actually uses. `.sops.yaml`'s creation_rules apply when a file is
# *created*; an existing encrypted file is governed by its embedded
# sops_unencrypted_regex. So editing .sops.yaml changes nothing about a file
# that already exists, and the two can disagree silently.
#
# That is not theoretical. Adding ADMIN_UID to .sops.yaml and then setting it
# encrypted the value, because `sops set` consulted the stale embedded regex --
# and an encrypted ADMIN_UID is unreadable by 010, which runs before the age
# key exists. It looked like a working change for as long as nobody looked.
embedded="$(grep -m1 '^sops_unencrypted_regex=' "$env_file" | cut -d= -f2-)"
declared="$(awk -F\' '/^[[:space:]]*unencrypted_regex:/ { print $2; exit }' .sops.yaml)"
test -n "$embedded" || { echo "$env_file has no embedded unencrypted_regex"; exit 1; }
if [[ "$embedded" != "$declared" ]]; then
  echo "$env_file was encrypted under a different exemption policy:"
  echo "  embedded in the file: $embedded"
  echo "  declared in .sops.yaml: $declared"
  echo "sops will use the embedded one, so a key added now may be encrypted"
  echo "against the rules. Re-encrypt the file under the current policy:"
  echo "  adm reencrypt"
  exit 1
fi

echo "${BOX_NAME}.env: every key is encrypted or deliberately exempt"
echo "exemption policy matches .sops.yaml: $declared"

Keys that are misnamed, and keys nothing reads

Two ways an env file can be wrong while looking right, both silent until now.

A key that is not a shell identifier is unusable, not merely ugly. It is accepted by the file, exported into every block, and unreachable from all of them: bash reads ${my-key:-x} as ${my} with a default, so the value is simply never seen. Tested rather than assumed — the key arrives in the environment and the block cannot name it.

A key nothing in the repo reads is almost always a typo. This is the nastier of the two, because the symptom points at the wrong thing. Write ADMIN_USR and 010 fails with ADMIN_USER unset while the env plainly contains a key that looks like it was set — so the failure reads as a bug in the runbook rather than in what was typed.

adm env-check reports both, and suggests the near match when there is one, which is what turns the second case from a puzzle into a correction.

Its notion of "read by something" is deliberately loose: the key name appearing anywhere in bin/adm, a runbook, a hook or the top-level documents is enough. A key mentioned in prose is a key somebody knows about, and the alternative — tracking which keys are used — would flag every value that is only read on a box with a feature this one does not have.

cd "$ADM_ROOT"
adm env-check

Git hooks: refuse to commit or push secrets in the clear

encrypt-env above repairs a plaintext env file on the next replay, which is the wrong moment — by then it may already be pushed. Two hooks move the check earlier.

pre-commit catches the mistake at the cheapest possible point, against the staged content.

pre-push closes what pre-commit structurally cannot: commits made with --no-verify, commits made before the hooks were armed, and commits that arrived from another clone. It scans the commit range actually being pushed — including blobs added and later removed inside that range, because those still land in the remote's history, where no later cleanup reaches them.

Both source a single library, hooks/secret-scan.sh, so there is one definition of what counts as a secret. Two copies of a security check drift, and the copy that drifts is always the one nobody is watching.

All three files are written by the blocks below rather than merely referenced, because a runbook that points at a file in its own repo is not reproducible from the documentation — you would need this repo to rebuild it, which is the opposite of the point. The files in hooks/ are materialised consequences, tracked so they are reviewable and active in a fresh clone, and labelled as overwritten on replay.

They are tracked rather than left in .git/hooks/, which is not version-controlled: a hook there would be missing from every clone, absent exactly when a new box is set up by someone who has not read this file. core.hooksPath must still be set locally — git deliberately offers no way for a repository to arm its own hooks, since cloning would then run the author's code.

The shared library

cd "$ADM_ROOT"
install -d -m 0755 hooks
install -m 0644 /dev/stdin hooks/secret-scan.sh <<'LIB'
# Managed by runbooks/015-secrets.md -- hand edits are overwritten on replay.
#
# Sourced by hooks/pre-commit and hooks/pre-push. Defines checks and runs
# nothing by itself. Self-test: bash hooks/secret-scan.sh --self-test

SOPS_YAML=".sops.yaml"

# Bracketed characters so these patterns match a real key but NOT this line,
# which is what lets this file be committed at all. See the runbook.
KEY_PATTERNS='AGE-SECRET-KEY-[1]|BEGIN OPENSSH PRIVATE [K]EY|BEGIN [A-Z ]*PRIVATE KEY'

warn() { printf '%s\n' "$1" >&2; }

# Keys .sops.yaml deliberately leaves readable. Read, never duplicated, so
# the hooks cannot drift from the encryption rules.
unencrypted_regex() {
  awk -F"'" '/^[[:space:]]*unencrypted_regex:/ { print $2; exit }' \
      "$SOPS_YAML" 2>/dev/null
}

# check_env <label> <exempt-regex> <content> -> 0 clean, 1 problems
# A plaintext env holding nothing but exempt keys. `adm adopt` writes exactly
# this -- it must, because 015 has not run and nothing on the box can encrypt
# yet -- and the hooks that would object are not installed until 015 either,
# so committing adoption is both the documented next step and permanent. The
# blob then failed the scan on every publish thereafter, and the remedy was to
# allow it by sha, one box at a time, for three values the policy itself calls
# not-secret.
#
# Judged by content, not by sha: $ure *is* this repository's definition of
# what is not a secret, so a file containing only keys it names leaks nothing
# by that same definition. Conservative in two ways that matter. Any key the
# policy does not name fails, encrypted or not; and a single ENC[ value makes
# the file a sops file with its metadata missing rather than a plaintext one,
# which stays a failure -- without the MAC there is nothing to trust.
plain_env_all_exempt() {
  local ure="$1" content="$2" line key
  [[ -n "$ure" ]] || return 1
  grep -q '^sops_version=' <<<"$content" && return 1
  grep -q '=ENC\[' <<<"$content" && return 1
  while IFS= read -r line; do
    case "$line" in ''|'#'*) continue ;; esac
    [[ "$line" == *=* ]] || continue
    key="${line%%=*}"
    case "$key" in sops_*) continue ;; esac
    grep -qE "$ure" <<<"$key" || return 1
  done <<<"$content"
  return 0
}

check_env() {
  local label="$1" ure="$2" content="$3" problems=0 line key val
  if ! grep -q '^sops_version=' <<<"$content"; then
    if plain_env_all_exempt "$ure" "$content"; then
      return 0
    fi
    warn "  $label: not SOPS-encrypted (no sops_version line)"
    return 1
  fi
  while IFS= read -r line; do
    case "$line" in ''|'#'*) continue ;; esac
    [[ "$line" == *=* ]] || continue
    key="${line%%=*}"; val="${line#*=}"
    case "$key" in sops_*) continue ;; esac
    if [[ -n "$ure" ]] && grep -qE "$ure" <<<"$key"; then continue; fi
    if [[ "$val" != ENC\[* ]]; then
      warn "  $label: '$key' is not encrypted"
      problems=1
    fi
  done <<<"$content"
  return "$problems"
}

# scan_blob <label> <exempt-regex> <content> -> 0 clean, 1 problems
scan_blob() {
  local label="$1" ure="$2" content="$3" rc=0
  case "$label" in
    *.env) check_env "$label" "$ure" "$content" || rc=1 ;;
  esac
  if grep -qE "$KEY_PATTERNS" <<<"$content"; then
    warn "  $label: contains private key material"
    rc=1
  fi
  return "$rc"
}

# Two causes, two different fixes, and reaching for the wrong one is worse
# than doing nothing: encrypt-env on a hand-edited file encrypts the value
# that was already exposed and makes the problem look solved.
refuse() {
  warn ""
  warn "pre-$1: refusing to $1 secrets in the clear."
  warn ""
  warn "  a key added by hand to an encrypted file -- the usual cause:"
  warn "    delete that line, then re-add the value through sops, which"
  warn "    encrypts on save:   sops hosts/<box>/<box>.env"
  warn "    NOT encrypt-env -- it would encrypt what you already exposed."
  warn ""
  warn "  a file that was never encrypted at all, no sops_version line:"
  warn "    adm run 015-secrets.md --only encrypt-env"
  warn ""
  warn "  either way, a value that has been on disk in plaintext is exposed."
  warn "  rotate it. encrypting it afterwards hides it without un-exposing it."
  warn ""
  warn "  inspect:  sops -d --output-type dotenv hosts/<box>/<box>.env"
  warn "  override: ${REFUSE_OVERRIDE:-git $1 --no-verify}"
  warn ""
}

self_test() {
  local ure='^(BOX_NAME|ADMIN_USER)$' fails=0
  local ok='BOX_NAME=b
ADMIN_USER=a
GIT_REMOTE=ENC[AES256_GCM,data:x,type:str]
sops_version=3.13.3'
  local plain_value='BOX_NAME=b
GIT_REMOTE=ssh://git@example/r.git
sops_version=3.13.3'
  local no_meta='BOX_NAME=b
GIT_REMOTE=ENC[AES256_GCM,data:x,type:str]'
  # What adopt writes, before anything on the box can encrypt.
  local adopted='# written by adm adopt
BOX_NAME=b
ADMIN_USER=a'
  # Same shape, one key the policy does not name.
  local adopted_plus='BOX_NAME=b
ADMIN_USER=a
FORGE_TOKEN=glpat-deadbeef'
  # Only exempt keys, but one value is ciphertext and the metadata is gone:
  # a sops file that lost its MAC, not a plaintext one. The exempt-key test
  # alone would call this benign, so this is what makes the ENC[ guard in
  # plain_env_all_exempt load-bearing rather than decorative.
  local exempt_but_enc='BOX_NAME=b
ADMIN_USER=ENC[AES256_GCM,data:x,type:str]'

  scan_blob "a.env" "$ure" "$ok" 2>/dev/null \
    || { warn "self-test FAILED: encrypted env rejected"; fails=1; }
  scan_blob "a.env" "$ure" "$plain_value" 2>/dev/null \
    && { warn "self-test FAILED: plaintext secret accepted"; fails=1; }
  scan_blob "a.env" "$ure" "$no_meta" 2>/dev/null \
    && { warn "self-test FAILED: env without sops metadata accepted"; fails=1; }
  scan_blob "notes.txt" "$ure" "$(printf 'AGE-SECRET-KEY-%s' 1ABC)" 2>/dev/null \
    && { warn "self-test FAILED: private key accepted"; fails=1; }
  scan_blob "notes.txt" "$ure" "harmless text" 2>/dev/null \
    || { warn "self-test FAILED: innocent file rejected"; fails=1; }
  scan_blob "a.env" "$ure" "$adopted" 2>/dev/null \
    || { warn "self-test FAILED: plaintext env of only exempt keys rejected"; fails=1; }
  scan_blob "a.env" "$ure" "$adopted_plus" 2>/dev/null \
    && { warn "self-test FAILED: plaintext env with a non-exempt key accepted"; fails=1; }
  scan_blob "a.env" "$ure" "$exempt_but_enc" 2>/dev/null \
    && { warn "self-test FAILED: ciphertext without sops metadata accepted"; fails=1; }

  (( fails )) && { warn "secret-scan self-test FAILED"; return 1; }
  echo "secret-scan self-test ok (8 cases: encrypted, plaintext, no-metadata, key, innocent, all-exempt, all-exempt+secret, exempt-but-enc)"
}

# This file is sourced by the three hooks, and is also run directly: 015
# checks it with `bash hooks/secret-scan.sh --self-test`. That did nothing
# whatsoever until this line existed -- the file defined self_test, reached
# the end, and exited 0 -- so the check read as a test and was not one. The
# suite was covered anyway, because each hook carries the same dispatcher and
# those do run it; what was wrong was only the claim about where.
#
# Guarded on being executed rather than sourced, so sourcing this from a hook
# that was itself invoked with --self-test does not run the suite twice.
if [[ "${BASH_SOURCE[0]}" == "$0" && "${1:-}" == --self-test ]]; then
  self_test
  exit $?
fi
LIB
bash hooks/secret-scan.sh --self-test

cd "$ADM_ROOT"
test -f hooks/secret-scan.sh || { echo "hooks/secret-scan.sh missing"; exit 1; }
bash hooks/secret-scan.sh --self-test

pre-commit

cd "$ADM_ROOT"
install -m 0755 /dev/stdin hooks/pre-commit <<'PRECOMMIT'
#!/usr/bin/env bash
# Managed by runbooks/015-secrets.md -- hand edits are overwritten on replay.
# Scans STAGED content, which differs from the working tree exactly when
# someone has partially staged an edit.
set -uo pipefail
. "$(dirname "$0")/secret-scan.sh"
[[ "${1:-}" == --self-test ]] && { self_test; exit $?; }

rc=0
ure="$(unencrypted_regex)"
while IFS= read -r f; do
  [[ -n "$f" ]] || continue
  content="$(git show ":$f" 2>/dev/null)" || continue
  scan_blob "$f" "$ure" "$content" || rc=1
done < <(git diff --cached --name-only --diff-filter=ACM)

(( rc )) && refuse commit
exit "$rc"
PRECOMMIT
hooks/pre-commit --self-test

cd "$ADM_ROOT"
test -x hooks/pre-commit || { echo "hooks/pre-commit is not executable"; exit 1; }
hooks/pre-commit --self-test

pre-push

Reads the refs being pushed from stdin, in git's documented <local-ref> <local-sha> <remote-ref> <remote-sha> format. An all-zero remote-sha means a branch the remote does not have yet, so the range becomes "everything not already on a remote"; an all-zero local-sha is a ref deletion, which pushes no content and is skipped.

Blobs are deduplicated by hash, so a file touched in thirty commits is read once.

cd "$ADM_ROOT"
install -m 0755 /dev/stdin hooks/pre-push <<'PREPUSH'
#!/usr/bin/env bash
# Managed by runbooks/015-secrets.md -- hand edits are overwritten on replay.
#
# Scans every blob introduced by the commits being pushed -- including ones
# added and later removed within the range, since those still reach the
# remote's history. Closes the gap pre-commit cannot: --no-verify, commits
# predating the hooks, and commits from another clone.
set -uo pipefail
. "$(dirname "$0")/secret-scan.sh"
[[ "${1:-}" == --self-test ]] && { self_test; exit $?; }

rc=0
ure="$(unencrypted_regex)"
declare -A seen=()

while read -r _local_ref local_sha _remote_ref remote_sha; do
  [[ -n "${local_sha:-}" ]] || continue
  [[ "$local_sha" =~ ^0+$ ]] && continue          # ref deletion: no content
  # A remote sha this clone does not have means the ranges cannot be computed:
  # a new branch, or a force-push over history that was rewritten underneath
  # it. The first version let `git rev-list` fail, scanned the empty list it
  # produced, and allowed the push -- silently, and in exactly the case where
  # scanning matters most. Fall back to everything reachable from the tip.
  if [[ "${remote_sha:-}" =~ ^0+$ ]] \
     || ! git cat-file -e "${remote_sha}^{commit}" 2>/dev/null; then
    mapfile -t commits < <(git rev-list "$local_sha")
  else
    mapfile -t commits < <(git rev-list "$remote_sha..$local_sha")
  fi
  (( ${#commits[@]} )) || { echo "pre-push: no commits resolved; refusing" >&2; exit 1; }
  for c in "${commits[@]}"; do
    [[ -n "$c" ]] || continue
    while IFS= read -r f; do
      [[ -n "$f" ]] || continue
      blob="$(git rev-parse "$c:$f" 2>/dev/null)" || continue
      [[ -n "${seen[$blob]:-}" ]] && continue
      seen[$blob]=1
      content="$(git cat-file blob "$blob" 2>/dev/null)" || continue
      scan_blob "$f" "$ure" "$content" || rc=1
    done < <(git diff-tree --no-commit-id --name-only -r "$c")
  done
done

(( rc )) && refuse push
exit "$rc"
PREPUSH
hooks/pre-push --self-test

cd "$ADM_ROOT"
test -x hooks/pre-push || { echo "hooks/pre-push is not executable"; exit 1; }
hooks/pre-push --self-test

scan-history

pre-push gates what leaves for the forge. A published git bundle leaves by a different door: adm publish reads the object store's credentials from the env file and uploads directly, so nothing about a push is involved and the pre-push hook never runs. Someone who commits a secret and publishes without pushing would export it, and a bundle carries all history -- including a blob added and removed again, which stays reachable from the commit that introduced it.

So the same library gets a third caller, scanning everything a bundle would carry rather than one push's worth of commits.

That difference in scope brings a problem the push hook never has to face. unencrypted_regex is the list of names exempt from encryption, and it changes as the repo learns what it needs -- MACHINE_USERS was called HUMAN_USERS for a while, and before the move to hosts/ the env file was a template at the repo root. pre-push only ever scans the commits in one push, so older history is grandfathered by the simple fact of never being looked at again. A bundle ships all of it, so every past commit gets judged by today's list, and a variable renamed years later fails retroactively. Rewriting history to satisfy a policy it predates is not a reasonable answer.

The answer is an explicit list of blobs already examined and found harmless, each with its reason. It is keyed by blob sha, which is the safe key: a sha is its content, so an approved entry cannot later be made to hold something else. Allowing a path would grant a hiding place; allowing a sha grants exactly one known string of bytes. Adding an entry means editing this runbook and saying why, which is the point -- the judgment ends up written down next to the rule it bends, not buried in a config file.

The answer is an explicit list of blobs already examined and found harmless, each with its reason. It is keyed by blob sha, which is the safe key: a sha is its content, so an approved entry cannot later be made to hold something else. Allowing a path would grant a hiding place; allowing a sha grants exactly one known string of bytes.

That list is per box, at hosts/<box>/scan-history.allow, and it is written by hand rather than by a block here. It records judgments about this repository's history, which no other machine shares — a fork starts with none and accumulates its own. It lived in this runbook for a while, with two blob shas from one repository's history inside it, which made a generic runbook carry a single box's data: exactly what AGENTS.md forbids, and nothing was checking.

Hand-written, like each box's AS-FOUND.md and for the same reason: it is a record of decisions a person made, not a state a machine can converge on. Replaying this runbook must not invent or discard one.

cd "$ADM_ROOT"
# No assertion that the file exists -- a repo whose history is clean has
# nothing to allow, and that is the better state to be in. What is asserted
# is that any entry is a full 40-character sha. A short one would still match
# the grep in scan-history, but as a prefix, and a prefix allows more than the
# single blob that was actually reviewed.
bad=0; files=0
for f in hosts/*/scan-history.allow; do
  [ -e "$f" ] || continue
  files=$((files+1))
  while read -r sha _; do
    case "$sha" in ''|'#'*) continue ;; esac
    [[ "$sha" =~ ^[0-9a-f]{40}$ ]] || { echo "$f: not a full blob sha: $sha"; bad=1; }
  done < "$f"
done
test "$bad" = 0 || exit 1
# And that no copy was left behind at the old generic path, which would be
# read by nothing and would quietly stop being maintained.
test ! -e hooks/scan-history.allow \
  || { echo "hooks/scan-history.allow still exists; it is now per-box"; exit 1; }
echo "$files per-box allowlist(s), every entry a full sha"

cd "$ADM_ROOT"
install -m 0755 /dev/stdin hooks/scan-history <<'SCANALL'
#!/usr/bin/env bash
# Managed by runbooks/015-secrets.md -- hand edits are overwritten on replay.
#
# Scans every blob reachable from every ref: the whole of what `git bundle
# --all` would carry. Not a git hook -- git has no "before you publish" event
# -- so `adm publish` calls it directly, and it fails closed.
set -uo pipefail
. "$(dirname "$0")/secret-scan.sh"
[[ "${1:-}" == --self-test ]] && { self_test; exit $?; }

# Per box, because these are judgments about one repository's history and no
# other machine shares them. Concatenated rather than chosen: a repo has one
# adopted box, and reading them all avoids needing to know which.
allowed() { cat hosts/*/scan-history.allow 2>/dev/null; }
# There is no --no-verify to suggest here; publish is not a git command.
REFUSE_OVERRIDE="review the blob, then add its sha to hooks/scan-history.allow
            via runbooks/015-secrets.md#scan-history-allow"

rc=0
ure="$(unencrypted_regex)"
declare -A used=()

# rev-list --objects prints "<sha> <path>", and the path may contain spaces,
# so the path is taken as the rest of the line rather than as a field.
while read -r blob path; do
  [[ -n "$blob" && -n "$path" ]] || continue
  content="$(git cat-file blob "$blob" 2>/dev/null)" || continue
  out="$(scan_blob "$path" "$ure" "$content" 2>&1)" && continue
  if allowed | grep -qE "^$blob([[:space:]]|\$)"; then
    used[$blob]=1
    continue
  fi
  printf '%s\n' "$out" >&2
  # The sha is the first thing anyone needs: it is how you read the blob and
  # how you would allow it. scan_blob reports a path, and a path in history
  # is ambiguous across commits.
  warn "    blob $blob -- read it with: git cat-file blob $blob"
  rc=1
done < <(
  git rev-list --objects --all \
    | git cat-file --batch-check='%(objectname) %(objecttype) %(rest)' \
    | sed -n 's/^\([0-9a-f]\{7,\}\) blob \(..*\)$/\1 \2/p' \
    | sort -u
)

# An entry that matches nothing is either a typo or a leftover from rewritten
# history. Reported, not fatal: a stale allowance grants nothing, but a list
# nobody prunes is a list nobody reads.
while read -r sha _; do
  case "$sha" in ''|'#'*) continue ;; esac
  [[ -n "${used[$sha]:-}" ]] || warn "note: allowance matches no blob in history: $sha"
done < <(allowed)

(( rc )) && refuse publish
exit "$rc"
SCANALL
hooks/scan-history --self-test

cd "$ADM_ROOT"
test -x hooks/scan-history || { echo "hooks/scan-history is not executable"; exit 1; }
hooks/scan-history --self-test
# The point of the thing: it must pass over this repo's own history. If this
# fails, something in a past commit is in the clear and no bundle may ship.
hooks/scan-history || { echo "history contains a secret in the clear"; exit 1; }

Arm them

cd "$ADM_ROOT"
git config core.hooksPath hooks
git config --get core.hooksPath

cd "$ADM_ROOT"
test "$(git config core.hooksPath)" = "hooks" \
  || { echo "core.hooksPath is '$(git config core.hooksPath)', expected 'hooks'"; exit 1; }
# Git tracks the executable bit; losing it would hand a clone a dead hook.
for spec in hooks/pre-commit:100755 hooks/pre-push:100755 hooks/secret-scan.sh:100644; do
  f="${spec%%:*}"; want="${spec##*:}"
  mode="$(git ls-files -s "$f" | awk '{print $1}')"
  case "$mode" in
    "$want") : ;;
    "")      echo "$f is not tracked yet -- commit it so clones get it"; exit 1 ;;
    *)       echo "$f is committed as $mode, not $want"; exit 1 ;;
  esac
done
echo "hooks armed: pre-commit, pre-push, shared library, all tracked"

The detector cannot contain its own patterns

The first version of this refused to commit itself, and correctly so: its KEY_PATTERNS spelled the age-key prefix out in full, so the hook's own source — and the runbook containing it — looked exactly like a leaked key.

The tempting fix is to exempt hooks/ and runbooks/ from the key scan. That is a loophole: a real key pasted into a runbook is a plausible mistake, and would then sail through.

The actual fix is to write each pattern so it matches a real key but not its own text, by bracketing one character:

AGE-SECRET-KEY-[1]           matches a real age key, not this line
BEGIN OPENSSH PRIVATE [K]EY
BEGIN [A-Z ]*PRIVATE KEY     already self-safe: '[' is not in [A-Z ]

The self-test builds its sample with printf for the same reason. This is a general hazard for any secret scanner kept in the repository it scans, and it is why the third pattern needed no change while the first two did.

Checking it yourself

--self-test exercises the logic against inline samples, but it does not prove git calls the hooks. These do, and leave nothing behind. From the repo root:

# pre-commit
printf 'BOX_NAME=test\nAPI_TOKEN=hunter2\n' > test.env
git add test.env && git commit -m 'should be refused'      # must fail
git reset && rm -f test.env

# pre-push: commit past the hook deliberately, then try to push
printf 'AGE-SECRET-KEY-%s\n' 1QQQ > leaked-key.txt
git add leaked-key.txt
git commit --no-verify -m 'bypassed pre-commit'
git push                                                   # must fail
git reset --hard HEAD~1 && rm -f leaked-key.txt

The second case is the one worth running: it reproduces the exact situation pre-push exists for — a secret that got past pre-commit because someone used --no-verify.

test.env deliberately sits at the repo root rather than in a box directory, so adm cannot mistake it for a real box while the test is in progress.

Limits

• Neither sees the working tree. Both run at commit or at push time, so a plaintext value sitting in <box>.env right now is invisible to them until someone tries to commit it — and the exposure began when it was written, not when it was committed. env-fully-encrypted above closes that gap, on every adm verify.
• --no-verify bypasses both, deliberately. A check that cannot be overridden becomes one people disable permanently; one that can be is a speed bump in the right place. pre-push is what makes a bypassed pre-commit survivable.
• Server-side is still unguarded. Both hooks run on this box, so a push from a different clone with hooks unarmed is unchecked. Only a forge-side pre-receive hook closes that, and we do not control the forge's hooks from here.
• Neither can retract history. Anything already committed in the clear stays in the repository — see DECISIONS.md #004, which is why this repo is shareable but not publishable.

Prove a round trip

Configuration asserts the setup; this proves the thing that matters — that a secret written in the repo comes back out intact on this box.

cd "${ADM_BOX_DIR:?ADM_BOX_DIR unset}"
got="$(sops -d --input-type dotenv --output-type dotenv "${BOX_NAME}.env" \
       | awk -F= '$1 == "GIT_REMOTE" { print $2 }')"
test -n "$got" || { echo "GIT_REMOTE did not decrypt"; exit 1; }
case "$got" in
  ssh://*) : ;;
  *) echo "GIT_REMOTE decrypted to something unexpected"; exit 1 ;;
esac
# And adm's own loader must agree, since that is what blocks actually see.
test -n "${GIT_REMOTE:-}" \
  || { echo "adm did not export GIT_REMOTE into this block"; exit 1; }
test "$GIT_REMOTE" = "$got" \
  || { echo "adm exported a different GIT_REMOTE than sops decrypts"; exit 1; }
echo "secret round trip ok: sops and adm agree on GIT_REMOTE"

What this does and does not protect

Does: the repo can be pushed to a shared or public remote without revealing the encrypted values. History is reviewable, because keys stay readable and only values are opaque.

Does not: protect the secrets from anyone with access to this box. The age key is on it, in plaintext, by design — adm replay has to work unattended. The deploy key in $HOME/.ssh is plaintext for the same reason. Encrypting <box>.env changes the repo's exposure, not the box's.

Does not, on its own, make this repo shareable. Encrypting one file is necessary and nowhere near sufficient — see DECISIONS.md #004 for the rest of what leaks, notably hosts/<box>/facts/, which records internal addresses, listening sockets and account names in plain text.

020 — Off-box remote for this repo

Required — every box runs this.

Give /opt/admin somewhere to live that isn't the box it documents.

Until this runbook ran, the only copy of this repo's history was on rea-dev itself, which defeats most of the purpose: the rebuild procedure was stored exclusively on the machine it exists to rebuild.

The remote is a Forgejo instance, prepared in advance. Its URL is $GIT_REMOTE from <box>.env — an encrypted value, because it names a private forge and account and this repo is meant to be shareable. So this runbook depends on 015-secrets.md having made decryption work; without it every block here fails fast with a clear message rather than operating on an empty URL.

Host and port are derived from that URL rather than stored separately, so there is one source of truth and nothing to drift.

Why a per-repo deploy key, owned by admin

The key lives at $HOME/.ssh/id_ed25519_admin_repo — named for its one job rather than called id_ed25519, because it is a deploy key scoped to a single repository, not admin's general-purpose identity. If this box ever needs to reach a second repo on the same forge, it gets its own key and its own Host alias; nothing here has to be untangled first.

Root holds no key and no SSH config, and must not acquire either — see 010-admin-user.md. Outbound SSH from this box is admin's business.

The key has no passphrase. Pushes have to work unattended, and a passphrase stored anywhere the box can read it is not a passphrase. The blast radius is bounded by what the key can reach: write access to this one repository. That is a real exposure — someone with the key can rewrite the box's own documentation — and it is accepted deliberately rather than by omission.

The comment must be a single token

A public key line is <type> <base64> <comment>, and Forgejo's deploy-key form splits it on whitespace. A descriptive, multi-word comment produces an eight-field line, which sends its parser down a path where it tries to base64-decode part of the comment and fails:

Cannot verify your SSH key: extractTypeFromBase64Key:
invalid key format: illegal base64 data at input byte 6

The first version of this block used a multi-word description naming the repo and path — informative in ssh-keygen -l output, and rejected by the only system that had to accept it. Keep the comment to one token; the runbook is where the explanation belongs.

The comment is converged, not just set at creation: if an existing key carries the wrong one, the block repairs it in place rather than leaving a key whose public line cannot be pasted anywhere. ssh-keygen -c rewrites the comment without touching the keypair, so the fingerprint is stable and the forge needs no update.

key="$HOME/.ssh/id_ed25519_admin_repo"
comment="${ADMIN_USER:-admin}@${BOX_NAME}"
install -d -m 0700 "$HOME/.ssh"
if [[ ! -f "$key" ]]; then
  ssh-keygen -t ed25519 -N "" -f "$key" -C "$comment"
  fresh=1
fi
current="$(awk '{ $1=""; $2=""; sub(/^ +/, ""); print }' "$key.pub")"
if [[ "$current" != "$comment" ]]; then
  echo "repairing key comment: '$current' -> '$comment'"
  ssh-keygen -c -P "" -C "$comment" -f "$key" >/dev/null
fi
chmod 0600 "$key"
chmod 0644 "$key.pub"
ssh-keygen -lf "$key.pub"
# Only when it was just generated. On every other run the key has long since
# been authorised and printing it again would be noise; this is the one
# moment someone has to act on it.
if [[ "${fresh:-}" == 1 ]]; then
  echo
  echo "This is a new key, so it is not authorised on the forge yet."
  echo "The forge-authorized check below prints it with the URL to add it to."
fi

key="$HOME/.ssh/id_ed25519_admin_repo"
test "$(stat -c '%a %U:%G' "$key")" = "600 $ADMIN_USER:$ADMIN_USER"
test "$(stat -c '%a' "$key.pub")" = "644"
# Passphrase-less by design: an empty passphrase must open it non-interactively.
ssh-keygen -y -P "" -f "$key" >/dev/null \
  || { echo "key is passphrase-protected; unattended push will hang"; exit 1; }
# Exactly three fields, or the forge's deploy-key form rejects the paste.
nf="$(awk '{print NF}' "$key.pub")"
test "$nf" -eq 3 \
  || { echo "public key line has $nf fields, expected 3 -- comment must be one token"; exit 1; }
ssh-keygen -lf "$key.pub"

Pin the forge's host key

Pinned as $FORGE_HOST_FINGERPRINT in <box>.env, captured from this box on 2026-10-18. That value is deliberately not encrypted: a public host key fingerprint is not a secret, and is useless without knowing the host — which is. Worth knowing before you go looking for one: this forge offers only an RSA host key, no ed25519.

This fingerprint was obtained by ssh-keyscan, which is trust-on-first-use: it proves only that something answering on that port presented this key at that moment. Because the same person administers the forge, it should be cross-checked out of band against the server's own /etc/ssh/ssh_host_rsa_key.pub — until then the pin records continuity, not authenticity.

Pinning it in the runbook rather than accepting whatever turns up means a rebuild, or a replay after a MITM, fails loudly instead of silently trusting a new key.

# Everything host-specific comes from <box>.env; the remote is an encrypted
# value, so 015 must have run.
url="${GIT_REMOTE:?GIT_REMOTE unset -- run 015-secrets.md so secrets decrypt}"
hp="${url#ssh://}"; hp="${hp#*@}"; hp="${hp%%/*}"
host="${hp%%:*}"; port="${hp##*:}"
[[ "$port" == "$host" ]] && port=22
kh="$HOME/.ssh/known_hosts"
expected="${FORGE_HOST_FINGERPRINT:?FORGE_HOST_FINGERPRINT unset in <box>.env}"
install -d -m 0700 "$HOME/.ssh"
touch "$kh"; chmod 0600 "$kh"
if ! ssh-keygen -F "[$host]:$port" -f "$kh" >/dev/null; then
  tmp="$(mktemp)"
  ssh-keyscan -p "$port" "$host" > "$tmp" 2>/dev/null
  test -s "$tmp" || { echo "keyscan returned nothing for $host:$port"; rm -f "$tmp"; exit 1; }
  if ! ssh-keygen -lf "$tmp" | awk '{print $2}' | grep -qx "$expected"; then
    echo "HOST KEY MISMATCH -- refusing to trust $host:$port"
    echo "expected: $expected"
    echo "got:"; ssh-keygen -lf "$tmp" | sed 's/^/  /'
    rm -f "$tmp"; exit 1
  fi
  cat "$tmp" >> "$kh"; rm -f "$tmp"
fi
ssh-keygen -F "[$host]:$port" -f "$kh" -l

# Everything host-specific comes from <box>.env; the remote is an encrypted
# value, so 015 must have run.
url="${GIT_REMOTE:?GIT_REMOTE unset -- run 015-secrets.md so secrets decrypt}"
hp="${url#ssh://}"; hp="${hp#*@}"; hp="${hp%%/*}"
host="${hp%%:*}"; port="${hp##*:}"
[[ "$port" == "$host" ]] && port=22
kh="$HOME/.ssh/known_hosts"
expected="${FORGE_HOST_FINGERPRINT:?FORGE_HOST_FINGERPRINT unset in <box>.env}"
ssh-keygen -F "[$host]:$port" -f "$kh" -l | awk '{print $3}' \
  | grep -qx "$expected" \
  || { echo "pinned host key for the forge is missing or changed"; exit 1; }
echo "forge host key pinned and unchanged"

SSH config for admin

The remote URL already carries the user and port, so this stanza's User and Port are belt and braces. Its real job is IdentityFile plus IdentitiesOnly, so git presents this one deploy key and nothing else.

Written whole from a heredoc, never appended — per AGENTS.md, an appending block duplicates its own output on the second run.

# Everything host-specific comes from <box>.env; the remote is an encrypted
# value, so 015 must have run.
url="${GIT_REMOTE:?GIT_REMOTE unset -- run 015-secrets.md so secrets decrypt}"
hp="${url#ssh://}"; hp="${hp#*@}"; hp="${hp%%/*}"
host="${hp%%:*}"; port="${hp##*:}"
[[ "$port" == "$host" ]] && port=22
user="${url#ssh://}"; user="${user%%@*}"
install -d -m 0700 "$HOME/.ssh"
install -m 0600 /dev/stdin "$HOME/.ssh/config" <<EOF
# Managed by $ADM_ROOT/runbooks/020-admin-repo-remote.md
# Hand edits are overwritten on the next \`adm replay\`.

Host $host
    Port $port
    User $user
    IdentityFile ~/.ssh/id_ed25519_admin_repo
    IdentitiesOnly yes
EOF
cat "$HOME/.ssh/config"

# Everything host-specific comes from <box>.env; the remote is an encrypted
# value, so 015 must have run.
url="${GIT_REMOTE:?GIT_REMOTE unset -- run 015-secrets.md so secrets decrypt}"
hp="${url#ssh://}"; hp="${hp#*@}"; hp="${hp%%/*}"
host="${hp%%:*}"; port="${hp##*:}"
[[ "$port" == "$host" ]] && port=22
test "$(stat -c '%a %U:%G' "$HOME/.ssh/config")" = "600 $ADMIN_USER:$ADMIN_USER"
g="$(ssh -G "$host" 2>/dev/null)"
grep -qx "port $port" <<<"$g" || { echo "port $port not applied"; exit 1; }
grep -q "^identityfile.*id_ed25519_admin_repo" <<<"$g" \
  || { echo "deploy key not selected for the forge"; exit 1; }
echo "ssh config resolves correctly for the forge"

The git remote

cd "$ADM_ROOT"
url="${GIT_REMOTE:?GIT_REMOTE unset -- run 015-secrets.md so secrets decrypt}"
if git remote get-url origin >/dev/null 2>&1; then
  git remote set-url origin "$url"
else
  git remote add origin "$url"
fi
git remote -v

cd "$ADM_ROOT"
test "$(git remote get-url origin)" = "${GIT_REMOTE:?GIT_REMOTE unset}" \
  || { echo "origin does not match GIT_REMOTE from <box>.env"; exit 1; }
echo "origin matches GIT_REMOTE"

The manual step: authorise the deploy key

This is the one step in this repo that a human must perform by hand, and it cannot be automated away without handing the box credentials to the forge's API — a strictly larger secret than the key it would install.

Add the public key printed by admin-deploy-key as a deploy key on the forge. For Forgejo and Gitea the page is https://<host>/<org>/<repo>/settings/keys; the forge-authorized verify below prints the exact URL for this box, derived from $GIT_REMOTE, so the address is not written into a shareable file.

Enable write access. Forgejo and Gitea create deploy keys read-only by default, and a read-only key fails at git push with a message about the repository not being found rather than anything about permissions — which wastes a surprising amount of time.

The verify below fails until that is done. That is correct behaviour, not a defect: like reboot-needed in 000, it is an honest report of something untrue.

cd "$ADM_ROOT"
# Everything host-specific comes from <box>.env; the remote is an encrypted
# value, so 015 must have run.
url="${GIT_REMOTE:?GIT_REMOTE unset -- run 015-secrets.md so secrets decrypt}"
hp="${url#ssh://}"; hp="${hp#*@}"; hp="${hp%%/*}"
host="${hp%%:*}"; port="${hp##*:}"
[[ "$port" == "$host" ]] && port=22
path="${url#ssh://*@"$hp"/}"; path="${path%.git}"
GIT_SSH_COMMAND='ssh -o BatchMode=yes -o ConnectTimeout=10' \
  git ls-remote origin >/dev/null 2>&1 \
  || { echo "deploy key not authorised yet (or lacks access)."
       echo
       echo "Add this public key at:"
       echo "  https://$host/$path/settings/keys"
       echo "giving it WRITE access. The title can be anything; the key's"
       echo "own comment names the box, which most forges offer as the"
       echo "default."
       echo
       cat "$HOME/.ssh/id_ed25519_admin_repo.pub"
       echo
       echo "Printed in full rather than named by path, because it is a"
       echo "public key: nothing is risked by it appearing here, and the"
       echo "alternative is whoever reads this going to fetch it."
       exit 1; }
echo "forge reachable and key authorised"

The forge API token

The ssh key above pushes branches. Filing an issue, opening a pull request and merging it go over the forge's HTTP API instead, which wants a token — and adm issue, adm land and adm upstream all read it from <box>.env as FORGE_TOKEN.

No runbook mentioned it until now, because bin/adm reads it rather than a block does. That had a consequence beyond the documentation gap: adm adopt and adm env-fill both derive the list of keys a new box needs from the ${KEY:?} assertions in these runbooks, so neither knew the most secret value in the repository existed. A box bootstrapped from scratch got as far as 015 with nothing telling it to collect a token, and found out at the first adm land.

Create one in the forge under Settings → Applications → Generate New Token with the write:issue scope, then add it with adm env-fill — never by editing the file, for the reasons 015 gives.

cd "$ADM_ROOT"
: "${FORGE_TOKEN:?not set in $BOX_NAME.env -- create one in the forge under Settings -> Applications (scope write:issue) and add it with: adm env-fill}"
# Checked through adm rather than curl. adm derives the API base and the
# owner/repo slug from GIT_REMOTE, and doing that again here would be a
# second derivation free to disagree with the one that does the work. A dry
# run exercises exactly that path, and prints no token -- which a curl would,
# because the Authorization header would sit in argv where every process on
# the box can read it. 070 makes the same argument about agent arguments.
#
# -b '' rather than no body: with no -b and a non-tty stdin, adm issue reads
# the body from stdin, and a verify block does not have one to give.
adm issue -n -b '' "probe: can adm address the forge API" >/dev/null
echo "FORGE_TOKEN present; adm can address the forge API"

Why this step is not automated

The forge has an API for deploy keys, so 020 could register it and the manual step would disappear. It does not, deliberately.

Adding a key through POST /repos/{owner}/{repo}/keys needs a token with repository scope. FORGE_TOKEN is created with write:issue, which is what adm issue, adm land and adm upstream need, and it is not enough — measured, not assumed:

GET /repos/<slug>/keys  ->  HTTP 403 Forbidden

The finest scope that can add a key also permits editing the repository's settings, cutting releases and deleting branches. So automating one manual step would mean every box storing a credential that can reshape its repository — and since 080 embeds the encrypted env, that wider token travels inside the published document. A larger prize behind the same ciphertext, to save one paste.

What is automated instead is the part that costs nothing: the key is printed in full, beside the URL to add it to. An agent can do that much safely, because a public key is public — which is the one thing in this repository an agent may hold and read aloud.

What a rebuild means for this key

A rebuilt box generates a new keypair, so the forge step must be repeated by hand. That is an honest gap in the "replay reproduces the box" claim — one manual action sits in the middle of it.

The alternative is to treat the private key as a managed secret: keep it in the password manager and have this runbook install it rather than generate it, making replay fully unattended at the cost of a long-lived secret stored off-box. Not chosen yet; generating fresh is the safer default and the manual step is cheap at this scale. Revisit if this box is ever rebuilt often enough for it to chafe.

Push

Safe to re-run: an up-to-date branch pushes as a no-op. It fails on genuine divergence, which is a real condition worth stopping for rather than papering over with a force push.

On a rebuild you would clone from this remote rather than push to it — the repo has to exist on the box before it can replay anything, so cloning is the bootstrap. See the rebuild section in README.md.

cd "$ADM_ROOT"
git push -u origin master
git branch -vv --list master

cd "$ADM_ROOT"
# master against origin/master by name. Using HEAD and @{upstream} meant this
# reported "master and its upstream disagree" while standing on a branch that
# simply had no upstream yet -- which says nothing about master.
test "$(git rev-parse master)" = "$(git rev-parse origin/master)" \
  || { echo "master and origin/master disagree"; exit 1; }
echo "master is in sync with origin/master"

Upstream: where the runbooks come from

origin is this machine's private repo. upstream is the public adm repository every box's generic half comes from, and this repo is a fork of it: its first commit is upstream's first commit, same object id. That is what makes pulling fixes an ordinary merge instead of an argument.

That first commit's id is written down in the check below, the way FORGE_HOST_FINGERPRINT is written down in 015: a fact about something remote, recorded so it can be checked without asking the remote. The check needs it because a box rebuilt from the published document has no history and begins with git init -- a fresh, unrelated root. Asking upstream would answer the question, but only from a box that can reach the network, and the boxes most likely to have the wrong root are exactly the ones rebuilt offline. Recorded, the answer is available always, so a wrong root is a failure rather than a skipped check.

cd "$ADM_ROOT"
test -n "${ADM_UPSTREAM:-}" || ADM_UPSTREAM=https://github.com/EnigmaCurry/adm.git
git remote get-url upstream >/dev/null 2>&1 \
  || git remote add upstream "$ADM_UPSTREAM"
git remote set-url upstream "$ADM_UPSTREAM"
# Fetch over https, push over ssh. Pulling from upstream is routine and must
# not depend on an agent being forwarded; pushing is occasional, needs a key
# the public repo has authorised, and is the direction worth making awkward.
test -n "${ADM_UPSTREAM_PUSH:-}" \
  || ADM_UPSTREAM_PUSH="$(printf '%s' "$ADM_UPSTREAM" \
       | sed -E 's|^https://([^/]+)/|git@\1:|')"
git remote set-url --push upstream "$ADM_UPSTREAM_PUSH"
git remote get-url upstream
git remote get-url --push upstream

cd "$ADM_ROOT"
url="$(git remote get-url upstream 2>/dev/null)" \
  || { echo "no upstream remote; run this runbook"; exit 1; }
case "$url" in
  https://*) : ;;
  *) echo "upstream should be fetched over https: it is public, needs no key,"
     echo "and an ssh url would make a routine fetch depend on an agent"; exit 1 ;;
esac
# A fork shares upstream's root commit. Without that there is no merge base,
# and every per-box file arrives as a modify/delete conflict on first merge.
#
# Compared against the recorded id, not against a fetch. This check used to
# fetch and skip itself when the network was down, which passed the one box
# that most needed failing: a box rebuilt from the published document has an
# unrelated root by construction, and an offline rebuild is how it gets there.
# Declared here and nowhere else. Override it together with $ADM_UPSTREAM
# when forking adm for a fleet of your own: a root commit from one upstream
# means nothing to another.
test -n "${ADM_UPSTREAM_ROOT:-}" \
  || ADM_UPSTREAM_ROOT=5f06bf3ecc68dca60df75b27a247521ed6e38ce4
root_local="$(git rev-list --max-parents=0 HEAD | tail -1)"
if test "$root_local" != "$ADM_UPSTREAM_ROOT"; then
  echo "this repo does not share upstream's root commit:"
  echo "  here:     $root_local"
  echo "  upstream: $ADM_UPSTREAM_ROOT"
  echo "merges would need --allow-unrelated-histories and would conflict"
  echo "on every per-box file. See DECISIONS 007."
  echo
  echo "A repo rebuilt from the published document starts this way, because"
  echo "git init makes its own root. To re-fork it without losing the work:"
  echo
  echo "  git fetch upstream master"
  echo "  git reset --soft upstream/master     # move HEAD, keep the tree"
  echo "  git commit -m 'adopt this box onto upstream'"
  echo
  echo "The soft reset leaves the index holding this tree while HEAD becomes"
  echo "upstream's tip, so the commit that follows has upstream as its parent"
  echo "and every file exactly as it is now. Not --amend: that would replace"
  echo "upstream's own tip commit instead of building on it."
  echo
  echo "Files upstream tracks that the document does not carry -- hooks/ --"
  echo "will show as deleted until 015 writes them again, which it does."
  echo "Then push once with --force-with-lease; the root commit changed."
  exit 1
fi
# When the network is there, confirm the recorded id is still upstream's. It
# would not be if upstream were rewritten -- which has happened once -- and a
# recorded fact that has gone stale is worse than no fact.
if git fetch -q upstream 2>/dev/null; then
  root_up="$(git rev-list --max-parents=0 upstream/master | tail -1)"
  test "$root_up" = "$ADM_UPSTREAM_ROOT" \
    || { echo "the recorded root is stale: this runbook says"
         echo "  $ADM_UPSTREAM_ROOT"
         echo "but $(git remote get-url upstream) now roots at"
         echo "  $root_up"
         echo "upstream was rewritten. Update ADM_UPSTREAM_ROOT in this"
         echo "runbook, and expect to re-fork every box."; exit 1; }
fi
push="$(git remote get-url --push upstream 2>/dev/null)"
case "$push" in
  git@*|ssh://*) : ;;
  *) echo "upstream pushes over '$push'; it should be ssh, so a push uses a"
     echo "key rather than prompting for a password on a headless box"; exit 1 ;;
esac
echo "upstream $url (push $push), sharing root ${root_local:0:12}"