kayg/infraternal

Watch

Fork 0

Throwaway test server for a second account (D328) #326

New issue

Open

opened 2026-11-03 12:46:43 +00:00 by kayg · 6 comments

kayg commented

2026-11-03 12:46:43 +00:00

Owner

Owner decision D328: a second, limited account for testing runs on a throwaway test server (for example test.ssh.kayg.org), not through invites ("test server because invites would then belong to EE").

Job J138 is read-only discovery and a plan. Findings and the plan are in the first comment. The repo is public: no secrets here.

Owner decision D328: a second, limited account for testing runs on a throwaway test server (for example `test.ssh.kayg.org`), not through invites ("test server because invites would then belong to EE"). Job J138 is read-only discovery and a plan. Findings and the plan are in the first comment. The repo is public: no secrets here.

kayg commented

2026-11-03 12:46:43 +00:00

Author

Owner

J138 findings and plan (read-only discovery; nothing was changed on any host)

Base: origin/dev, checked 2026-11-03. Owner decision D328: "test server because invites would then belong to EE".

Findings

1. How public traffic reaches production

ssh.kayg.org resolves to the public A record 49.205.172.130. It lands on Traefik 3.7.x on the fridge-ingress VM (10.69.0.99 / 10.69.69.99, 2 CPU, 4 GiB, idle). Traefik runs as traefik.service and is configured by /etc/traefik/traefik.yml.
The file provider watches /etc/traefik/dynamic/ (watch: true). A new *.yml there is picked up without a restart.
The production route is /etc/traefik/dynamic/infraternal-ssh.yml. It is router Host(ssh.kayg.org) on websecure, with tls.certResolver: cf. Its service is a loadBalancer to http://10.69.0.89:7011 with passHostHeader: true. Port 80 redirects to 443.
TLS comes from the ACME resolver cf (Let's Encrypt, DNS-01 through the Cloudflare provider). Storage is /var/lib/traefik/acme.json. Each hostname gets its own certificate on first request. There is no wildcard certificate.
The Cloudflare credential is at /etc/traefik/cloudflare-token, copied to /var/lib/traefik/cloudflare-token-runtime by a unit drop-in. These are paths only. I did not read or use the token.
On the production host, the server runs with --behind-proxy --trusted-proxy 10.69.0.99. The host's nftables (/etc/nftables/infraternal-host.nft, loaded by infraternal-firewall.service) uses a default-drop input policy. It allows tcp 7011 only from 10.69.0.99 (on eth0) and 10.69.69.99 (on wt0). A second port needs its own nft rule.

2. DNS

kayg.org is on Cloudflare (NS joel.ns.cloudflare.com and stella.ns.cloudflare.com).
test.ssh.kayg.org does not exist yet. The zone's *.kayg.org wildcard does not cover it, because ssh.kayg.org already exists as a name and a wildcard does not reach below it. So the record must be created explicitly.
The Traefik host holds a Cloudflare token that issues DNS-01 TXT records. It also has /var/lib/traefik/dns-backups/, which shows a past DNS edit (the Honcho cutover). Unverified: whether this token has Zone DNS Edit rights (A records) or only TXT/ACME rights. Checking means using the token (one read-only zone list), so I did not do it.
Who adds the record: if the owner approves using that token, we can add it ourselves. Otherwise the owner adds one A record in the Cloudflare dashboard (see the plan).

3. Can the server run a second instance? Yes, with no code change. Every location is a flag:

--data <dir> (state, sqlite, CA material), --listen <addr:port>, --public-origin <https url>, --web-root, --behind-proxy, --trusted-proxy. --remote-listen and --remote-ca-dir are optional (production uses them for enrolled hosts).
The passkey RP id is derived from the host of --public-origin (runtime.rs:573). The session and ceremony cookies are __Host- prefixed (host-only). A test server on test.ssh.kayg.org therefore shares no passkeys or sessions with production.
Catch, the runtime dir: the embedded infraternal agent and the session shims default to /run/infraternal when running as root (crates/infraternal-agent/src/sessions.rs:184). A second root instance would share that directory with production. Run the test instance as a dedicated non-root user instead. Then the path becomes $XDG_RUNTIME_DIR/infraternal. Set XDG_RUNTIME_DIR to a RuntimeDirectory= of the test unit.
Non-root means cold and locked: per docs/operations/deploy.md, a non-root server restarts with the CA locked. A passkey administrator must unlock it after each restart. That is fine for a throwaway server.
Not yet verified: I did not start the binary. Step 1 of the plan checks this on a dev VM first.

4. Headroom on the production host (root@10.69.69.89)

RAM: 7.7 GiB total, 7.1 GiB available, no swap. The running server uses about 40 MB RSS (+8 MB for its agent).
Disk: 46 GiB free of 49 GiB. /var/lib/infraternal is 5.1 MB. A release directory in /opt/infraternal/<rev> is 51 MB.
CPU: 4 cores, load average 0.00.
/opt/infraternal/ff93201e5 and its binary are mode 555 root, world-readable, so a non-root user can run them.
Verdict: a second instance fits easily.

Plan (shortest safe path)

Legend: [me] = I can do it. [owner] = the owner must do it or give the go (production host, public DNS, or a credential).

[me] Prove the flags on a dev VM, not on production. Run the same ff93201e5 binary as a non-root user with --listen 127.0.0.1:<port>, a throwaway --data, a --public-origin and XDG_RUNTIME_DIR=<0700 dir>. Confirm that /healthz answers and that the embedded agent starts under $XDG_RUNTIME_DIR/infraternal. Needs a build-slot. Nothing touches production.
[owner] DNS: add test.ssh.kayg.org A 49.205.172.130, DNS-only (grey cloud), TTL auto. The address is the same as ssh.kayg.org. Alternative: the owner says "use the Traefik token" and I add it through the Cloudflare API after one read-only scope check. Say which.
[owner] Go for the production host edits (steps 4 to 6). They are small and additive, but the host is production. Then [me] does them.
[me, after go] Firewall: add iifname "eth0" ip saddr 10.69.0.99 tcp dport 7012 accept to /etc/nftables/infraternal-host.nft and load it live with nft add rule (no table reload, so there is no gap in the default-drop policy).

[me, after go] User and unit on 10.69.69.89: useradd --system --home-dir /var/lib/infraternal-test --shell /usr/sbin/nologin infraternal-test, then /etc/systemd/system/infraternal-test.service:

[Unit]
Description=infraternal throwaway test server (D328)
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=infraternal-test
UMask=0077
StateDirectory=infraternal-test
StateDirectoryMode=0700
RuntimeDirectory=infraternal-test
RuntimeDirectoryMode=0700
Environment=XDG_RUNTIME_DIR=/run/infraternal-test
WorkingDirectory=/var/lib/infraternal-test
ExecStart=/opt/infraternal/ff93201e5/infraternal server --data /var/lib/infraternal-test --listen 10.69.0.89:7012 --public-origin https://test.ssh.kayg.org --web-root /opt/infraternal/ff93201e5/web --behind-proxy --trusted-proxy 10.69.0.99
Restart=on-failure
RestartSec=2
KillMode=process
LimitMEMLOCK=8388608
[Install]
WantedBy=multi-user.target

Then daemon-reload and enable --now infraternal-test. It never touches infraternal-server.service, /var/lib/infraternal or port 7011. No --remote-listen: the test server starts with the embedded host only. (If the second account must also see a remote host, add --remote-listen 10.69.69.89:7444 plus a wt0 nft rule. It is not needed for the account test.)

[me, after go] Proxy route on fridge-ingress: create /etc/traefik/dynamic/infraternal-test.yml (root, 0644, same as the sibling files), a copy of infraternal-ssh.yml with router infraternal-test, rule Host(\test.ssh.kayg.org`), certResolver: cf, and server http://10.69.0.89:7012`. Traefik reloads on its own. The certificate is issued on the first request by DNS-01 (about 1 minute). No Traefik restart and no edit of existing files.
[owner/tester] First use: open https://test.ssh.kayg.org, create the first administrator with a passkey, then unlock the CA. The tester creates the limited second account there.
Updating alongside production [me, after production deploys]: the production deploy script does not know this unit. After a production deploy to a new <rev>, change the two /opt/infraternal/<rev> paths in the unit, then daemon-reload and systemctl restart infraternal-test. It restarts cold, so the owner or tester unlocks the CA again. The old <rev> directories stay in /opt/infraternal, so the test unit can keep an older revision. Check the rev against /opt/infraternal/ first; do not point it at a pruned directory.
Complete removal [me, after go], in this order:
1. rm /etc/traefik/dynamic/infraternal-test.yml on fridge-ingress (Traefik drops the route).
2. [owner] delete the test.ssh.kayg.org A record in Cloudflare (or [me] via API if approved in step 2).
3. systemctl disable --now infraternal-test, rm /etc/systemd/system/infraternal-test.service, systemctl daemon-reload.
4. rm -rf /var/lib/infraternal-test (this destroys the test accounts, as intended), then userdel infraternal-test.
5. Remove the 7012 nft rule from /etc/nftables/infraternal-host.nft and from the live table (nft -a list chain inet infraternal_host input, then nft delete rule … handle N).
6. The ACME certificate entry stays in /var/lib/traefik/acme.json. It is harmless and expires on its own. Removing it means hand-editing a secrets file, so I would leave it.

Risks and open points

The scope of the Cloudflare token (A-record rights) is unknown. Step 2 asks the owner to choose.
The non-root XDG_RUNTIME_DIR setup is untested until step 1.
Production is unaffected by design: separate user, state dir, runtime dir, port and origin. The only shared piece is the nft file (step 4, one added line) and Traefik's dynamic directory (one new file).
No code or repository change, so no scripts/verify fast run (gate not applicable).

## J138 findings and plan (read-only discovery; nothing was changed on any host) Base: origin/dev, checked 2026-11-03. Owner decision D328: "test server because invites would then belong to EE". ### Findings **1. How public traffic reaches production** - `ssh.kayg.org` resolves to the public A record 49.205.172.130. It lands on **Traefik 3.7.x on the `fridge-ingress` VM** (`10.69.0.99` / `10.69.69.99`, 2 CPU, 4 GiB, idle). Traefik runs as `traefik.service` and is configured by `/etc/traefik/traefik.yml`. - The file provider watches `/etc/traefik/dynamic/` (`watch: true`). A new `*.yml` there is picked up without a restart. - The production route is `/etc/traefik/dynamic/infraternal-ssh.yml`. It is router `Host(ssh.kayg.org)` on `websecure`, with `tls.certResolver: cf`. Its service is a `loadBalancer` to `http://10.69.0.89:7011` with `passHostHeader: true`. Port 80 redirects to 443. - TLS comes from the ACME resolver `cf` (Let's Encrypt, **DNS-01 through the Cloudflare provider**). Storage is `/var/lib/traefik/acme.json`. Each hostname gets its own certificate on first request. There is no wildcard certificate. - The Cloudflare credential is at `/etc/traefik/cloudflare-token`, copied to `/var/lib/traefik/cloudflare-token-runtime` by a unit drop-in. These are paths only. I did not read or use the token. - On the production host, the server runs with `--behind-proxy --trusted-proxy 10.69.0.99`. The host's nftables (`/etc/nftables/infraternal-host.nft`, loaded by `infraternal-firewall.service`) uses a default-drop input policy. It allows tcp 7011 only from 10.69.0.99 (on eth0) and 10.69.69.99 (on wt0). **A second port needs its own nft rule.** **2. DNS** - `kayg.org` is on **Cloudflare** (NS `joel.ns.cloudflare.com` and `stella.ns.cloudflare.com`). - `test.ssh.kayg.org` does not exist yet. The zone's `*.kayg.org` wildcard does not cover it, because `ssh.kayg.org` already exists as a name and a wildcard does not reach below it. So the record must be created explicitly. - The Traefik host holds a Cloudflare token that issues DNS-01 TXT records. It also has `/var/lib/traefik/dns-backups/`, which shows a past DNS edit (the Honcho cutover). **Unverified: whether this token has Zone DNS Edit rights (A records) or only TXT/ACME rights.** Checking means using the token (one read-only zone list), so I did not do it. - **Who adds the record:** if the owner approves using that token, we can add it ourselves. Otherwise the owner adds one A record in the Cloudflare dashboard (see the plan). **3. Can the server run a second instance?** Yes, with no code change. Every location is a flag: - `--data <dir>` (state, sqlite, CA material), `--listen <addr:port>`, `--public-origin <https url>`, `--web-root`, `--behind-proxy`, `--trusted-proxy`. `--remote-listen` and `--remote-ca-dir` are optional (production uses them for enrolled hosts). - The passkey RP id is derived from the host of `--public-origin` (`runtime.rs:573`). The session and ceremony cookies are `__Host-` prefixed (host-only). A test server on `test.ssh.kayg.org` therefore shares no passkeys or sessions with production. - **Catch, the runtime dir:** the embedded infraternal agent and the session shims default to `/run/infraternal` when running as root (`crates/infraternal-agent/src/sessions.rs:184`). A second root instance would share that directory with production. Run the test instance as a **dedicated non-root user** instead. Then the path becomes `$XDG_RUNTIME_DIR/infraternal`. Set `XDG_RUNTIME_DIR` to a `RuntimeDirectory=` of the test unit. - **Non-root means cold and locked:** per `docs/operations/deploy.md`, a non-root server restarts with the CA locked. A passkey administrator must unlock it after each restart. That is fine for a throwaway server. - Not yet verified: I did not start the binary. Step 1 of the plan checks this on a dev VM first. **4. Headroom on the production host (`root@10.69.69.89`)** - RAM: 7.7 GiB total, 7.1 GiB available, no swap. The running server uses about 40 MB RSS (+8 MB for its agent). - Disk: 46 GiB free of 49 GiB. `/var/lib/infraternal` is 5.1 MB. A release directory in `/opt/infraternal/<rev>` is 51 MB. - CPU: 4 cores, load average 0.00. - `/opt/infraternal/ff93201e5` and its binary are mode 555 root, world-readable, so a non-root user can run them. - Verdict: a second instance fits easily. ### Plan (shortest safe path) Legend: **[me]** = I can do it. **[owner]** = the owner must do it or give the go (production host, public DNS, or a credential). 1. **[me] Prove the flags on a dev VM, not on production.** Run the same `ff93201e5` binary as a non-root user with `--listen 127.0.0.1:<port>`, a throwaway `--data`, a `--public-origin` and `XDG_RUNTIME_DIR=<0700 dir>`. Confirm that `/healthz` answers and that the embedded agent starts under `$XDG_RUNTIME_DIR/infraternal`. Needs a build-slot. Nothing touches production. 2. **[owner] DNS:** add `test.ssh.kayg.org A 49.205.172.130`, DNS-only (grey cloud), TTL auto. The address is the same as `ssh.kayg.org`. *Alternative:* the owner says "use the Traefik token" and I add it through the Cloudflare API after one read-only scope check. Say which. 3. **[owner] Go for the production host edits (steps 4 to 6).** They are small and additive, but the host is production. Then **[me]** does them. 4. **[me, after go] Firewall:** add `iifname "eth0" ip saddr 10.69.0.99 tcp dport 7012 accept` to `/etc/nftables/infraternal-host.nft` and load it live with `nft add rule` (no table reload, so there is no gap in the default-drop policy). 5. **[me, after go] User and unit on `10.69.69.89`:** `useradd --system --home-dir /var/lib/infraternal-test --shell /usr/sbin/nologin infraternal-test`, then `/etc/systemd/system/infraternal-test.service`: ```ini [Unit] Description=infraternal throwaway test server (D328) After=network-online.target Wants=network-online.target [Service] Type=simple User=infraternal-test UMask=0077 StateDirectory=infraternal-test StateDirectoryMode=0700 RuntimeDirectory=infraternal-test RuntimeDirectoryMode=0700 Environment=XDG_RUNTIME_DIR=/run/infraternal-test WorkingDirectory=/var/lib/infraternal-test ExecStart=/opt/infraternal/ff93201e5/infraternal server --data /var/lib/infraternal-test --listen 10.69.0.89:7012 --public-origin https://test.ssh.kayg.org --web-root /opt/infraternal/ff93201e5/web --behind-proxy --trusted-proxy 10.69.0.99 Restart=on-failure RestartSec=2 KillMode=process LimitMEMLOCK=8388608 [Install] WantedBy=multi-user.target ``` Then `daemon-reload` and `enable --now infraternal-test`. It never touches `infraternal-server.service`, `/var/lib/infraternal` or port 7011. No `--remote-listen`: the test server starts with the embedded host only. (If the second account must also see a remote host, add `--remote-listen 10.69.69.89:7444` plus a wt0 nft rule. It is not needed for the account test.) 6. **[me, after go] Proxy route** on `fridge-ingress`: create `/etc/traefik/dynamic/infraternal-test.yml` (root, 0644, same as the sibling files), a copy of `infraternal-ssh.yml` with router `infraternal-test`, rule `Host(\`test.ssh.kayg.org\`)`, `certResolver: cf`, and server `http://10.69.0.89:7012`. Traefik reloads on its own. The certificate is issued on the first request by DNS-01 (about 1 minute). No Traefik restart and no edit of existing files. 7. **[owner/tester] First use:** open `https://test.ssh.kayg.org`, create the first administrator with a passkey, then unlock the CA. The tester creates the limited second account there. 8. **Updating alongside production [me, after production deploys]:** the production deploy script does not know this unit. After a production deploy to a new `<rev>`, change the two `/opt/infraternal/<rev>` paths in the unit, then `daemon-reload` and `systemctl restart infraternal-test`. It restarts cold, so the owner or tester unlocks the CA again. The old `<rev>` directories stay in `/opt/infraternal`, so the test unit can keep an older revision. Check the rev against `/opt/infraternal/` first; do not point it at a pruned directory. 9. **Complete removal [me, after go], in this order:** 1. `rm /etc/traefik/dynamic/infraternal-test.yml` on `fridge-ingress` (Traefik drops the route). 2. **[owner]** delete the `test.ssh.kayg.org` A record in Cloudflare (or [me] via API if approved in step 2). 3. `systemctl disable --now infraternal-test`, `rm /etc/systemd/system/infraternal-test.service`, `systemctl daemon-reload`. 4. `rm -rf /var/lib/infraternal-test` (this destroys the test accounts, as intended), then `userdel infraternal-test`. 5. Remove the 7012 nft rule from `/etc/nftables/infraternal-host.nft` and from the live table (`nft -a list chain inet infraternal_host input`, then `nft delete rule … handle N`). 6. The ACME certificate entry stays in `/var/lib/traefik/acme.json`. It is harmless and expires on its own. Removing it means hand-editing a secrets file, so I would leave it. ### Risks and open points - The scope of the Cloudflare token (A-record rights) is unknown. Step 2 asks the owner to choose. - The non-root `XDG_RUNTIME_DIR` setup is untested until step 1. - Production is unaffected by design: separate user, state dir, runtime dir, port and origin. The only shared piece is the nft file (step 4, one added line) and Traefik's dynamic directory (one new file). - No code or repository change, so no `scripts/verify fast` run (gate not applicable).

kayg commented

2026-11-03 12:51:45 +00:00

Author

Owner

Owner, 2026-11-03: DNS via the Traefik Cloudflare token: "yes". Go for the production-host changes (nft rule, infraternal-test user and unit, Traefik route): "sure". Admin of the test server: "since it's your test account, why don't you become admin?" The orchestrator will be its admin, using a software passkey (a virtual WebAuthn authenticator) whose credential stays on a build VM, mode 0600, never in the repo or logs. Order: J139 dev-VM verification, then the token scope check, DNS, nft, unit, route, then admin set-up.

kayg commented

2026-11-03 12:52:16 +00:00

Author

Owner

J139 result: step 1 proven on a dev VM (nothing touched production or fridge-ingress)

Host: infraternal-dev-01-compute01, user claude (non-root). Binary: the existing ~/deploy-ff93201e5-result (nix build of .#packages.x86_64-linux.default at ff93201e5, bin/infraternal plus share/infraternal/web). No build was needed. No repo change; gate not applicable.

Setup. Two infraternal server processes side by side, each with its own mktemp dir (0700), --data <tmp>/X/data, XDG_RUNTIME_DIR=<tmp>/X/xdg, --web-root, --behind-proxy --trusted-proxy 127.0.0.1:

A (production stand-in): --listen 127.0.0.1:17811 --public-origin https://A.example.invalid
B (test): --listen 127.0.0.1:17812 --public-origin https://B.example.invalid, later https://test.example.invalid

Evidence.

Both start. Each logs server_ca_startup ... outcome="cold" then infraternal server started listen=127.0.0.1:1781x ... behind_proxy=true. /healthz returns 200 on both.
No shared state. Runtime dirs resolve to $XDG_RUNTIME_DIR/infraternal (not /run/infraternal): each has its own plugins.sock and sessions/, owner claude, 0700 dirs. The listening unix sockets are all separate paths: A|B/xdg/infraternal/plugins.sock, A|B/data/restart/control.sock, A|B/data/setup-token.sock. /run/infraternal on this VM is a root-owned 0700 dir (another root process uses it); its mtime was unchanged, so the non-root instances never touched it. That confirms the sessions.rs:184 collision is avoided by running non-root.
B's sign-in page serves: GET / returns 200 (1759 B, the SPA shell with #app), and /signin returns 200. Sign-in itself is a client-side view, so this proves the shell and static files only, not a passkey ceremony.
B restarts cold with the CA locked: SIGTERM, B exits in under 5 s, healthz goes 000, then starting it again on the same --data logs server_ca_startup outcome="cold" and serves 200. Its plugins.sock is removed on exit and recreated on start.
Stopping B leaves A untouched: same PID, same RSS (52528 KB), /healthz 200 before and after, its sockets still listening, 0 error/warn lines in A's log.
Cleanup: A and B stopped by PID (no pkill -f on the VM), temp dirs and my scripts removed, /tmp/j139* gone, no leftover infraternal server processes.

Plan unit text: no change needed to the flags. The ExecStart flags, --behind-proxy --trusted-proxy, XDG_RUNTIME_DIR and the non-root cold/locked behaviour all work as written.

Not tested here (needs the real host, say so before step 3):

The systemd directives (User=, StateDirectory=, RuntimeDirectory=, LimitMEMLOCK, KillMode=process) and UMask=0077. I ran plain processes, not a unit. On the real host, systemd-analyze verify plus the first start covers it.
RuntimeDirectory=infraternal-test sets XDG_RUNTIME_DIR=/run/infraternal-test in the unit's Environment=, so the agent will use /run/infraternal-test/infraternal. The test confirmed the $XDG_RUNTIME_DIR/infraternal rule, but I did not see the agent write there under systemd.
A --listen 10.69.0.89:7012 bind (not loopback) and the nft rule.
Embedded host/agent activity beyond startup (no session was opened).

Slip to disclose. While locating the brief I ran a stray pkill -f "find / -name BRIEF" (one local, one over ssh). It only matched my own search. Nothing else on the VM was affected.

## J139 result: step 1 proven on a dev VM (nothing touched production or fridge-ingress) Host: `infraternal-dev-01-compute01`, user `claude` (non-root). Binary: the existing `~/deploy-ff93201e5-result` (nix build of `.#packages.x86_64-linux.default` at `ff93201e5`, `bin/infraternal` plus `share/infraternal/web`). No build was needed. No repo change; gate not applicable. **Setup.** Two `infraternal server` processes side by side, each with its own `mktemp` dir (0700), `--data <tmp>/X/data`, `XDG_RUNTIME_DIR=<tmp>/X/xdg`, `--web-root`, `--behind-proxy --trusted-proxy 127.0.0.1`: - A (production stand-in): `--listen 127.0.0.1:17811 --public-origin https://A.example.invalid` - B (test): `--listen 127.0.0.1:17812 --public-origin https://B.example.invalid`, later `https://test.example.invalid` **Evidence.** 1. Both start. Each logs `server_ca_startup ... outcome="cold"` then `infraternal server started listen=127.0.0.1:1781x ... behind_proxy=true`. `/healthz` returns 200 on both. 2. No shared state. Runtime dirs resolve to `$XDG_RUNTIME_DIR/infraternal` (not `/run/infraternal`): each has its own `plugins.sock` and `sessions/`, owner `claude`, 0700 dirs. The listening unix sockets are all separate paths: `A|B/xdg/infraternal/plugins.sock`, `A|B/data/restart/control.sock`, `A|B/data/setup-token.sock`. `/run/infraternal` on this VM is a root-owned 0700 dir (another root process uses it); its mtime was unchanged, so the non-root instances never touched it. That confirms the `sessions.rs:184` collision is avoided by running non-root. 3. B's sign-in page serves: `GET /` returns 200 (1759 B, the SPA shell with `#app`), and `/signin` returns 200. Sign-in itself is a client-side view, so this proves the shell and static files only, not a passkey ceremony. 4. B restarts cold with the CA locked: SIGTERM, B exits in under 5 s, healthz goes 000, then starting it again on the same `--data` logs `server_ca_startup outcome="cold"` and serves 200. Its `plugins.sock` is removed on exit and recreated on start. 5. Stopping B leaves A untouched: same PID, same RSS (52528 KB), `/healthz` 200 before and after, its sockets still listening, 0 error/warn lines in A's log. 6. Cleanup: A and B stopped by PID (no `pkill -f` on the VM), temp dirs and my scripts removed, `/tmp/j139*` gone, no leftover `infraternal server` processes. **Plan unit text: no change needed to the flags.** The `ExecStart` flags, `--behind-proxy --trusted-proxy`, `XDG_RUNTIME_DIR` and the non-root cold/locked behaviour all work as written. **Not tested here (needs the real host, say so before step 3):** - The systemd directives (`User=`, `StateDirectory=`, `RuntimeDirectory=`, `LimitMEMLOCK`, `KillMode=process`) and `UMask=0077`. I ran plain processes, not a unit. On the real host, `systemd-analyze verify` plus the first start covers it. - `RuntimeDirectory=infraternal-test` sets `XDG_RUNTIME_DIR=/run/infraternal-test` in the unit's `Environment=`, so the agent will use `/run/infraternal-test/infraternal`. The test confirmed the `$XDG_RUNTIME_DIR/infraternal` rule, but I did not see the agent write there under systemd. - A `--listen 10.69.0.89:7012` bind (not loopback) and the nft rule. - Embedded host/agent activity beyond startup (no session was opened). **Slip to disclose.** While locating the brief I ran a stray `pkill -f "find / -name BRIEF"` (one local, one over ssh). It only matched my own search. Nothing else on the VM was affected.

kayg commented

2026-11-03 13:03:10 +00:00

Author

Owner

J140 started. Plan: (1) read the first-run setup token from this instance's own socket (infraternal-test user only; production untouched), (2) run the setup wizard in Playwright Chromium on compute01 with a CDP virtual authenticator (ctap2/internal/resident/UV/PRF), (3) unlock the machine CA (D168), (4) export the credential + recovery key to ~claude/.config/infraternal-test-admin/ (0700/0600), plus a sign-in helper, (5) report whether Community can make a second account. No repo changes. Estimate: ~1 h.

Finding so far: the token is served on <data>/setup-token.sock and read with infraternal server setup-token --data <dir> (cli.rs, runtime.rs start_setup_token_socket). It is stored only in a 0600 file on compute01; not shown here.

J140 started. Plan: (1) read the first-run setup token from this instance's own socket (`infraternal-test` user only; production untouched), (2) run the setup wizard in Playwright Chromium on compute01 with a CDP virtual authenticator (ctap2/internal/resident/UV/PRF), (3) unlock the machine CA (D168), (4) export the credential + recovery key to `~claude/.config/infraternal-test-admin/` (0700/0600), plus a sign-in helper, (5) report whether Community can make a second account. No repo changes. Estimate: ~1 h. Finding so far: the token is served on `<data>/setup-token.sock` and read with `infraternal server setup-token --data <dir>` (cli.rs, runtime.rs `start_setup_token_socket`). It is stored only in a 0600 file on compute01; not shown here.

kayg commented

2026-11-03 13:13:45 +00:00

Author

Owner

J140 report (no secrets below; the repo is public).

Done

Setup token: read from this instance's own socket (<data>/setup-token.sock) via infraternal server setup-token --data /var/lib/infraternal-test as the infraternal-test user. Production untouched. Stored in a 0600 file on compute01 only.
Setup wizard run in Playwright Chromium 149 on compute01 against https://test.ssh.kayg.org with a CDP virtual authenticator (ctap2, internal, resident key, UV, hasPrf: true). Username orchestrator, display name "Orchestrator (Claude)". Passkey and recovery key (24 words) completed. The account exists and is signed in (screenshot attached: empty canvas, 1440x900).
Credential export: WebAuthn.getCredentials gives credential id, rpId, private key, user handle and sign count, but no PRF secret (Chrome 149 Credential has no such field). Saved with the recovery key and a session storage-state.json in ~claude/.config/infraternal-test-admin/ (dir 0700, files 0600). Helper: ~claude/infraternal-ops/test-admin-signin.ts.

Blocked / not done

Passkey sign-in from the exported credential does not work. Re-importing the credential gives a valid signature (login-finish 200) but the authenticator returns no PRF output (checked on two fresh authenticators), so the account key cannot be unwrapped and the app logs out straight away. The server also tracks the signature counter monotonically; the helper now imports a time-based counter for that.
Step 3 (unlock the machine CA) not done. The unlock needs a live PRF ceremony. I closed the setup browser (and its virtual authenticator) before doing it. That was my mistake, and the PRF secret is lost with it.
What does work: restoring the saved session (storage-state.json) opens the signed-in canvas with no passkey (the session is 30 days, sliding, D28). The Settings > Hosts screenshot did not show a readable CA row, so there is no CA-state evidence.
Observed, not investigated: a toast "This terminal request is not valid. Try opening a new window." on the first canvas load of this empty server.
Compute01's resolver still caches the miss for test.ssh.kayg.org although the public A record is 49.205.172.130. The helper falls back to a Chromium host-resolver rule in that case.

Step 5: can Community make a second, non-admin account? No.

Auth actions in auth/http.rs: setup-start, registration-finish, login-*, add-start (a second passkey for the same user), recovery, device approval, API keys. There is no create-user, list-users or invite action.
Setup is gated by NOT EXISTS (SELECT 1 FROM users WHERE owner=1): once an owner exists, setup_check returns SetupClosed.
No ee/ directory exists on dev. /remote/invitations is for remote machines, not people.
D31 puts invites and multi-user management in ee/; D154 G8 describes local users by invite link. D328: a second, limited test account runs on this throwaway server "not through invites", but nothing in the code creates one. That needs an owner decision and building; I built nothing.

Needs a decision (orchestrator / kayg)
a) Keep one long-lived headless Chromium on compute01 holding a virtual authenticator that never restarts, and register a fresh passkey on it, via the recovery key (24 h wait, D25) or by resetting this empty throwaway instance (new setup token). Fragile: a reboot loses the PRF secret and with it the account.
b) Reset the empty instance and redo setup in one browser session that also does the CA unlock and any other PRF step. Needs the owner's go (deletes /var/lib/infraternal-test).
c) A test-only way to get PRF-capable sign-in (for example a software authenticator whose secret can be stored) is a repo change; I stopped there.

My browsers have exited (none of mine running). infraternal-ops/j140-setup.ts is the one-shot setup script (contains no secrets).

J140 report (no secrets below; the repo is public). **Done** 1. Setup token: read from this instance's own socket (`<data>/setup-token.sock`) via `infraternal server setup-token --data /var/lib/infraternal-test` as the `infraternal-test` user. Production untouched. Stored in a 0600 file on compute01 only. 2. Setup wizard run in Playwright Chromium 149 on compute01 against https://test.ssh.kayg.org with a CDP virtual authenticator (ctap2, internal, resident key, UV, `hasPrf: true`). Username `orchestrator`, display name "Orchestrator (Claude)". Passkey and recovery key (24 words) completed. The account exists and is signed in (screenshot attached: empty canvas, 1440x900). 4. Credential export: `WebAuthn.getCredentials` gives credential id, rpId, private key, user handle and sign count, but **no PRF secret** (Chrome 149 `Credential` has no such field). Saved with the recovery key and a session `storage-state.json` in `~claude/.config/infraternal-test-admin/` (dir 0700, files 0600). Helper: `~claude/infraternal-ops/test-admin-signin.ts`. **Blocked / not done** - **Passkey sign-in from the exported credential does not work.** Re-importing the credential gives a valid signature (login-finish 200) but the authenticator returns no PRF output (checked on two fresh authenticators), so the account key cannot be unwrapped and the app logs out straight away. The server also tracks the signature counter monotonically; the helper now imports a time-based counter for that. - **Step 3 (unlock the machine CA) not done.** The unlock needs a live PRF ceremony. I closed the setup browser (and its virtual authenticator) before doing it. That was my mistake, and the PRF secret is lost with it. - What does work: restoring the saved session (`storage-state.json`) opens the signed-in canvas with no passkey (the session is 30 days, sliding, D28). The Settings > Hosts screenshot did not show a readable CA row, so there is no CA-state evidence. - Observed, not investigated: a toast "This terminal request is not valid. Try opening a new window." on the first canvas load of this empty server. - Compute01's resolver still caches the miss for `test.ssh.kayg.org` although the public A record is 49.205.172.130. The helper falls back to a Chromium host-resolver rule in that case. **Step 5: can Community make a second, non-admin account? No.** - Auth actions in `auth/http.rs`: `setup-start`, `registration-finish`, `login-*`, `add-start` (a second passkey for the *same* user), recovery, device approval, API keys. There is no create-user, list-users or invite action. - Setup is gated by `NOT EXISTS (SELECT 1 FROM users WHERE owner=1)`: once an owner exists, `setup_check` returns `SetupClosed`. - No `ee/` directory exists on dev. `/remote/invitations` is for remote machines, not people. - D31 puts invites and multi-user management in `ee/`; D154 G8 describes local users by invite link. D328: a second, limited test account runs on this throwaway server "not through invites", but nothing in the code creates one. That needs an owner decision and building; I built nothing. **Needs a decision (orchestrator / kayg)** a) Keep one long-lived headless Chromium on compute01 holding a virtual authenticator that never restarts, and register a fresh passkey on it, via the recovery key (24 h wait, D25) or by resetting this empty throwaway instance (new setup token). Fragile: a reboot loses the PRF secret and with it the account. b) Reset the empty instance and redo setup in one browser session that also does the CA unlock and any other PRF step. Needs the owner's go (deletes `/var/lib/infraternal-test`). c) A test-only way to get PRF-capable sign-in (for example a software authenticator whose secret can be stored) is a repo change; I stopped there. My browsers have exited (none of mine running). `infraternal-ops/j140-setup.ts` is the one-shot setup script (contains no secrets).

signed-in-canvas-desktop.png

16 KiB

kayg commented

2026-11-03 13:30:40 +00:00

Author

Owner

J141 report (no secrets below).

Result: done and proven. The admin orchestrator ("Orchestrator (Claude)") exists on https://test.ssh.kayg.org with a durable software authenticator, and the machine CA is unlocked.

What I did

Reset the empty test instance (stop, move /var/lib/infraternal-test, start). I did this three times while diagnosing. Production (infraternal-server, /var/lib/infraternal) was never touched and is still active. All three backup dirs were empty setup-only instances and are deleted now that the new setup is proven.
Test-only software authenticator, outside the repo, on compute01 in ~claude/infraternal-ops/: sw-authenticator-page.js (injected with addInitScript, test origin only) and sw-authenticator.ts (loads/saves state). It does none attestation and ES256 assertions with a stored P-256 key. PRF is HMAC-SHA-256 under a stored per-credential secret over SHA-256("WebAuthn PRF\0" || input). It supports eval and evalByCredential and returns prf.enabled on create. It returns real PublicKeyCredential and response prototypes, so the app's instanceof checks pass.
State is persisted in ~claude/.config/infraternal-test-admin/ (dir 0700, files 0600): authenticator.json (key, PRF secret, credential id, user handle, counter) and recovery-key.txt. The counter is incremented and saved on every assertion. The stale J140 files (credentials.json, storage-state.json) and the spent setup token are deleted.
test-admin-signin.ts now signs in from a fresh browser profile with it (--unlock-ca optional). j141-setup.ts is the one-shot setup script.

Proof (same stored credential, fresh browser each time)

Setup with the setup token, passkey, 24-word recovery key, then CA unlock in the same run: ok.
Sign out (palette, confirm), login screen shown, fresh sign-in with passkey + PRF: ok. /remote/ca/status is initialized:true, locked:false.
systemctl restart infraternal-test (log: server_ca_startup outcome="cold"), fresh sign-in: status locked:true and the toast "The server restarted. Confirm your passkey once to reconnect your hosts." Unlock via the palette row with the stored credential: "Host connections unlocked", status locked:false, Settings > Hosts shows "Unlocked".

Server config change you need to know about (owner-approved in this job)
The test unit had no --remote-listen / --remote-ca-dir, so /remote/ca/* was not mounted (POST answered 405) and the CA could never be unlocked. This is why J140 saw no CA row. I added a drop-in on 10.69.69.89, test unit only: /etc/systemd/system/infraternal-test.service.d/remote-ca.conf. It repeats the ExecStart and adds --remote-listen 10.69.69.89:7444 --remote-ca-dir /var/lib/infraternal-test/remote-ca. I added no firewall rule for 7444 (D291). The data dir /var/lib/infraternal-test/remote-ca is 0700 infraternal-test; recreate it after any data-dir reset.
Removal steps: rm /etc/systemd/system/infraternal-test.service.d/remote-ca.conf; rmdir /etc/systemd/system/infraternal-test.service.d; systemctl daemon-reload; systemctl restart infraternal-test. Delete /var/lib/infraternal-test/remote-ca with the instance.

Observation (possible product issue, not fixed)
Right after first-run setup the palette shows no "Unlock host connections" row, although /remote/ca/status already says initialized:false, locked:true. After a page reload the row appears. Repro: reset an empty instance with --remote-listen set, run setup, open the palette (Ctrl+Shift+K), type "Unlock": no match until reload. Not investigated further.

Toast "This terminal request is not valid. Try opening a new window."

Evidence: a WebSocket trace of the first canvas load. The client adds the console view (canvas.layout.apply), and about 1.6 s later the console sends ssh.session.open{}. About 160 ms later the server answers {"error":"terminal unavailable","reason":"invalid"}. The client maps reason: invalid to that toast (apps/tracer/src/terminal-refusal.ts:13).
Not a bad request: on dev ssh.session.open with {} is valid in websocket.rs (BrowserTerminalOpen { host_id: Option }, fixed 80x24), so the invalid comes from the infraternal agent. crates/infraternal-agent/src/daemon.rs:291 maps PermissionDenied/InvalidInput/InvalidData from the open to OpenRefusal::Invalid.
Most likely cause (not proven): the embedded infraternal agent on the test server runs rootless as the service user infraternal-test (uid 994, nologin shell). logins::account (crates/infraternal-agent/src/logins.rs:~35-60) resolves the person's name orchestrator as a host account. There is no such OS user, and a non-root agent returns PermissionDenied ("preferred account is absent on a rootless host") instead of falling back as root does. Production runs as root, where the missing account falls back to root with a note. I did not create an OS user or change the unit to test this, and the journal has no line for it.
Product angle: a rootless embedded host with a person whose name is not an OS account gets a "not valid" toast, which is the wrong wording and gives no way forward. It also appears on every first canvas load, because the default console view opens a shell on its own.
Repro: fresh instance run as a non-root service user, set up an admin whose username is not an OS account, open the canvas once. Same on current dev; the test server runs ff93201e5.
A cheap check for whoever picks it up: create an OS user named orchestrator on the test host and reload. If the shell opens, the cause is confirmed.

Evidence screenshots attached below (1920x1080 desktop, no secrets).

J141 report (no secrets below). **Result: done and proven.** The admin `orchestrator` ("Orchestrator (Claude)") exists on https://test.ssh.kayg.org with a durable software authenticator, and the machine CA is unlocked. **What I did** 1. Reset the empty test instance (stop, move `/var/lib/infraternal-test`, start). I did this three times while diagnosing. Production (`infraternal-server`, `/var/lib/infraternal`) was never touched and is still active. All three backup dirs were empty setup-only instances and are deleted now that the new setup is proven. 2. Test-only software authenticator, outside the repo, on compute01 in `~claude/infraternal-ops/`: `sw-authenticator-page.js` (injected with `addInitScript`, test origin only) and `sw-authenticator.ts` (loads/saves state). It does `none` attestation and ES256 assertions with a stored P-256 key. PRF is HMAC-SHA-256 under a stored per-credential secret over SHA-256("WebAuthn PRF\0" || input). It supports `eval` and `evalByCredential` and returns `prf.enabled` on create. It returns real `PublicKeyCredential` and response prototypes, so the app's `instanceof` checks pass. 3. State is persisted in `~claude/.config/infraternal-test-admin/` (dir 0700, files 0600): `authenticator.json` (key, PRF secret, credential id, user handle, counter) and `recovery-key.txt`. The counter is incremented and saved on every assertion. The stale J140 files (`credentials.json`, `storage-state.json`) and the spent setup token are deleted. 4. `test-admin-signin.ts` now signs in from a fresh browser profile with it (`--unlock-ca` optional). `j141-setup.ts` is the one-shot setup script. **Proof (same stored credential, fresh browser each time)** - Setup with the setup token, passkey, 24-word recovery key, then CA unlock in the same run: ok. - Sign out (palette, confirm), login screen shown, fresh sign-in with passkey + PRF: ok. `/remote/ca/status` is `initialized:true, locked:false`. - `systemctl restart infraternal-test` (log: `server_ca_startup outcome="cold"`), fresh sign-in: status `locked:true` and the toast "The server restarted. Confirm your passkey once to reconnect your hosts." Unlock via the palette row with the stored credential: "Host connections unlocked", status `locked:false`, Settings > Hosts shows "Unlocked". **Server config change you need to know about (owner-approved in this job)** The test unit had no `--remote-listen` / `--remote-ca-dir`, so `/remote/ca/*` was not mounted (POST answered 405) and the CA could never be unlocked. This is why J140 saw no CA row. I added a drop-in on 10.69.69.89, test unit only: `/etc/systemd/system/infraternal-test.service.d/remote-ca.conf`. It repeats the `ExecStart` and adds `--remote-listen 10.69.69.89:7444 --remote-ca-dir /var/lib/infraternal-test/remote-ca`. I added no firewall rule for 7444 (D291). The data dir `/var/lib/infraternal-test/remote-ca` is 0700 `infraternal-test`; recreate it after any data-dir reset. **Removal steps:** `rm /etc/systemd/system/infraternal-test.service.d/remote-ca.conf; rmdir /etc/systemd/system/infraternal-test.service.d; systemctl daemon-reload; systemctl restart infraternal-test`. Delete `/var/lib/infraternal-test/remote-ca` with the instance. **Observation (possible product issue, not fixed)** Right after first-run setup the palette shows no "Unlock host connections" row, although `/remote/ca/status` already says `initialized:false, locked:true`. After a page reload the row appears. Repro: reset an empty instance with `--remote-listen` set, run setup, open the palette (Ctrl+Shift+K), type "Unlock": no match until reload. Not investigated further. **Toast "This terminal request is not valid. Try opening a new window."** - Evidence: a WebSocket trace of the first canvas load. The client adds the `console` view (`canvas.layout.apply`), and about 1.6 s later the console sends `ssh.session.open{}`. About 160 ms later the server answers `{"error":"terminal unavailable","reason":"invalid"}`. The client maps `reason: invalid` to that toast (`apps/tracer/src/terminal-refusal.ts:13`). - Not a bad request: on dev `ssh.session.open` with `{}` is valid in `websocket.rs` (`BrowserTerminalOpen { host_id: Option }`, fixed 80x24), so the `invalid` comes from the infraternal agent. `crates/infraternal-agent/src/daemon.rs:291` maps `PermissionDenied`/`InvalidInput`/`InvalidData` from the open to `OpenRefusal::Invalid`. - Most likely cause (not proven): the embedded infraternal agent on the test server runs rootless as the service user `infraternal-test` (uid 994, nologin shell). `logins::account` (`crates/infraternal-agent/src/logins.rs:~35-60`) resolves the person's name `orchestrator` as a host account. There is no such OS user, and a non-root agent returns `PermissionDenied` ("preferred account is absent on a rootless host") instead of falling back as root does. Production runs as root, where the missing account falls back to root with a note. I did not create an OS user or change the unit to test this, and the journal has no line for it. - Product angle: a rootless embedded host with a person whose name is not an OS account gets a "not valid" toast, which is the wrong wording and gives no way forward. It also appears on every first canvas load, because the default console view opens a shell on its own. - Repro: fresh instance run as a non-root service user, set up an admin whose username is not an OS account, open the canvas once. Same on current dev; the test server runs `ff93201e5`. - A cheap check for whoever picks it up: create an OS user named `orchestrator` on the test host and reload. If the shell opens, the cause is confirmed. Evidence screenshots attached below (1920x1080 desktop, no secrets).

warm-signed-in-canvas-desktop.png

71 KiB

warm-ca-unlocked-settings-desktop.png

110 KiB

signed-out-desktop.png

36 KiB

cold-locked-toast-desktop.png

79 KiB

cold-ca-unlocked-settings-desktop.png

114 KiB