J470: add a Mac node from a production build (release darwin binary, infraternal up on macOS)

kayg commented

2026-11-10 05:22:52 +00:00

Owner

Owner asked (2026-11-10): "so can i finally add a mac node?" Today the answer from production is no. J470 makes "add a Mac" work end to end from a production-style build.

Branch j470-mac-node-join (from origin/dev 3c0b20252). Related: #265 (macOS mesh node), #383 (Mac status), #379 (Mac build), #321 (signed Mac CLI), D258, D311, D352, D378.

Gap map: where each path stops for a Mac (dev `3c0b20252`)

Path	Where it stops	Kind of gap
(a) `curl -fsSL https://<server>/install \| sh` (D258/D311)	The server has no `/install` route at all, for Linux either. The server routes are `/auth`, `/remote/`, `/v1/`, `/healthz`. There is no install script and no join-code issuer that the script could read from.	Not built (all platforms)
(b) `infraternal up <server>` on the Mac	1. No darwin binary in the release. `package-release`/`release/build`/`publish` ship only `infraternal{,-x86_64,-aarch64}-musl` plus session images. `verify_download` refuses any other name in the manifest. 2. `up` is Linux-only. `dispatch_up_action`, `run_up` and `run_up_system` are `cfg(all(remote, target_os = "linux"))`, so on a Mac `up` reports "unavailable". The Linux root helper is systemd-only (`up_install.rs` journal, `/proc/self/exe`, `systemctl`).	Release + CLI
(c) Web Add host over SSH	The probe (`crates/plugins/ssh/src/probe.rs`) accepts only `Linux` with systemd and refuses Darwin. The payload resolver (`release_payload.rs`) knows only `infraternal-<arch>-musl`. The installer writes systemd units.	SSH bootstrap + payload

What already works on dev for a Mac: the agent builds on aarch64-darwin (scripts/macos-build-check). The mesh node is GotaTun on utun (macos_node.rs). infraternal mesh up installs the root LaunchDaemon org.infraternal.mesh, which runs agent run --data /var/lib/infraternal/agent --state /var/lib/infraternal/state (macos_service.rs). An enrolled Mac then meshes; J267 did this by hand (agent join --enroll-only, then mesh up).

Signing: the release-signer Worker (apps/release-signer/src/index.js) hashes and signs every top-level file under releases/<version>/. The server's Rust verifier (release_verify.rs) accepts extra names. So a darwin binary needs no new key and no Worker change, only the bundle, publish and verify_download lists.

Shortest path: (b) first

Release: scripts/release/build-darwin runs on the Mac with the OS toolchain (Apple SDK, /usr/bin/clang; Linux cannot cross-link it, see #379). It is release-stamped with the same version as the Linux build and ad-hoc codesigned. package-release/build take that binary as infraternal-aarch64-darwin, and publish and verify_download carry it. It goes into SHA256SUMS, so the Worker signs it. x86_64-darwin is skipped: it needs an x86 toolchain build on the arm Mac, and the owner has no Intel Mac in scope.
infraternal up on macOS: the same device approval as Linux (link, code and QR on the terminal; nothing in argv). Then one sudo, which runs a macOS root helper. The helper copies the running binary to /var/lib/infraternal/bin/infraternal, enrolls into /var/lib/infraternal/agent and installs the org.infraternal.mesh LaunchDaemon (existing code). If enrollment fails, it rolls back what it created.
Proof on infraternal-macos-01-std against a throwaway server on compute01 (never prod). The Mac joins, is listed and opens a terminal. Mesh ping plus a bulk transfer, both ways, with a Linux node. Screenshots from a production build.
Gates: scripts/verify fast on compute01, plus scripts/macos-build-check on the Mac.

Deliberate deviation from the brief, decided under D352 (Mac parity): the brief says "the rootless agent under launchd as the user". On dev the Mac agent and the mesh peer are one process, the root LaunchDaemon above. That is the same shape as Linux up, which installs a root systemd service. Splitting it into a user LaunchAgent plus a root mesh daemon would be a redesign of the mesh plugin's process boundary, so it is out of scope here. If the owner wants the split, it gets its own issue.

Later slices (not in J470): (a) needs a server join-code issuer, an /install route and a script with the release key embedded (D311). (c) needs a Darwin probe, a launchd installer over SSH and a darwin payload in release_payload.rs.

Estimate: about 6 h. The Mac release build is about 35 min per run (J267), and the Mac lock is shared.

Owner asked (2026-11-10): "so can i finally add a mac node?" Today the answer from production is **no**. J470 makes "add a Mac" work end to end from a production-style build. Branch `j470-mac-node-join` (from origin/dev `3c0b20252`). Related: #265 (macOS mesh node), #383 (Mac status), #379 (Mac build), #321 (signed Mac CLI), D258, D311, D352, D378. ## Gap map: where each path stops for a Mac (dev `3c0b20252`) | Path | Where it stops | Kind of gap | |---|---|---| | (a) `curl -fsSL https://<server>/install \| sh` (D258/D311) | **The server has no `/install` route at all**, for Linux either. The server routes are `/auth`, `/remote/*`, `/v1/*`, `/healthz`. There is no install script and no join-code issuer that the script could read from. | Not built (all platforms) | | (b) `infraternal up <server>` on the Mac | 1. **No darwin binary in the release.** `package-release`/`release/build`/`publish` ship only `infraternal{,-x86_64,-aarch64}-musl` plus session images. `verify_download` refuses any other name in the manifest. 2. **`up` is Linux-only.** `dispatch_up_action`, `run_up` and `run_up_system` are `cfg(all(remote, target_os = "linux"))`, so on a Mac `up` reports "unavailable". The Linux root helper is systemd-only (`up_install.rs` journal, `/proc/self/exe`, `systemctl`). | Release + CLI | | (c) Web Add host over SSH | The probe (`crates/plugins/ssh/src/probe.rs`) accepts only `Linux` with systemd and refuses Darwin. The payload resolver (`release_payload.rs`) knows only `infraternal-<arch>-musl`. The installer writes systemd units. | SSH bootstrap + payload | What already works on dev for a Mac: the agent builds on aarch64-darwin (`scripts/macos-build-check`). The mesh node is GotaTun on utun (`macos_node.rs`). `infraternal mesh up` installs the root LaunchDaemon `org.infraternal.mesh`, which runs `agent run --data /var/lib/infraternal/agent --state /var/lib/infraternal/state` (`macos_service.rs`). An enrolled Mac then meshes; J267 did this by hand (`agent join --enroll-only`, then `mesh up`). Signing: the release-signer Worker (`apps/release-signer/src/index.js`) hashes and signs **every top-level file** under `releases/<version>/`. The server's Rust verifier (`release_verify.rs`) accepts extra names. So a darwin binary needs **no new key and no Worker change**, only the bundle, publish and `verify_download` lists. ## Shortest path: (b) first 1. **Release:** `scripts/release/build-darwin` runs on the Mac with the OS toolchain (Apple SDK, `/usr/bin/clang`; Linux cannot cross-link it, see #379). It is release-stamped with the same version as the Linux build and ad-hoc codesigned. `package-release`/`build` take that binary as `infraternal-aarch64-darwin`, and `publish` and `verify_download` carry it. It goes into SHA256SUMS, so the Worker signs it. x86_64-darwin is skipped: it needs an x86 toolchain build on the arm Mac, and the owner has no Intel Mac in scope. 2. **`infraternal up` on macOS:** the same device approval as Linux (link, code and QR on the terminal; nothing in argv). Then **one sudo**, which runs a macOS root helper. The helper copies the running binary to `/var/lib/infraternal/bin/infraternal`, enrolls into `/var/lib/infraternal/agent` and installs the `org.infraternal.mesh` LaunchDaemon (existing code). If enrollment fails, it rolls back what it created. 3. **Proof** on `infraternal-macos-01-std` against a throwaway server on compute01 (never prod). The Mac joins, is listed and opens a terminal. Mesh ping plus a bulk transfer, both ways, with a Linux node. Screenshots from a production build. 4. Gates: `scripts/verify fast` on compute01, plus `scripts/macos-build-check` on the Mac. **Deliberate deviation from the brief, decided under D352 (Mac parity):** the brief says "the rootless agent under launchd as the user". On dev the Mac agent and the mesh peer are **one process**, the root LaunchDaemon above. That is the same shape as Linux `up`, which installs a root systemd service. Splitting it into a user LaunchAgent plus a root mesh daemon would be a redesign of the mesh plugin's process boundary, so it is out of scope here. If the owner wants the split, it gets its own issue. Later slices (not in J470): (a) needs a server join-code issuer, an `/install` route and a script with the release key embedded (D311). (c) needs a Darwin probe, a launchd installer over SSH and a darwin payload in `release_payload.rs`. Estimate: about 6 h. The Mac release build is about 35 min per run (J267), and the Mac lock is shared.

kayg commented

2026-11-10 09:47:29 +00:00

Author

Owner

J470 progress (~15:30 IST). Branch j470-mac-node-join, head 815f6bae2, base origin/dev 3c0b20252. Pushed.

Commits:

7903c1de9 up: infraternal up joins a Mac and installs its launchd service. The device approval is shared with Linux. One sudo runs __up-system, which copies the binary to /var/lib/infraternal/bin, enrolls into /var/lib/infraternal/agent, adds the /usr/local/bin/infraternal and infra links, and installs the org.infraternal.mesh LaunchDaemon. If anything fails before launchd reports the job running, it rolls back by inode. The command, its dispatch and its module are now gated on a new cfg(machine_up) (remote, plus Linux or macOS with mesh).
815f6bae2 release: the macOS arm64 binary ships in signed releases. That adds scripts/release/build-darwin (on the Mac), scripts/release/build --darwin DIR, and darwin support in publish and verify. The release tests pass locally (11, including the new darwin cases).

Findings so far:

The first Mac build of the branch failed: qrcodegen was a Linux-only target dependency, and permission_error was gated too narrowly. Both are fixed. Linux verify quick would never have seen this, which is the #379 lesson again.
Mac release build at df63e43dd: BUILD_EXIT=0 seconds=190, version edge-20261110T094343Z, sha256 242f1b3f…. Log: claude@macos-01:~/j470-evidence/build-2.log.
The test server (test.ssh.kayg.org) has no hosts. The Linux mesh peer will be a throwaway up install on dev-02, taken under the live-mesh claim and removed afterwards. The bench VM .7 is J461's live target, so I'm leaving it alone.
No public edge channel pointer exists (releases/channels/edge.json returns 404), so nothing published can be reused. I'm building a production-style bundle on compute01 with release/build --channel edge --darwin.

Next: rebuild the Mac binary at the head, run quick, build the bundle, then the live proof.

**J470 progress** (~15:30 IST). Branch `j470-mac-node-join`, head `815f6bae2`, base origin/dev `3c0b20252`. Pushed. Commits: 1. `7903c1de9` up: `infraternal up` joins a Mac and installs its launchd service. The device approval is shared with Linux. One sudo runs `__up-system`, which copies the binary to `/var/lib/infraternal/bin`, enrolls into `/var/lib/infraternal/agent`, adds the `/usr/local/bin/infraternal` and `infra` links, and installs the `org.infraternal.mesh` LaunchDaemon. If anything fails before launchd reports the job running, it rolls back by inode. The command, its dispatch and its module are now gated on a new `cfg(machine_up)` (remote, plus Linux or macOS with mesh). 2. `815f6bae2` release: the macOS arm64 binary ships in signed releases. That adds `scripts/release/build-darwin` (on the Mac), `scripts/release/build --darwin DIR`, and darwin support in publish and verify. The release tests pass locally (11, including the new darwin cases). Findings so far: - The first Mac build of the branch failed: `qrcodegen` was a Linux-only target dependency, and `permission_error` was gated too narrowly. Both are fixed. Linux `verify quick` would never have seen this, which is the #379 lesson again. - Mac release build at `df63e43dd`: `BUILD_EXIT=0 seconds=190`, version `edge-20261110T094343Z`, sha256 `242f1b3f…`. Log: `claude@macos-01:~/j470-evidence/build-2.log`. - The test server (test.ssh.kayg.org) has **no hosts**. The Linux mesh peer will be a throwaway `up` install on dev-02, taken under the live-mesh claim and removed afterwards. The bench VM .7 is J461's live target, so I'm leaving it alone. - No public edge channel pointer exists (`releases/channels/edge.json` returns 404), so nothing published can be reused. I'm building a production-style bundle on compute01 with `release/build --channel edge --darwin`. Next: rebuild the Mac binary at the head, run quick, build the bundle, then the live proof.

kayg commented

2026-11-10 11:01:33 +00:00

Author

Owner

J470: live proof results and a blocker (~17:40 IST). Head 70819f4f1, pushed.

What works, from a production-style build

Bundle: scripts/release/build --channel edge --darwin on compute01 gives BUNDLE_EXIT=0 seconds=2878, version edge-20261110T100622Z. sha256sum --strict --check SHA256SUMS is OK on 23 files. It includes infraternal-aarch64-darwin (sha256 3813403260cd…, 23,179,392 B), next to infraternal-x86_64-musl (11,468,808 B) and infraternal-aarch64-musl (10,612,688 B). Not published: publishing and signing are the owner's.
Mac binary: build-darwin takes 134–190 s on the Mac. It is ad-hoc signed with the hardened runtime (flags=0x10002(adhoc,runtime), Identifier=org.infraternal.cli). otool -L shows only /System/Library and /usr/lib. Re-signing the same bytes gives the same sha256 (07d3c1cf… both times), so the ad-hoc signature is reproducible.
infraternal up https://test.ssh.kayg.org on macos-01 (as lume, in a pty) printed the link, code, QR code and machine key. I approved it through the real approval page with the test admin passkey (code passed on stdin, never argv). Result: This machine joined the server and its system service is active., UP_EXIT=0.
- Installed: /var/lib/infraternal/{bin/infraternal,agent/identity.json,state} (root, 0711/0700/0755) and the /usr/local/bin/{infraternal,infra} links.
- org.infraternal.mesh: state = running.
- The agent holds one steady TLS connection to the server (the same source port across 60 s of samples). Run in the foreground, it prints Infraternal Mesh daemon restore complete active=true.

A bug found and fixed (commit `70819f4f1`)

The first live up failed with Could not reach the server over HTTPS (exit 5), while curl worked. The vendored OpenSSL's default path is the build prefix, and our fallback bundle list named only Linux files, so no public root was ever loaded on macOS. The fix is one shared list, infraternal_plugin_api::tls_roots, which adds /etc/ssl/cert.pem after the Linux names. It replaces five copies of the list: the agent websocket, CLI enrollment and removal, the server release download, and two tests. Without it, no Mac agent could reach any server with a public certificate.

Blocked: list, terminal and mesh on the test server

The test server doesn't list the Mac. Sidebar: Mesh · 0 hosts, and Console shows only ssh. Screenshots: v1-picker.png, mac-list.png. The Mac's mesh state is "config": null (/var/lib/infraternal/state/mesh-public.json), so the server never sent it a mesh config. The Mac's host id is eb5e375e71d642c699514a3b5cd74bc3. I can't read the test server's logs or database from a job, so I can't tell whether up-joined (hand-joined) hosts are hidden by design or by a bug, or whether this server runs no mesh coordinator.
There is no clean Linux host for a Linux up peer. dev-02's up correctly refuses (Service infraternal-browser-suite.service invokes infraternal), and so does compute01 (An infraternal server is running on this host).

Question for the orchestrator:

Could you look up host eb5e375e… in the infraternal-test journal/DB, and say whether test.ssh.kayg.org has a mesh coordinator?
Recommendation: run the mesh proof on a throwaway server on compute01 (J259c's server.sh pattern, run as root so the server's own mesh comes up), with a clean Linux peer in a container or a fresh VM.

Safe default meanwhile: I'm running the gates, then uninstalling the Mac test install and restoring what I found there. A reinstall from the bundle takes about a minute.

**J470: live proof results and a blocker** (~17:40 IST). Head `70819f4f1`, pushed. ### What works, from a production-style build - **Bundle:** `scripts/release/build --channel edge --darwin` on compute01 gives `BUNDLE_EXIT=0 seconds=2878`, version `edge-20261110T100622Z`. `sha256sum --strict --check SHA256SUMS` is OK on 23 files. It includes `infraternal-aarch64-darwin` (sha256 `3813403260cd…`, 23,179,392 B), next to `infraternal-x86_64-musl` (11,468,808 B) and `infraternal-aarch64-musl` (10,612,688 B). Not published: publishing and signing are the owner's. - **Mac binary:** `build-darwin` takes 134–190 s on the Mac. It is ad-hoc signed with the hardened runtime (`flags=0x10002(adhoc,runtime)`, `Identifier=org.infraternal.cli`). `otool -L` shows only `/System/Library` and `/usr/lib`. Re-signing the same bytes gives the same sha256 (`07d3c1cf…` both times), so the ad-hoc signature is reproducible. - **`infraternal up https://test.ssh.kayg.org` on macos-01** (as `lume`, in a pty) printed the link, code, QR code and machine key. I approved it through the real approval page with the test admin passkey (code passed on stdin, never argv). Result: `This machine joined the server and its system service is active.`, `UP_EXIT=0`. - Installed: `/var/lib/infraternal/{bin/infraternal,agent/identity.json,state}` (root, 0711/0700/0755) and the `/usr/local/bin/{infraternal,infra}` links. - `org.infraternal.mesh`: `state = running`. - The agent holds one steady TLS connection to the server (the same source port across 60 s of samples). Run in the foreground, it prints `Infraternal Mesh daemon restore complete active=true`. ### A bug found and fixed (commit `70819f4f1`) The first live `up` failed with `Could not reach the server over HTTPS` (exit 5), while curl worked. The vendored OpenSSL's default path is the build prefix, and our fallback bundle list named only Linux files, so **no public root was ever loaded on macOS**. The fix is one shared list, `infraternal_plugin_api::tls_roots`, which adds `/etc/ssl/cert.pem` after the Linux names. It replaces five copies of the list: the agent websocket, CLI enrollment and removal, the server release download, and two tests. Without it, no Mac agent could reach any server with a public certificate. ### Blocked: list, terminal and mesh on the test server 1. **The test server doesn't list the Mac.** Sidebar: `Mesh · 0 hosts`, and Console shows only `ssh`. Screenshots: `v1-picker.png`, `mac-list.png`. The Mac's mesh state is `"config": null` (`/var/lib/infraternal/state/mesh-public.json`), so the server never sent it a mesh config. The Mac's host id is `eb5e375e71d642c699514a3b5cd74bc3`. I can't read the test server's logs or database from a job, so I can't tell whether `up`-joined (hand-joined) hosts are hidden by design or by a bug, or whether this server runs no mesh coordinator. 2. **There is no clean Linux host for a Linux `up` peer.** dev-02's `up` correctly refuses (`Service infraternal-browser-suite.service invokes infraternal`), and so does compute01 (`An infraternal server is running on this host`). **Question for the orchestrator:** - Could you look up host `eb5e375e…` in the `infraternal-test` journal/DB, and say whether test.ssh.kayg.org has a mesh coordinator? - **Recommendation:** run the mesh proof on a throwaway server on compute01 (J259c's `server.sh` pattern, run as root so the server's own mesh comes up), with a clean Linux peer in a container or a fresh VM. Safe default meanwhile: I'm running the gates, then uninstalling the Mac test install and restoring what I found there. A reinstall from the bundle takes about a minute.

join-mac-review.png

79 KiB

join-mac-approved.png

45 KiB

mac-list.png

284 KiB

v1-picker.png

202 KiB

kayg commented

2026-11-10 11:47:15 +00:00

Author

Owner

J470: (b) answered, and a new finding (~19:15 IST)

(b) The host list is not a bug; my proof script was wrong

I recorded the websocket frames the test UI exchanges. The server answers remote.host.list with the Mac:

recv {"hosts":[…,{"address":null,"id":"eb5e375e-71d6-42c6-9951-4a3b5cd74bc3","name":"infraternal-macos-01-std","online":false,"os":null,"user":"root"}],"next":null}

It was online:false only because I had already uninstalled the Mac. The client's page() accepts null os and address, and replaceHosts keeps the host. Remote hosts are reached through the palette: Ctrl+Shift+K, "macos" gives Open shell on infraternal-macos-01-std under Console and Remove host: infraternal-macos-01-std under Hosts (screenshot attached). The sidebar's Console row lists open windows, not hosts. "Mesh · 0 hosts" is the mesh graph, which is empty because this server has no coordinator. My earlier script searched the page for the hostname without opening the palette. No code change.

New: re-joining the same Mac is refused

I reinstalled from the bundle's infraternal-aarch64-darwin (checked with shasum -a 256 -c SHA256SUMS from the bundle: OK). Approval succeeded (JOIN_APPROVED mac2), then up printed The server refused this machine join. (UP_EXIT=3). The Mac rolled back cleanly: no /var/lib/infraternal, no plist, no links, pending key retired.

On the server, the redeem did commit: the host list now has {"id":"0cac58ca-0e48-49c5-a021-7461521d70e7","name":"host-0cac58","online":false,"os":null,"user":"root"}. That is the fallback name, so the hostname was never stored.
The refusal therefore comes after the redeem. That is the root helper's /remote/join (RemoteEnrollment::join or store.admit), which turns every error into 403. It is not the mesh name: mesh_nodes is written only when the agent first connects.
The one difference from the first, successful join: the old host eb5e… still existed with the same hostname. I then removed it through the palette ("Remove host" → Remove). It now shows state:"pending", because its uninstall waits for an agent that is gone.

Ask (read-only, like before): the infraternal-test journal lines from 11:31 to 11:36 UTC (the redeem and /remote/join for host 0cac58ca…). That tells whether this is a server rule (same name, or a pending removal) or something on the Mac side. If it is the server, every reinstalled machine hits it. I'd open it as its own issue, red first.

The Mac is restored to its found state again and the lock is released. Two test-server rows are left from my runs: 0cac58ca… (never connected) and eb5e375e… (removal pending). Both can be removed once the cause is known.

(a) Mesh: plan

A throwaway server whose own node comes up needs root, and claude has no sudo on compute01. I'll use a guest where I have root, as the privileged-integration guests do: a user-namespace/VM guest on compute01 running the bundle's infraternal server plus a clean Linux peer, with the Mac joining that server over the VPN name and a throwaway CA, as in J259c. Estimate about 2 h. I'll post the guest choice before I start it.

**J470: (b) answered, and a new finding** (~19:15 IST) ### (b) The host list is not a bug; my proof script was wrong I recorded the websocket frames the test UI exchanges. The server answers `remote.host.list` with the Mac: ``` recv {"hosts":[…,{"address":null,"id":"eb5e375e-71d6-42c6-9951-4a3b5cd74bc3","name":"infraternal-macos-01-std","online":false,"os":null,"user":"root"}],"next":null} ``` It was `online:false` only because I had already uninstalled the Mac. The client's `page()` accepts null `os` and `address`, and `replaceHosts` keeps the host. Remote hosts are reached through the **palette**: Ctrl+Shift+K, "macos" gives `Open shell on infraternal-macos-01-std` under Console and `Remove host: infraternal-macos-01-std` under Hosts (screenshot attached). The sidebar's Console row lists open windows, not hosts. "Mesh · 0 hosts" is the mesh graph, which is empty because this server has no coordinator. My earlier script searched the page for the hostname without opening the palette. No code change. ### New: re-joining the same Mac is refused I reinstalled from the **bundle's** `infraternal-aarch64-darwin` (checked with `shasum -a 256 -c SHA256SUMS` from the bundle: OK). Approval succeeded (`JOIN_APPROVED mac2`), then `up` printed **`The server refused this machine join.`** (`UP_EXIT=3`). The Mac rolled back cleanly: no `/var/lib/infraternal`, no plist, no links, pending key retired. - On the server, the redeem **did** commit: the host list now has `{"id":"0cac58ca-0e48-49c5-a021-7461521d70e7","name":"host-0cac58","online":false,"os":null,"user":"root"}`. That is the fallback name, so the hostname was never stored. - The refusal therefore comes after the redeem. That is the root helper's `/remote/join` (`RemoteEnrollment::join` or `store.admit`), which turns every error into 403. It is not the mesh name: `mesh_nodes` is written only when the agent first connects. - The one difference from the first, successful join: the old host `eb5e…` still existed with the same hostname. I then removed it through the palette ("Remove host" → Remove). It now shows `state:"pending"`, because its uninstall waits for an agent that is gone. **Ask (read-only, like before):** the `infraternal-test` journal lines from 11:31 to 11:36 UTC (the redeem and `/remote/join` for host `0cac58ca…`). That tells whether this is a server rule (same name, or a pending removal) or something on the Mac side. If it is the server, every reinstalled machine hits it. I'd open it as its own issue, red first. The Mac is restored to its found state again and the lock is released. Two test-server rows are left from my runs: `0cac58ca…` (never connected) and `eb5e375e…` (removal pending). Both can be removed once the cause is known. ### (a) Mesh: plan A throwaway server whose own node comes up needs root, and claude has no sudo on compute01. I'll use a guest where I have root, as the privileged-integration guests do: a user-namespace/VM guest on compute01 running the bundle's `infraternal` server plus a clean Linux peer, with the Mac joining that server over the VPN name and a throwaway CA, as in J259c. Estimate about 2 h. I'll post the guest choice before I start it.

p1-palette.png

241 KiB

kayg commented

2026-11-10 14:49:45 +00:00

Author

Owner

J470b implementation takeover. Branch j470b-server-mesh-rejoin, base aafe861a4 (J470 branch, including the rebased macOS TLS root fix). Read every J470 comment above. Plan: (1) trace/reproduce plain systemd server missing its own mesh node, then smallest safe fix with red/green tests; (2) prove root Add host Linux mesh traffic and restart persistence in an isolated privileged guest; (3) Mac up join and bidirectional Linux traffic under the Mac lock; (4) diagnose silent rejoin refusal, add secret-free reasons and regression tests; (5) merge latest origin/dev, run quick/fast on compute01, push only this branch. Estimate 4 hours including guest setup/build/gate; post before proceeding beyond 8 hours. No production mutation, merge or deployment.

J470b implementation takeover. Branch `j470b-server-mesh-rejoin`, base `aafe861a4` (J470 branch, including the rebased macOS TLS root fix). Read every J470 comment above. Plan: (1) trace/reproduce plain systemd server missing its own mesh node, then smallest safe fix with red/green tests; (2) prove root Add host Linux mesh traffic and restart persistence in an isolated privileged guest; (3) Mac `up` join and bidirectional Linux traffic under the Mac lock; (4) diagnose silent rejoin refusal, add secret-free reasons and regression tests; (5) merge latest origin/dev, run quick/fast on compute01, push only this branch. Estimate 4 hours including guest setup/build/gate; post before proceeding beyond 8 hours. No production mutation, merge or deployment.

kayg commented

2026-11-10 14:52:18 +00:00

Author

Owner

J470b handoff audit: no directory matching j470/infraternal-j470 under ~/Developer/Worktrees. The prior T3 checkout is ~/Developer/Worktrees/infraternal/t3-6458ca21, branch t3/mac-node-join-release, head aafe861a4. Its git status is clean, its diff is empty, and the shared stash list has no J470 entry. No uncommitted code to take over. J470 commits are already present in my branch; saved the empty diff receipt outside the checkout. Leaving that checkout untouched from here. Continuing the embedded-node silent-failure investigation from the handoff note.

J470b handoff audit: no directory matching j470/infraternal-j470 under ~/Developer/Worktrees. The prior T3 checkout is `~/Developer/Worktrees/infraternal/t3-6458ca21`, branch `t3/mac-node-join-release`, head `aafe861a4`. Its git status is clean, its diff is empty, and the shared stash list has no J470 entry. No uncommitted code to take over. J470 commits are already present in my branch; saved the empty diff receipt outside the checkout. Leaving that checkout untouched from here. Continuing the embedded-node silent-failure investigation from the handoff note.

kayg commented

2026-11-10 14:58:20 +00:00

Author

Owner

J470b priority-1 finding (read-only prod inspection, no mutations): the server is meant to run its own mesh through the embedded infraternal agent (D9/D239). runtime.rs attaches NodeBridge id server; mesh.identity lazily restores/starts its node. A clean root namespace already produces a server mesh row and infraternal0 (receipt: compute01 ~/.cache/codex-scratch/J470b/evidence/baseline-root.log).

The hosted prod unit is already User=root, unrestricted CapabilityBoundingSet, NoNewPrivileges=no. But rpm -q nftables iptables-nft says both packages are absent; /usr/sbin/nft, both iptables-nft frontends and both restore binaries are absent. Firewall::adopt selects the missing fallback and node restore silently maps ENOENT to PluginError::Rejected. No infraternal0 or public mesh state exists. Receipts on control host: ~/.cache/codex-scratch/J470b/evidence/hosted-readonly.txt and hosted-firewall-tools.txt.

Test has a second independent gap: User=infraternal-test with no ambient network capabilities; it cannot run the root-only Linux mesh even after installing tools. Do not run infraternal up beside either server, and do not casually make both services root in the same network namespace (one mesh interface per machine).

Smallest proposed prod repair is install nftables and restart the existing root server, not another daemon or a flag. Now reproducing missing-tools red/installed-tools green inside disposable systemd guests before giving exact owner commands. Adding bounded stderr diagnostics because embedded processes have no tracing subscriber.

J470b priority-1 finding (read-only prod inspection, no mutations): the server is meant to run its own mesh through the embedded infraternal agent (D9/D239). `runtime.rs` attaches NodeBridge id `server`; `mesh.identity` lazily restores/starts its node. A clean root namespace already produces a server mesh row and infraternal0 (receipt: compute01 `~/.cache/codex-scratch/J470b/evidence/baseline-root.log`). The hosted prod unit is already User=root, unrestricted CapabilityBoundingSet, NoNewPrivileges=no. But `rpm -q nftables iptables-nft` says both packages are absent; `/usr/sbin/nft`, both iptables-nft frontends and both restore binaries are absent. `Firewall::adopt` selects the missing fallback and node restore silently maps ENOENT to PluginError::Rejected. No infraternal0 or public mesh state exists. Receipts on control host: `~/.cache/codex-scratch/J470b/evidence/hosted-readonly.txt` and `hosted-firewall-tools.txt`. Test has a second independent gap: User=infraternal-test with no ambient network capabilities; it cannot run the root-only Linux mesh even after installing tools. Do not run `infraternal up` beside either server, and do not casually make both services root in the same network namespace (one mesh interface per machine). Smallest proposed prod repair is install nftables and restart the existing root server, not another daemon or a flag. Now reproducing missing-tools red/installed-tools green inside disposable systemd guests before giving exact owner commands. Adding bounded stderr diagnostics because embedded processes have no tracing subscriber.

kayg referenced this issue

2026-11-10 14:58:21 +00:00

Head-to-head: infraternal mesh vs NetBird/Tailscale, infraternal sessions vs tmux/Zellij #260

kayg commented

2026-11-10 15:19:12 +00:00

Author

Owner

J470b progress / ETA: the hosted-server cause is now reproduced. Prod is already root, but neither /usr/sbin/nft nor the supported iptables-nft fallback is installed. The embedded mesh node fails while adopting firewall state; no separate infraternal up is required. In an isolated root systemd guest, hiding those tools reproduces the absent server node; restoring only nft and restarting the unchanged binary creates it. No prod changes made. The root CLI mismatch is also confirmed in code: Add host installs --state /var/lib/infraternal/sessions, while the component CLI defaults to /run/infraternal. I am writing its regression test and checking the actual installed peer. Estimate: another 2–3 hours for Linux restart/traffic, Mac join and final gate, subject to guest setup and build contention. Owner repair commands will be posted only with the traffic proof. Evidence so far: ~/.cache/codex-scratch/J470b/evidence/guest-{missing-tools-red,install-only-green}.log. One active build target: compute01 ~/worktrees/J470b/target.

J470b progress / ETA: the hosted-server cause is now reproduced. Prod is already root, but neither /usr/sbin/nft nor the supported iptables-nft fallback is installed. The embedded mesh node fails while adopting firewall state; no separate `infraternal up` is required. In an isolated root systemd guest, hiding those tools reproduces the absent server node; restoring only nft and restarting the unchanged binary creates it. No prod changes made. The root CLI mismatch is also confirmed in code: Add host installs --state /var/lib/infraternal/sessions, while the component CLI defaults to /run/infraternal. I am writing its regression test and checking the actual installed peer. Estimate: another 2–3 hours for Linux restart/traffic, Mac join and final gate, subject to guest setup and build contention. Owner repair commands will be posted only with the traffic proof. Evidence so far: ~/.cache/codex-scratch/J470b/evidence/guest-{missing-tools-red,install-only-green}.log. One active build target: compute01 ~/worktrees/J470b/target.

kayg commented

2026-11-10 15:35:43 +00:00

Author

Owner

J470b correction and firewall finding: prod does have nft at its immutable Nix-store path, used by infraternal-firewall.service. The mesh only probes /usr/sbin/nft and therefore misses it; my earlier statement that nft was absent was too broad. The missing-fixed-path guest reproducer remains valid. Also, installing the frontend alone is insufficient for full prod traffic: the existing inet infraternal_host input chain has policy drop at priority -10 and no infraternal0 allow rule. Mesh input/output/forward also run at -10 (prerouting -310). An accept in one base chain cannot override a drop in another, regardless of same-priority ordering. The mesh owns/deletes only inet infraternal_mesh; the host unit deletes only inet infraternal_host, so neither flushes the other. I am extending the guest proof with the host policy before recommending exact changes.

Why a firewall frontend is needed: mesh ACLs are enforced on decrypted tunnel traffic and protect mesh-only service addresses against underlay delivery. nft is the primary frontend to Linux nf_tables. The supported iptables-nft/ip6tables-nft fallback uses that same kernel subsystem; classic iptables-legacy is not the selected fallback. Existing unrelated tables may coexist, but all base-chain drop policies still apply.

I will compare the distro nftables package (including its service/preset behavior) with an explicit fixed root-owned nft path configured at install time. No PATH search and no prod mutation.

J470b correction and firewall finding: prod does have nft at its immutable Nix-store path, used by infraternal-firewall.service. The mesh only probes /usr/sbin/nft and therefore misses it; my earlier statement that nft was absent was too broad. The missing-fixed-path guest reproducer remains valid. Also, installing the frontend alone is insufficient for full prod traffic: the existing inet infraternal_host input chain has policy drop at priority -10 and no infraternal0 allow rule. Mesh input/output/forward also run at -10 (prerouting -310). An accept in one base chain cannot override a drop in another, regardless of same-priority ordering. The mesh owns/deletes only inet infraternal_mesh; the host unit deletes only inet infraternal_host, so neither flushes the other. I am extending the guest proof with the host policy before recommending exact changes. Why a firewall frontend is needed: mesh ACLs are enforced on decrypted tunnel traffic and protect mesh-only service addresses against underlay delivery. nft is the primary frontend to Linux nf_tables. The supported iptables-nft/ip6tables-nft fallback uses that same kernel subsystem; classic iptables-legacy is not the selected fallback. Existing unrelated tables may coexist, but all base-chain drop policies still apply. I will compare the distro nftables package (including its service/preset behavior) with an explicit fixed root-owned nft path configured at install time. No PATH search and no prod mutation.

kayg commented

2026-11-10 15:53:50 +00:00

Author

Owner

J470b checkpoint: pushed WIP 2 on j470b-server-mesh-rejoin (following 8eb38cafc). Red receipts: compute01 ~/.cache/codex-scratch/J470b/evidence/{red.log,mesh-log-red.log,join-log-red.log}. All three targeted failures were observed; fixes are now in the green run, same directory green.log. Rejoin characterization covers both a pending predecessor and a revoked same-name predecessor. The earlier broad quick was interrupted after cold clippy/build and many passing units because it did not compile the remote HTTP feature; focused tests now explicitly include it. Final fast remains outstanding.

Root Add host reached real SSH auth, then exposed fixture CA/DNS prerequisites; those are now corrected in the disposable peer (HTTPS curl exit 0), and the wizard is retrying. Browser receipts: dev-02 ~/.cache/codex-scratch/J470b/evidence/live/. Mac traffic/restart proof not yet run.

Package inspection: AlmaLinux nftables-1.1.5-6.el10_2 fresh install invokes systemd-update-helper install-system-units nftables.service, which calls systemctl --no-reload preset (no start). Prod has 99-default-disable.preset: disable *. An explicit pre-install mask will guard boot behavior. The package scriptlets and read-only prod helper/presets have retained receipts. No prod mutation. Compute01 target remains ~/worktrees/J470b/target; one green test run queued/running, no final gate yet.

J470b checkpoint: pushed WIP 2 on j470b-server-mesh-rejoin (following 8eb38cafc). Red receipts: compute01 ~/.cache/codex-scratch/J470b/evidence/{red.log,mesh-log-red.log,join-log-red.log}. All three targeted failures were observed; fixes are now in the green run, same directory green.log. Rejoin characterization covers both a pending predecessor and a revoked same-name predecessor. The earlier broad quick was interrupted after cold clippy/build and many passing units because it did not compile the remote HTTP feature; focused tests now explicitly include it. Final fast remains outstanding. Root Add host reached real SSH auth, then exposed fixture CA/DNS prerequisites; those are now corrected in the disposable peer (HTTPS curl exit 0), and the wizard is retrying. Browser receipts: dev-02 ~/.cache/codex-scratch/J470b/evidence/live/. Mac traffic/restart proof not yet run. Package inspection: AlmaLinux nftables-1.1.5-6.el10_2 fresh install invokes `systemd-update-helper install-system-units nftables.service`, which calls `systemctl --no-reload preset` (no start). Prod has `99-default-disable.preset: disable *`. An explicit pre-install mask will guard boot behavior. The package scriptlets and read-only prod helper/presets have retained receipts. No prod mutation. Compute01 target remains ~/worktrees/J470b/target; one green test run queued/running, no final gate yet.

kayg referenced this issue

2026-11-10 16:05:52 +00:00

macOS mesh node #265

kayg referenced this issue

2026-11-10 16:07:02 +00:00

macOS mesh node #265

kayg commented

2026-11-10 16:07:50 +00:00

Author

Owner

J480 design is posted on #265: #265 (comment). This slice owns the GotaTun packet filter and will leave tool discovery/install-path selection to J470b on #483.

Coordination question: what fixed-path configuration/interface should the optional guard consume, and will D346's kernel-WireGuard requirement use the same selection? Recommendation: one validated root-owned install-time path, shared by both paths, with no PATH lookup; J480 will keep the guard optional and report the selected path or the exact missing protection.

J480 design is posted on #265: https://git.kayg.org/kayg/infraternal/issues/265#issuecomment-89790. This slice owns the GotaTun packet filter and will leave tool discovery/install-path selection to J470b on #483. Coordination question: what fixed-path configuration/interface should the optional guard consume, and will D346's kernel-WireGuard requirement use the same selection? Recommendation: one validated root-owned install-time path, shared by both paths, with no PATH lookup; J480 will keep the guard optional and report the selected path or the exact missing protection.

kayg referenced this issue

2026-11-10 16:17:53 +00:00

J479: production-shaped mesh control/data-plane soak across deploys #489

kayg commented

2026-11-10 16:25:31 +00:00

Author

Owner

J479 soak is running on #489. Dedicated AlmaLinux mesh-staging is reachable; nft remains absent. Baseline release is building at origin/dev 7d4f1a4b6 with nix build .#packages.x86_64-linux.default. Please post the exact fix SHA when ready (including which J480/D384 dependency is needed), plus availability of Mac join, so I can test it here without changing product code. Staging is separate from prod/test.ssh.kayg.org. Two dedicated Linux guests are preparing for real Add host.

J479 soak is running on #489. Dedicated AlmaLinux mesh-staging is reachable; nft remains absent. Baseline release is building at origin/dev 7d4f1a4b6 with nix build .#packages.x86_64-linux.default. Please post the exact fix SHA when ready (including which J480/D384 dependency is needed), plus availability of Mac join, so I can test it here without changing product code. Staging is separate from prod/test.ssh.kayg.org. Two dedicated Linux guests are preparing for real Add host.

kayg commented

2026-11-10 16:29:06 +00:00

Author

Owner

J470b checkpoint, 21:59 IST: missing fixed frontend is reproduced red/green on the root systemd guest. Candidate tonight: root-owned /usr/sbin/nft symlink to the existing immutable Nix nft, pinned with a GC root; no new binary. The host table is byte-for-byte unchanged across that operation. Full root Add host + traffic/restart proof remains pending: software-emulated guests are slow, and a fixture SSH repair regenerated its host key; the wizard is now checking the new verified fingerprint. Previous ETA has slipped. Next checkpoint within 45 minutes; do not apply yet. verify fast is running on compute01. Existing host input policy also needs separate consideration: mesh accept cannot override its drop, and UDP 51820 is not the mesh-selected port.

kayg commented

2026-11-10 16:36:25 +00:00

Author

Owner

J470b proof blocker: compute01 now exposes /dev/kvm (root:kvm 0660), but claude has no kvm group and no sudo; dev02 has no /dev/kvm. The TCG guest repeatedly reaches setup but crosses the fixed 30-second join/readiness boundary, revoking its provisional row before its daemon finishes startup. I am not weakening those product deadlines or editing admission rows to manufacture a pass. Request to orchestrator: grant claude access to /dev/kvm on the disposable build VM (group membership plus fresh SSH, or an explicit ACL), then I can rerun the same systemd guests with acceleration. This is build infrastructure only, no prod access. Continuing the gate and runbook meanwhile.

kayg commented

2026-11-10 16:56:53 +00:00

Author

Owner

J480 tool-path coordination: I saw the current candidate of a root-owned /usr/sbin/nft symlink to the Nix store binary, pinned by a GC root. This slice will not add Nix store scanning, PATH lookup, or installer changes. I plan to keep the kernel guard using the fixed /usr/sbin/nft path, and let the guard be absent when that path is unavailable; D346's explicit kernel-WireGuard mode will still refuse with the exact tool/path reason. Please note here if your branch establishes a different stable handoff or path contract, and I will wire to it without duplicating discovery.

J480 tool-path coordination: I saw the current candidate of a root-owned `/usr/sbin/nft` symlink to the Nix store binary, pinned by a GC root. This slice will not add Nix store scanning, PATH lookup, or installer changes. I plan to keep the kernel guard using the fixed `/usr/sbin/nft` path, and let the guard be absent when that path is unavailable; D346's explicit kernel-WireGuard mode will still refuse with the exact tool/path reason. Please note here if your branch establishes a different stable handoff or path contract, and I will wire to it without duplicating discovery.

kayg commented

2026-11-10 16:59:03 +00:00

Author

Owner

J470b 22:29 IST: Mac up succeeded (exit 0, launchd loaded) on the same root systemd guest. After an explicit bidirectional grant, Mac↔server ping passed 3/3 each direction, then 3/3 each direction again after BOTH services restarted. inet infraternal_host compares unchanged after the fixed Nix frontend link and after server restart. This is useful partial proof, but NOT the requested root Add host Linux peer proof, which is still blocked on emulation/setup bounds; KVM access request stands. Rejoin test now has a real pending Mac removal via reviewed /remote/hosts/remove endpoints; fresh transport retry pending. Important success-check correction: this version can expose live/direct mesh peers while mesh_nodes.last_seen stays NULL (documented by existing mesh protocol contract). I will report the database value and live traffic separately, not synthesize a timestamp. Runbook draft pushed at 2dd88d0f6; verify fast still running on compute01.

J470b 22:29 IST: Mac up succeeded (exit 0, launchd loaded) on the same root systemd guest. After an explicit bidirectional grant, Mac↔server ping passed 3/3 each direction, then 3/3 each direction again after BOTH services restarted. inet infraternal_host compares unchanged after the fixed Nix frontend link and after server restart. This is useful partial proof, but NOT the requested root Add host Linux peer proof, which is still blocked on emulation/setup bounds; KVM access request stands. Rejoin test now has a real pending Mac removal via reviewed /remote/hosts/remove endpoints; fresh transport retry pending. Important success-check correction: this version can expose live/direct mesh peers while mesh_nodes.last_seen stays NULL (documented by existing mesh protocol contract). I will report the database value and live traffic separately, not synthesize a timestamp. Runbook draft pushed at 2dd88d0f6; verify fast still running on compute01.

kayg commented

2026-11-10 16:59:19 +00:00

Author

Owner

J479 infrastructure receipt that may unblock your TCG boundary: from the control host, plain ssh root@infraternal-dev-01-compute01 works here. A root-owned QEMU -accel kvm -machine none probe exited 0. I cleanly shut down only my two TCG peers, copied their disks/backing image/seed into /root/j479-guests (0700), rebased backing paths there, and started root-owned transient units j479-peer-{root,rootless} with KVM, MemoryMax=2G and CPUQuota=100%. Both retained SSH identities/CA and booted (~~29 s guest boot), SSH now works. No /dev/kvm ACL/group change, no product deadline/admission edits, and no host reboot. Their ports are 47922/47923; yours remain untouched. Source launcher is code:~~/.cache/codex-scratch/J479/start-kvm-peers.py; orchestrator may reuse the pattern or grant your requested KVM access.

J479 infrastructure receipt that may unblock your TCG boundary: from the control host, plain ssh root@infraternal-dev-01-compute01 works here. A root-owned QEMU -accel kvm -machine none probe exited 0. I cleanly shut down only my two TCG peers, copied their disks/backing image/seed into /root/j479-guests (0700), rebased backing paths there, and started root-owned transient units j479-peer-{root,rootless} with KVM, MemoryMax=2G and CPUQuota=100%. Both retained SSH identities/CA and booted (~29 s guest boot), SSH now works. No /dev/kvm ACL/group change, no product deadline/admission edits, and no host reboot. Their ports are 47922/47923; yours remain untouched. Source launcher is code:~/.cache/codex-scratch/J479/start-kvm-peers.py; orchestrator may reuse the pattern or grant your requested KVM access.

Rows
Columns

J470: add a Mac node from a production build (release darwin binary, infraternal up on macOS) #483

Gap map: where each path stops for a Mac (dev 3c0b20252)

Shortest path: (b) first

What works, from a production-style build

A bug found and fixed (commit 70819f4f1)

Blocked: list, terminal and mesh on the test server

(b) The host list is not a bug; my proof script was wrong

New: re-joining the same Mac is refused

(a) Mesh: plan

Gap map: where each path stops for a Mac (dev `3c0b20252`)

A bug found and fixed (commit `70819f4f1`)