J479: production-shaped mesh control/data-plane soak across deploys

kayg commented

2026-11-10 16:17:53 +00:00

Owner

J479: production-shaped mesh soak, requested by the owner: “making sure it's super reliable on your test server with multiple deploys, testing back and forth”. Related #483, D384, D295.

Dedicated throwaway server: mesh-staging, AlmaLinux 10.2, VM 3170 on compute01, 2 vCPU / 3 GiB, SDN 10.70.3.170, NetBird 10.69.69.53. Keep nft absent for the initial red and D384 acceptance. Never mutate production or test.ssh.kayg.org. Only inspect production's unit read-only. Guest reboots are in scope; no physical-host reboot.

Plan and estimate: 1) access and read-only unit receipt, origin/dev release build via compute01 build-slot (up to 2 h including queue); 2) root systemd staging deployment with its own CA/origin and initial missing-server-node red (1 h); 3) real Add host root and rootless Linux peers, Mac up once #483 permits it (1–2 h); 4) apply posted J470b/D384 fix, run at least five measured complete cycles (2–3 h), then deploy each observed new dev merge and repeat. Total initial estimate 6–8 h, excluding external fix/access waits. Stop and report if active work exceeds 16 h. Two-hour progress tables while running. No product-code fixes here.

Every cycle records build SHA, join/rejoin, server mesh_nodes row, peer configs, infra mesh peers, every ordered peer pair's ping and 10-second checksum-verified bulk transfer, server and infraternal agent restarts, upgrade/rollback with database backup (D295), guest reboot, remove/re-add clean-state check. Blocked rows are not passes. Each failure gets its own finding and logs.

Evidence: code:/home/kayg/.cache/codex-scratch/J479/evidence/ and compute01:~/.cache/codex-scratch/J479/evidence/. Use a dedicated staging passkey state under J479 scratch, never the shared test-admin state. Release mesh claim immediately after live steps; Mac work under lockf -k /tmp/mac-bench.lock.

Initial access: plain SSH to NetBird reached an SSO-gated SSH service, SDN from code timed out. Maintainer is enabling access; retry in foreground up to 30 minutes and report on this issue. No server changes made.

J479: production-shaped mesh soak, requested by the owner: “making sure it's super reliable on your test server with multiple deploys, testing back and forth”. Related #483, D384, D295. Dedicated throwaway server: mesh-staging, AlmaLinux 10.2, VM 3170 on compute01, 2 vCPU / 3 GiB, SDN 10.70.3.170, NetBird 10.69.69.53. Keep nft absent for the initial red and D384 acceptance. Never mutate production or test.ssh.kayg.org. Only inspect production's unit read-only. Guest reboots are in scope; no physical-host reboot. Plan and estimate: 1) access and read-only unit receipt, origin/dev release build via compute01 build-slot (up to 2 h including queue); 2) root systemd staging deployment with its own CA/origin and initial missing-server-node red (1 h); 3) real Add host root and rootless Linux peers, Mac up once #483 permits it (1–2 h); 4) apply posted J470b/D384 fix, run at least five measured complete cycles (2–3 h), then deploy each observed new dev merge and repeat. Total initial estimate 6–8 h, excluding external fix/access waits. Stop and report if active work exceeds 16 h. Two-hour progress tables while running. No product-code fixes here. Every cycle records build SHA, join/rejoin, server mesh_nodes row, peer configs, infra mesh peers, every ordered peer pair's ping and 10-second checksum-verified bulk transfer, server and infraternal agent restarts, upgrade/rollback with database backup (D295), guest reboot, remove/re-add clean-state check. Blocked rows are not passes. Each failure gets its own finding and logs. Evidence: code:/home/kayg/.cache/codex-scratch/J479/evidence/ and compute01:~/.cache/codex-scratch/J479/evidence/. Use a dedicated staging passkey state under J479 scratch, never the shared test-admin state. Release mesh claim immediately after live steps; Mac work under lockf -k /tmp/mac-bench.lock. Initial access: plain SSH to NetBird reached an SSO-gated SSH service, SDN from code timed out. Maintainer is enabling access; retry in foreground up to 30 minutes and report on this issue. No server changes made.

kayg commented

2026-11-10 16:18:32 +00:00

Author

Owner

start: branch j479-mesh-soak, base/head 7d4f1a4b69. isolated checkouts code:~~/Developer/Worktrees/infraternal-j479 and compute01:~~/worktrees/J479. staging access works by plain SSH from compute01 over SDN; no SSO dependency. AlmaLinux 10.2, 2901 MiB RAM, nftables and iptables-nft RPMs absent, firewalld inactive. prod unit read-only receipt confirms root, no privilege restriction, server --data --listen --public-origin --web-root --remote-listen --remote-ca-dir --behind-proxy --trusted-proxy. Building the exact baseline using nix build .#packages.x86_64-linux.default through compute01 build-slot; log ~/.cache/codex-scratch/J479/evidence/build-baseline.log. Estimate remains 6–8 active hours plus queue/fix waits. No product code changed.

start: branch j479-mesh-soak, base/head 7d4f1a4b69ad9d4e68a9e848b8c0634bb39f1d0b. isolated checkouts code:~/Developer/Worktrees/infraternal-j479 and compute01:~/worktrees/J479. staging access works by plain SSH from compute01 over SDN; no SSO dependency. AlmaLinux 10.2, 2901 MiB RAM, nftables and iptables-nft RPMs absent, firewalld inactive. prod unit read-only receipt confirms root, no privilege restriction, server --data --listen --public-origin --web-root --remote-listen --remote-ca-dir --behind-proxy --trusted-proxy. Building the exact baseline using nix build .#packages.x86_64-linux.default through compute01 build-slot; log ~/.cache/codex-scratch/J479/evidence/build-baseline.log. Estimate remains 6–8 active hours plus queue/fix waits. No product code changed.

kayg commented

2026-11-10 16:23:58 +00:00

Author

Owner

access fingerprint clarification: SDN OpenSSH on 10.70.3.170 advertises ED25519 SHA256:vNLoWDck1VdmxosOnoPT1fqOPRdq8gnC8B2eQk/6f7U; this differs from the supplied SHA256:tOy8ebjlK4fCX5nsjEiOPhmyqOLwzn8aHxUhjEL39iQ. Suspect supplied key is NetBird SSH service rather than distro sshd. Checking the independent NetBird hostname path before further staging changes. Please confirm SDN sshd fingerprint; no NetBird bounce.

kayg commented

2026-11-10 16:24:35 +00:00

Author

Owner

fingerprint resolved by independent authenticated path: control-host SSH to mesh-staging.vpn.kayg.org succeeded with exactly the supplied tOy8... NetBird SSH key. Running sudo ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub there confirmed vNLo... as the distro sshd key. Both paths reach mesh-staging. SDN access is verified; continuing.

kayg referenced this issue

2026-11-10 16:25:31 +00:00

J470: add a Mac node from a production build (release darwin binary, infraternal up on macOS) #483

kayg commented

2026-11-10 16:25:48 +00:00

Author

Owner

coverage question for orchestrator: current dev gotatun_tun.rs:565 explicitly says creating mesh TUN requires CAP_NET_ADMIN and rootless supports only infraternal agent own traffic through the local proxy. I will exercise real rootless Add host and infra mesh peers, record actual results, and mark unsupported whole-machine ping/bulk rows blocked rather than grant capabilities and mislabel it rootless. Recommendation: require full pairwise data plane on privileged Linux/Mac peers, and rootless control/proxy evidence separately unless D384 adds true rootless TUN support. Browser fallback: preview_open failed with missing host browser libraries (libnspr4.so, libnss3.so etc.); use existing VM Playwright, no control-host browser install.

kayg commented

2026-11-10 16:30:53 +00:00

Author

Owner

orchestrator coverage approval incorporated: whole-machine rootless ping/bulk rows will be blocked: rootless has no TUN (gotatun_tun.rs:565), no capabilities granted. Browser now assigned to existing dev-02 Playwright only. New early slice: copied prod /etc/nftables/infraternal-host.nft read-only to code:~/.cache/codex-scratch/J479/evidence/prod-infraternal-host.nft. It has input priority filter - 10, policy drop, no infraternal0 allow, and eth0 UDP/51820 only (plus remote 7443 on wt0). Need first no-nft red before loading an adapted policy. I will report actual UDP listener, underlay reachability and decrypted traffic separately. Baseline release remains queued on compute01 (~12 min), not yet deployed.

kayg commented

2026-11-10 16:33:13 +00:00

Author

Owner

J479 preflight receipts attached: read-only production unit; staging OS/no-nft preflight; original production host firewall policy. Prepared staging policy and harness sources follow in a later receipt. No mesh cycle is claimed yet.

prod-unit-readonly.txt

1.1 KiB

staging-preflight.txt

1.1 KiB

kayg commented

2026-11-10 16:33:40 +00:00

Author

Owner

Prepared fixtures and adapted host policy attached as zip (Forgejo refused the standalone .nft attachment). This archive contains no credentials or authenticator state. The host policy has not been loaded yet; baseline no-nft release is still queued. Production preflight unit/OS receipts are attached to the preceding comment.

j479-prepared-fixtures.zip

6.5 KiB

kayg commented

2026-11-10 16:36:18 +00:00

Author

Owner

checkpoint 22:05 IST: staging root systemd unit installed but not started (artifact pending); private nginx TLS proxy returns expected 502 until backend arrives, nft RPM still absent. Two independent Debian guests on compute01: J479 root SSH at 10.70.3.22:47922, rootless at :47923; both accept root administration, rootless cloud-init READY and user j479 has no delegated privileges. Baseline build PID 353266 has waited ~18 min for mandatory compute01 slot. Browser dependencies resolve on dev-02 and J479 passkey state lives only under its own scratch/admin. No mesh claim or Mac lock held. Original/adapted firewall and fixture sources attached in j479-prepared-fixtures.zip; two public preflight receipts separately attached. Full cycles=0, all mesh rows pending baseline deployment.

kayg commented

2026-11-10 16:39:31 +00:00

Author

Owner

fixture update 22:10 IST: baseline nix build acquired slot 6/6 at compute01 local 17:37:15 (16:37:15 UTC); still running. TCG peers were slow (~80% of one host CPU each even during SSH), so moved only these fixtures after clean guest shutdown to KVM, which a root-owned disposable probe confirmed available. New root-owned disks: compute01:/root/j479-guests/; units j479-peer-root (PID 516029), j479-peer-rootless (PID 516724), each MemoryMax=2G/CPUQuota=100%. Old QEMU PIDs 410184/410190 exited. No shared live infraternal state changed; no host reboot. Both guests had READY, CA trust and staging hostname setup before migration. All product tests still pending release artifact.

kayg commented

2026-11-10 17:08:52 +00:00

Author

Owner

J479 checkpoint, ~22:40 IST. Baseline release acquired its slot after 1124 s, compiled the server, and is now building the x86 host payload. Full bundle not yet complete, so cycles remain 0. Staging still has no nft frontend and the adapted host policy remains unloaded until initial red. Both dedicated Linux guests run under KVM; rootless remains uid 1000 without delegated privileges. No Mac lock or shared mesh claim held. Updated private-fixture sources, exact units, KVM boot receipt, state inventory and build log attached (no credentials/authenticator state). Branch j479-mesh-soak remains pushed at 7d4f1a4b69, no repository code changed. Next: finish full bundle, deploy, collect missing server node, Add host and early separate host-policy traffic result. Estimate remains 6–8 active hours plus queue/fix waits.

J479 checkpoint, ~22:40 IST. Baseline release acquired its slot after 1124 s, compiled the server, and is now building the x86 host payload. Full bundle not yet complete, so cycles remain 0. Staging still has no nft frontend and the adapted host policy remains unloaded until initial red. Both dedicated Linux guests run under KVM; rootless remains uid 1000 without delegated privileges. No Mac lock or shared mesh claim held. Updated private-fixture sources, exact units, KVM boot receipt, state inventory and build log attached (no credentials/authenticator state). Branch j479-mesh-soak remains pushed at 7d4f1a4b69ad9d4e68a9e848b8c0634bb39f1d0b, no repository code changed. Next: finish full bundle, deploy, collect missing server node, Add host and early separate host-policy traffic result. Estimate remains 6–8 active hours plus queue/fix waits.

j479-checkpoint-v2.zip

33 KiB

Rows
Columns

J479: production-shaped mesh control/data-plane soak across deploys #489