kayg/infraternal
Watch
1
0
Fork
You've already forked infraternal
 0
Code Issues 356 Pull requests Projects Releases Packages Wiki Activity Actions

Mesh: Linux→Mac throughput collapses to <1 Mbit/s when the underlay MTU is below mesh MTU+overhead (fixed MTU, no PMTU/MSS clamp) #381

New issue
Open
opened 2026-11-06 02:34:41 +00:00 by kayg · 321 comments
kayg commented 2026-11-06 02:34:41 +00:00
Owner
Copy link

Found by J267 (#375). Severity P2 (performance; CLAUDE.md priority 1). Full data and commands: #375, comment "J267: measurement".

What: the mesh MTU is a constant, gotatun_tun.rs:30 MESH_MTU = 1380 (Linux) and macos_node.rs:49 UTUN_MTU = 1280 (Mac). Nothing derives it from the path to the peer's endpoint, and nothing clamps MSS. When the underlay MTU is below inner MTU + 60 (IPv4 outer) or + 80 (IPv6), every full-size mesh packet fragments outside the tunnel. Towards the Mac that collapses TCP.

Measured (Mac VM ↔ compute01; underlay NetBird, MTU 1280, about 143 ms RTT; dev 9b0c731e8; the same GotaTun fork on both ends):

1 flow 8 flows
Plain path, Linux → Mac 132.4 / 145.1 Mbit/s 374.1 / 529.4 Mbit/s
Mesh, Linux → Mac 0.5 (repeat: 0.2) 33.5
Mesh, Mac → Linux 119.2 125.3

Per-second, 1 flow, mesh down: [0.0, 0.0, 0.0, 0.0, 1.08, 0.0, 1.16, 0.97, 0.0, …], MSS 1228. A full inner packet is 1280 B, so the GotaTun datagram is 1340 B, larger than the 1280 underlay.

UDP Linux → Mac by size (iperf3 -u -t 8): mesh 1100 B (unfragmented outside) loses 0% / 0.8% at 20 / 80 Mbit/s. Mesh 1200 B (fragmented) loses 37.7 / 19.4 / 19.2% at 20 Mbit/s and 12.5% at 80. The plain path with fragmented 1300 B datagrams is also lossy at rate (0 / 0 / 69% at 20 Mbit/s, 78% at 80). So the fragments die in the underlay; GotaTun itself is clean. The product problem is that the mesh always produces packets this path must fragment. Pings with 1252-byte payloads pass both ways, so it isn't a black hole.

Where it bites: any Mac whose path to a peer has MTU below 1340, or Linux ↔ Linux below 1440 (IPv4) / 1460 (IPv6). Examples: another VPN, mobile carriers, 1280-MTU IPv6 links. compute01's own eth0 is 1450, so Linux mesh over IPv6 there fragments too.

Repro: a Mac and a Linux node whose only route crosses a 1280-MTU link, then iperf3 -c <mac mesh ip> -t 12 from Linux. Expected: close to the plain path. Actual: under 1 Mbit/s. Raw data: lume@infraternal-macos-01-std:~/j267-evidence/m-*, claude@compute01:~/.cache/codex-scratch/j267/evidence/l2m-*.

Fix direction (owner/orchestrator choice, transport-level): per-peer PMTU (probe it or derive it from the endpoint route), an MSS clamp on the mesh interface, or a lower default MTU. Not chosen here.

Links: #260, #375, D331, D346, D352.

Found by J267 (#375). Severity **P2** (performance; CLAUDE.md priority 1). Full data and commands: #375, comment "J267: measurement". **What:** the mesh MTU is a constant, `gotatun_tun.rs:30 MESH_MTU = 1380` (Linux) and `macos_node.rs:49 UTUN_MTU = 1280` (Mac). Nothing derives it from the path to the peer's endpoint, and nothing clamps MSS. When the underlay MTU is below inner MTU + 60 (IPv4 outer) or + 80 (IPv6), every full-size mesh packet fragments outside the tunnel. Towards the Mac that collapses TCP. **Measured** (Mac VM ↔ compute01; underlay NetBird, MTU 1280, about 143 ms RTT; dev `9b0c731e8`; the same GotaTun fork on both ends): | | 1 flow | 8 flows | |---|---:|---:| | Plain path, Linux → Mac | 132.4 / 145.1 Mbit/s | 374.1 / 529.4 Mbit/s | | **Mesh, Linux → Mac** | **0.5** (repeat: **0.2**) | **33.5** | | Mesh, Mac → Linux | 119.2 | 125.3 | Per-second, 1 flow, mesh down: `[0.0, 0.0, 0.0, 0.0, 1.08, 0.0, 1.16, 0.97, 0.0, …]`, MSS 1228. A full inner packet is 1280 B, so the GotaTun datagram is 1340 B, larger than the 1280 underlay. UDP Linux → Mac by size (`iperf3 -u -t 8`): mesh 1100 B (unfragmented outside) loses **0% / 0.8%** at 20 / 80 Mbit/s. Mesh 1200 B (fragmented) loses **37.7 / 19.4 / 19.2%** at 20 Mbit/s and 12.5% at 80. The plain path with fragmented 1300 B datagrams is also lossy at rate (0 / 0 / 69% at 20 Mbit/s, 78% at 80). So the fragments die in the underlay; GotaTun itself is clean. The product problem is that the mesh **always** produces packets this path must fragment. Pings with 1252-byte payloads pass both ways, so it isn't a black hole. **Where it bites:** any Mac whose path to a peer has MTU below 1340, or Linux ↔ Linux below 1440 (IPv4) / 1460 (IPv6). Examples: another VPN, mobile carriers, 1280-MTU IPv6 links. compute01's own `eth0` is 1450, so Linux mesh over IPv6 there fragments too. **Repro:** a Mac and a Linux node whose only route crosses a 1280-MTU link, then `iperf3 -c <mac mesh ip> -t 12` from Linux. Expected: close to the plain path. Actual: under 1 Mbit/s. Raw data: `lume@infraternal-macos-01-std:~/j267-evidence/m-*`, `claude@compute01:~/.cache/codex-scratch/j267/evidence/l2m-*`. **Fix direction** (owner/orchestrator choice, transport-level): per-peer PMTU (probe it or derive it from the endpoint route), an MSS clamp on the mesh interface, or a lower default MTU. Not chosen here. Links: #260, #375, D331, D346, D352.
kayg commented 2026-11-06 02:48:13 +00:00
Author
Owner
Copy link

J288 start. Branch mesh-mtu-381, based on origin/dev 3dc58dc67b771c7e38a298c3c5ac437055b8ff44.

Plan (estimate: 5 hours):

  1. Reproduce the 1280-byte underlay failure before implementation and record the result. Add a failing Linux namespace fixture that observes the current oversized mesh MTU against a 1280-byte endpoint route.
  2. Derive one interface MTU from the smallest route MTU among configured peer endpoints, minus 80 bytes (covers IPv6 outer headers); use 1200 as the conservative fallback when no endpoint route can be queried. Recompute when applying peer endpoint configuration.
  3. Apply that MTU consistently to Linux kernel WireGuard, Linux GotaTun's TUN and packet watcher, and macOS utun/GotaTun send and receive limits. Keep MSS and packetization consistent with the interface MTU.
  4. Measure before/after on the real Mac ↔ compute01 path with ~/mesh-vm.claim and the Mac lock held per measurement step. Measure normal-MTU performance on the D348 perf VM with perf-run, 2 CPUs per test side, and its no-interference check.
  5. Run verify quick, then verify fast; attach logs and report the measurements and gate output here.

Mechanism and trade-offs: I recommend route-derived interface MTU, taking the minimum over peer endpoint routes and subtracting 80 bytes. The official WireGuard wg-quick docs describe automatic MTU derivation from endpoint routes or the default route, and its Linux implementation uses the smallest endpoint-route MTU minus 80: wg-quick manual, wg-quick Linux implementation. A fixed 1200-byte MTU is simpler and safe for 1280-byte overlays, but needlessly reduces packet size on normal paths; I will keep it only as the no-route fallback. PMTU/ICMP handling is more adaptive but requires new outer-error handling across the kernel, GotaTun and macOS. MSS clamping addresses TCP only and requires firewall changes; it does not protect UDP or other large IP packets. Route derivation is read-only and does not change the tunnel transport or engine, so I am proceeding with it. It will be refreshed when peer configuration is applied, following the standard bring-up model; it will not add a live route-change monitor.

J288 start. Branch `mesh-mtu-381`, based on `origin/dev` `3dc58dc67b771c7e38a298c3c5ac437055b8ff44`. Plan (estimate: 5 hours): 1. Reproduce the 1280-byte underlay failure before implementation and record the result. Add a failing Linux namespace fixture that observes the current oversized mesh MTU against a 1280-byte endpoint route. 2. Derive one interface MTU from the smallest route MTU among configured peer endpoints, minus 80 bytes (covers IPv6 outer headers); use 1200 as the conservative fallback when no endpoint route can be queried. Recompute when applying peer endpoint configuration. 3. Apply that MTU consistently to Linux kernel WireGuard, Linux GotaTun's TUN and packet watcher, and macOS utun/GotaTun send and receive limits. Keep MSS and packetization consistent with the interface MTU. 4. Measure before/after on the real Mac ↔ compute01 path with `~/mesh-vm.claim` and the Mac lock held per measurement step. Measure normal-MTU performance on the D348 perf VM with `perf-run`, 2 CPUs per test side, and its no-interference check. 5. Run `verify quick`, then `verify fast`; attach logs and report the measurements and gate output here. Mechanism and trade-offs: I recommend route-derived interface MTU, taking the minimum over peer endpoint routes and subtracting 80 bytes. The official WireGuard `wg-quick` docs describe automatic MTU derivation from endpoint routes or the default route, and its Linux implementation uses the smallest endpoint-route MTU minus 80: [wg-quick manual](https://git.zx2c4.com/wireguard-tools/about/src/man/wg-quick.8), [wg-quick Linux implementation](https://git.zx2c4.com/wireguard-tools/tree/src/wg-quick/linux.bash). A fixed 1200-byte MTU is simpler and safe for 1280-byte overlays, but needlessly reduces packet size on normal paths; I will keep it only as the no-route fallback. PMTU/ICMP handling is more adaptive but requires new outer-error handling across the kernel, GotaTun and macOS. MSS clamping addresses TCP only and requires firewall changes; it does not protect UDP or other large IP packets. Route derivation is read-only and does not change the tunnel transport or engine, so I am proceeding with it. It will be refreshed when peer configuration is applied, following the standard bring-up model; it will not add a live route-change monitor.
kayg commented 2026-11-06 02:50:40 +00:00
Author
Owner
Copy link

Dependency finding: #379 confirms the Mac release build at this base (3dc58dc67) fails in restart.rs:271 because sockopt::PeerCredentials is Linux-only. This blocks compiling the required Mac parity change and running the real Mac measurement from this base. Recommendation: keep the #379 fix atomic and separate, then rebase #381 on its dev merge before Mac build/proof (estimated additional time: about 45 minutes). I will proceed with the Linux 1280-MTU reproducer and implementation while that prerequisite is being resolved; I will not bundle unrelated #379 changes here.

Dependency finding: #379 confirms the Mac release build at this base (`3dc58dc67`) fails in `restart.rs:271` because `sockopt::PeerCredentials` is Linux-only. This blocks compiling the required Mac parity change and running the real Mac measurement from this base. Recommendation: keep the #379 fix atomic and separate, then rebase #381 on its dev merge before Mac build/proof (estimated additional time: about 45 minutes). I will proceed with the Linux 1280-MTU reproducer and implementation while that prerequisite is being resolved; I will not bundle unrelated #379 changes here.
kayg commented 2026-11-06 03:11:06 +00:00
Author
Owner
Copy link

J288 finding / test-first red: the new namespace proof ran against base 3dc58dc67 with both veth underlay links at MTU 1280. It reported {"underlay_mtu": 1280, "mesh_mtus": [1380, 1380]} and failed as expected: the 1380-byte inner MTU plus 80 bytes of worst-case WireGuard overhead exceeds the underlay and forces fragmentation. The test fixture is scripts/mesh-namespace-proof.py --underlay-mtu 1280; red log: claude@infraternal-dev-02-std:~/.cache/codex-scratch/j288/mtu-red.log.

That packet-size failure matches J267's real-path result in #375: Linux → Mac was 0.5 Mbit/s (repeat 0.2) at one flow and 33.5 Mbit/s at eight flows, while the plain route was 132–145 / 374–529 Mbit/s. No production code has changed yet.

J288 finding / test-first red: the new namespace proof ran against base `3dc58dc67` with both veth underlay links at MTU 1280. It reported `{"underlay_mtu": 1280, "mesh_mtus": [1380, 1380]}` and failed as expected: the 1380-byte inner MTU plus 80 bytes of worst-case WireGuard overhead exceeds the underlay and forces fragmentation. The test fixture is `scripts/mesh-namespace-proof.py --underlay-mtu 1280`; red log: `claude@infraternal-dev-02-std:~/.cache/codex-scratch/j288/mtu-red.log`. That packet-size failure matches J267's real-path result in #375: Linux → Mac was 0.5 Mbit/s (repeat 0.2) at one flow and 33.5 Mbit/s at eight flows, while the plain route was 132–145 / 374–529 Mbit/s. No production code has changed yet.
kayg commented 2026-11-06 03:28:19 +00:00
Author
Owner
Copy link

Progress update: kayg ordered a freeze on new work on dev-02 and the Mac VM while their shared USB drive is moved. I will not queue builds, tests, syncs, or Mac-lock work on either VM.

Already completed before the freeze: the dev-02 1280-byte namespace baseline ran red with mesh MTUs [1380, 1380] against underlay MTU 1280. Log: /home/claude/.cache/codex-scratch/j288/mtu-red.log on dev-02. The Mac route check also already observed compute01 through utun100 at MTU 1280.

Pending until the freeze lifts: Mac-only implementation/build/runtime proof and the real Mac ↔ compute01 before/after measurement. The separate #379 native Mac build fix is also a prerequisite for that build. I will continue Linux implementation, build/test on compute01, and use the perf VM for measurements.

Progress update: kayg ordered a freeze on new work on dev-02 and the Mac VM while their shared USB drive is moved. I will not queue builds, tests, syncs, or Mac-lock work on either VM. Already completed before the freeze: the dev-02 1280-byte namespace baseline ran red with mesh MTUs [1380, 1380] against underlay MTU 1280. Log: `/home/claude/.cache/codex-scratch/j288/mtu-red.log` on dev-02. The Mac route check also already observed compute01 through `utun100` at MTU 1280. Pending until the freeze lifts: Mac-only implementation/build/runtime proof and the real Mac ↔ compute01 before/after measurement. The separate #379 native Mac build fix is also a prerequisite for that build. I will continue Linux implementation, build/test on compute01, and use the perf VM for measurements.
kayg commented 2026-11-06 03:38:44 +00:00
Author
Owner
Copy link

The working branch has been rebased from the requested start at 3dc58dc67 onto current origin/dev 2ca748b8c07a5836462b380ef4e340d829731ec5, per the updated process rules. Mac-only work remains paused by the owner’s VM freeze; Linux work will be built and tested on compute01.

The working branch has been rebased from the requested start at `3dc58dc67` onto current `origin/dev` `2ca748b8c07a5836462b380ef4e340d829731ec5`, per the updated process rules. Mac-only work remains paused by the owner’s VM freeze; Linux work will be built and tested on compute01.
kayg commented 2026-11-06 03:45:00 +00:00
Author
Owner
Copy link

Perf readiness finding: on infraternal-perf-01-backup01, perf.slice currently reports AllowedCPUs=1-3 and cpuset.cpus.effective=1-3 (6 online CPUs; load 0.18). Evidence: /home/kayg/.cache/codex-scratch/j288/evidence/perf-d348-readiness.log on the control host. This cannot provide the D348 two-disjoint-CPU sets for both test sides. #260's latest J88i progress still says no D348 validation or counted hold until the pinning halves are complete.

I have not started a benchmark or changed the perf VM's CPU masks. Recommendation: keep the perf run pending until the D348 pinning/restore setup is complete or explicitly lifted, then measure the normal-MTU matched pair through perf-run. Linux implementation and compute01 checks continue meanwhile.

Perf readiness finding: on `infraternal-perf-01-backup01`, `perf.slice` currently reports `AllowedCPUs=1-3` and `cpuset.cpus.effective=1-3` (6 online CPUs; load 0.18). Evidence: `/home/kayg/.cache/codex-scratch/j288/evidence/perf-d348-readiness.log` on the control host. This cannot provide the D348 two-disjoint-CPU sets for both test sides. #260's latest J88i progress still says no D348 validation or counted hold until the pinning halves are complete. I have not started a benchmark or changed the perf VM's CPU masks. Recommendation: keep the perf run pending until the D348 pinning/restore setup is complete or explicitly lifted, then measure the normal-MTU matched pair through `perf-run`. Linux implementation and compute01 checks continue meanwhile.
kayg commented 2026-11-06 03:50:54 +00:00
Author
Owner
Copy link

Linux implementation WIP committed and pushed as bf367b36871246137033c2ce1c83ab1e5d2747a5 on mesh-mtu-381 (rebased base 2ca748b8c07a5836462b380ef4e340d829731ec5). It derives the smallest peer-endpoint route MTU minus 80 bytes, updates Linux kernel WireGuard and GotaTun's link/watcher, and makes the namespace fixture assert exact route-derived MTUs for 1280/1500 underlays.

No quick gate or post-change fixture run has started. At the last check, all four compute01 build slots were occupied and five runs were queued; I have not queued this job. Mac work remains pending on the std-VM freeze, and the perf measurement remains pending on D348's unproven 2+2 CPU masks. The perf readiness receipt is /home/kayg/.cache/codex-scratch/j288/evidence/perf-d348-readiness.log.

Linux implementation WIP committed and pushed as `bf367b36871246137033c2ce1c83ab1e5d2747a5` on `mesh-mtu-381` (rebased base `2ca748b8c07a5836462b380ef4e340d829731ec5`). It derives the smallest peer-endpoint route MTU minus 80 bytes, updates Linux kernel WireGuard and GotaTun's link/watcher, and makes the namespace fixture assert exact route-derived MTUs for 1280/1500 underlays. No quick gate or post-change fixture run has started. At the last check, all four compute01 build slots were occupied and five runs were queued; I have not queued this job. Mac work remains pending on the std-VM freeze, and the perf measurement remains pending on D348's unproven 2+2 CPU masks. The perf readiness receipt is `/home/kayg/.cache/codex-scratch/j288/evidence/perf-d348-readiness.log`.
kayg commented 2026-11-06 04:00:10 +00:00
Author
Owner
Copy link

WIP update pushed: 1f70aecc01196d4406682a397373afee7cb33626 (supersedes bf367b3). The MTU read/write paths now carry a proven interface index: kernel updates use the existing owner-proof check, and GotaTun rechecks its owner marker before writing the MTU, so a reused infraternal0 name cannot redirect an update to an unrelated link.

The compute01 worktree is at this head. The build slots remain occupied; no gate or post-change namespace run has started.

WIP update pushed: `1f70aecc01196d4406682a397373afee7cb33626` (supersedes `bf367b3`). The MTU read/write paths now carry a proven interface index: kernel updates use the existing owner-proof check, and GotaTun rechecks its owner marker before writing the MTU, so a reused `infraternal0` name cannot redirect an update to an unrelated link. The compute01 worktree is at this head. The build slots remain occupied; no gate or post-change namespace run has started.
kayg commented 2026-11-06 04:02:14 +00:00
Author
Owner
Copy link

WIP head updated to 43267bd8795d8512292981ad8fcb92c6f9aef81f and pushed. Kernel MTU changes now stay tied to the proven interface index; the owner check happens before applying the temporary access fence. GotaTun validates its owner marker before reading or writing MTU.

The compute01 worktree is at this head. All four slots were still busy at the last status check; I have not queued or started a gate.

WIP head updated to `43267bd8795d8512292981ad8fcb92c6f9aef81f` and pushed. Kernel MTU changes now stay tied to the proven interface index; the owner check happens before applying the temporary access fence. GotaTun validates its owner marker before reading or writing MTU. The compute01 worktree is at this head. All four slots were still busy at the last status check; I have not queued or started a gate.
kayg commented 2026-11-06 04:14:14 +00:00
Author
Owner
Copy link

J288 continuation (2026-11-06 09:44 IST). Branch mesh-mtu-381 remains at 43267bd8795d8512292981ad8fcb92c6f9aef81f, based on current origin/dev 2ca748b8c07a5836462b380ef4e340d829731ec5.

The std-VM freeze is in effect; I started no new work on dev-02 or the Mac VM. Mac MTU implementation/build/runtime proof and the real Mac ↔ compute01 before/after measurement remain pending until the freeze lifts.

  • Compute01 still has all four build slots busy and seven queued jobs. I have not queued this job; I will run quick, the 1280/1500 namespace proof, and fast there when a slot is available. Current status: /home/kayg/.cache/codex-scratch/j288/evidence/compute01-slot-status-20261106.log.
  • The perf VM still has AllowedCPUs=1-3 and EffectiveCPUs=1-3 out of six online CPUs, which cannot provide D348's disjoint two-CPU sets for both sides. I made no CPU-mask changes and started no benchmark. Current status: /home/kayg/.cache/codex-scratch/j288/evidence/perf-d348-readiness-20261106.log. The normal-MTU measurement remains pending until D348 pinning is proven or lifted.
**J288 continuation (2026-11-06 09:44 IST).** Branch `mesh-mtu-381` remains at `43267bd8795d8512292981ad8fcb92c6f9aef81f`, based on current `origin/dev` `2ca748b8c07a5836462b380ef4e340d829731ec5`. The std-VM freeze is in effect; I started no new work on dev-02 or the Mac VM. Mac MTU implementation/build/runtime proof and the real Mac ↔ compute01 before/after measurement remain pending until the freeze lifts. - Compute01 still has all four build slots busy and seven queued jobs. I have not queued this job; I will run quick, the 1280/1500 namespace proof, and fast there when a slot is available. Current status: `/home/kayg/.cache/codex-scratch/j288/evidence/compute01-slot-status-20261106.log`. - The perf VM still has `AllowedCPUs=1-3` and `EffectiveCPUs=1-3` out of six online CPUs, which cannot provide D348's disjoint two-CPU sets for both sides. I made no CPU-mask changes and started no benchmark. Current status: `/home/kayg/.cache/codex-scratch/j288/evidence/perf-d348-readiness-20261106.log`. The normal-MTU measurement remains pending until D348 pinning is proven or lifted.
kayg commented 2026-11-06 04:18:02 +00:00
Author
Owner
Copy link

J288 quick gate queued (2026-11-06 09:48 IST). Correction to my previous status: all compute01 slots were still occupied, so I queued scripts/verify quick through build-slot at queue position 6 to run on the required host. It is not queued on dev-02, and I have started no Mac work. Log: claude@compute01:~/.cache/codex-scratch/j288/logs/verify-quick.log; PID receipt: claude@compute01:~/.cache/codex-scratch/j288/logs/verify-quick.pid. I will wait for this run to finish before starting the namespace proof.

**J288 quick gate queued (2026-11-06 09:48 IST).** Correction to my previous status: all compute01 slots were still occupied, so I queued `scripts/verify quick` through build-slot at queue position 6 to run on the required host. It is not queued on dev-02, and I have started no Mac work. Log: `claude@compute01:~/.cache/codex-scratch/j288/logs/verify-quick.log`; PID receipt: `claude@compute01:~/.cache/codex-scratch/j288/logs/verify-quick.pid`. I will wait for this run to finish before starting the namespace proof.
kayg commented 2026-11-06 04:22:18 +00:00
Author
Owner
Copy link

J288 finding: quick stopped at formatting. Evidence from claude@compute01:~/.cache/codex-scratch/j288/logs/verify-quick.log: scope guard, migration names and Nix Cargo vendoring passed; fmt found one rustfmt-only wrap in the new macOS route-parser assertion at mesh_mtu.rs:150. No compile or test rows ran after fmt stopped. I changed only that wrapping in the control worktree and will sync the file to compute01, rerun quick through build-slot, then proceed in order.

**J288 finding: quick stopped at formatting.** Evidence from `claude@compute01:~/.cache/codex-scratch/j288/logs/verify-quick.log`: scope guard, migration names and Nix Cargo vendoring passed; fmt found one rustfmt-only wrap in the new macOS route-parser assertion at `mesh_mtu.rs:150`. No compile or test rows ran after fmt stopped. I changed only that wrapping in the control worktree and will sync the file to compute01, rerun quick through build-slot, then proceed in order.
kayg commented 2026-11-06 04:31:03 +00:00
Author
Owner
Copy link

Correction to the quick/fmt finding: I read only the tail of the first log and understated the formatter output. The complete second-run log, copied to /home/kayg/.cache/codex-scratch/j288/evidence/verify-quick-r2.log, shows four remaining rustfmt hunks across gotatun_node.rs, gotatun_tun.rs, and mesh_mtu.rs; the Mac parser assertion from the first run is already wrapped. Quick stopped at fmt before compilation/tests. I am applying exactly the formatter's remaining changes, syncing those files to compute01, then I will rerun quick.

**Correction to the quick/fmt finding:** I read only the tail of the first log and understated the formatter output. The complete second-run log, copied to `/home/kayg/.cache/codex-scratch/j288/evidence/verify-quick-r2.log`, shows four remaining rustfmt hunks across `gotatun_node.rs`, `gotatun_tun.rs`, and `mesh_mtu.rs`; the Mac parser assertion from the first run is already wrapped. Quick stopped at fmt before compilation/tests. I am applying exactly the formatter's remaining changes, syncing those files to compute01, then I will rerun quick.
kayg commented 2026-11-06 04:38:17 +00:00
Author
Owner
Copy link

J288 progress (2026-11-06 10:08 IST). The formatter corrections from the full quick-r2 output are synced to compute01; quick-r3 is queued at position 5 and has not started. Current slot receipt: /home/kayg/.cache/codex-scratch/j288/evidence/compute01-slot-status-20261106-0437utc.log. The r3 log is claude@compute01:~/.cache/codex-scratch/j288/logs/verify-quick-r3.log.

I have not queued the namespace proof or fast gate behind quick. No dev-02 or Mac VM work has started. Mac parity/runtime/path measurement and D348 perf measurement remain pending for the reasons recorded above.

**J288 progress (2026-11-06 10:08 IST).** The formatter corrections from the full quick-r2 output are synced to compute01; quick-r3 is queued at position 5 and has not started. Current slot receipt: `/home/kayg/.cache/codex-scratch/j288/evidence/compute01-slot-status-20261106-0437utc.log`. The r3 log is `claude@compute01:~/.cache/codex-scratch/j288/logs/verify-quick-r3.log`. I have not queued the namespace proof or fast gate behind quick. No dev-02 or Mac VM work has started. Mac parity/runtime/path measurement and D348 perf measurement remain pending for the reasons recorded above.
kayg commented 2026-11-06 04:43:13 +00:00
Author
Owner
Copy link

J288 finding: D345 documentation gate failed. Quick-r3 passed scope, migration names, vendoring, fmt, doc coverage and the D351 shape check, then the doc-comment check listed seven missing comments across gotatun_node.rs, gotatun_tun.rs, lib.rs, mesh_mtu.rs and wireguard.rs. Full evidence: /home/kayg/.cache/codex-scratch/j288/evidence/verify-quick-r3.log. No compile/unit-test rows ran after this stop. I’m adding the requested purpose and invariant comments to those items, then I’ll rerun quick.

**J288 finding: D345 documentation gate failed.** Quick-r3 passed scope, migration names, vendoring, fmt, doc coverage and the D351 shape check, then the doc-comment check listed seven missing comments across `gotatun_node.rs`, `gotatun_tun.rs`, `lib.rs`, `mesh_mtu.rs` and `wireguard.rs`. Full evidence: `/home/kayg/.cache/codex-scratch/j288/evidence/verify-quick-r3.log`. No compile/unit-test rows ran after this stop. I’m adding the requested purpose and invariant comments to those items, then I’ll rerun quick.
kayg commented 2026-11-06 04:59:37 +00:00
Author
Owner
Copy link

J288 finding: quick/rust clippy stopped with six mesh errors. The full output is /home/kayg/.cache/codex-scratch/j288/evidence/verify-quick-r4.log. It lists five doc_markdown errors for unquoted GotaTun/WireGuard in changed comments, plus unnecessary_wraps on lower_route_mtu returning Option<u32>. Vendored dependency warnings were non-fatal. The helper's result is always a value when a route is supplied, so I will return u32 and keep absence in the fold accumulator; no unit tests ran because quick stopped at clippy.

**J288 finding: quick/rust clippy stopped with six mesh errors.** The full output is `/home/kayg/.cache/codex-scratch/j288/evidence/verify-quick-r4.log`. It lists five `doc_markdown` errors for unquoted `GotaTun`/`WireGuard` in changed comments, plus `unnecessary_wraps` on `lower_route_mtu` returning `Option<u32>`. Vendored dependency warnings were non-fatal. The helper's result is always a value when a route is supplied, so I will return `u32` and keep absence in the fold accumulator; no unit tests ran because quick stopped at clippy.
kayg commented 2026-11-06 05:06:27 +00:00
Author
Owner
Copy link

J288 finding: quick-r5 fmt stopped at one closure layout. Full output at /home/kayg/.cache/codex-scratch/j288/evidence/verify-quick-r5.log shows the route-fold closure should keep Ok(Some(lower_route_mtu(lowest, endpoint_route_mtu(endpoint)?))) on one line. No compile or clippy checks ran after fmt. I’m applying that exact formatting and will rerun quick.

**J288 finding: quick-r5 fmt stopped at one closure layout.** Full output at `/home/kayg/.cache/codex-scratch/j288/evidence/verify-quick-r5.log` shows the route-fold closure should keep `Ok(Some(lower_route_mtu(lowest, endpoint_route_mtu(endpoint)?)))` on one line. No compile or clippy checks ran after fmt. I’m applying that exact formatting and will rerun quick.
kayg commented 2026-11-06 05:14:55 +00:00
Author
Owner
Copy link

J288 quick pre-check PASS (pre-check only; not the final gate).

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| scope guard               | PASS         |
| migration names           | PASS         |
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| clippy (workspace)        | PASS         |
| unit tests (changed)      | PASS         |
+---------------------------+--------------+

QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done.

The changed-package tests reported 55 passed, 0 failed. Full log: /home/kayg/.cache/codex-scratch/j288/evidence/verify-quick-r6.log. Next: run the isolated GotaTun namespace proof at underlay MTUs 1280 and 1500, then the fast gate on compute01.

**J288 quick pre-check PASS** (pre-check only; not the final gate). ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | scope guard | PASS | | migration names | PASS | | Nix Cargo vendoring | PASS | | fmt | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | clippy (workspace) | PASS | | unit tests (changed) | PASS | +---------------------------+--------------+ QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done. ``` The changed-package tests reported 55 passed, 0 failed. Full log: `/home/kayg/.cache/codex-scratch/j288/evidence/verify-quick-r6.log`. Next: run the isolated GotaTun namespace proof at underlay MTUs 1280 and 1500, then the fast gate on compute01.
kayg commented 2026-11-06 05:24:22 +00:00
Author
Owner
Copy link

J288 rebase and compute01 status.

  • Branch mesh-mtu-381 is rebased on current origin/dev 0aad23e1c02778705c1b69abb4f92edaab3072b9; pushed head: a2958f4bde195c6788482abd54a9d7e7e4f9024f.
  • The quick pre-check passed on the previous base (2ca748b8c), with 55 changed-package tests passing; it must be rerun after this rebase. Previous log: /home/kayg/.cache/codex-scratch/j288/evidence/verify-quick-r6.log.
  • The compute01 worktree is synced to the pushed head. Its four build slots are occupied with two queued jobs; I have not queued another run. I will rerun quick, then the 1280/1500 namespace proof and fast gate when a slot is free.
  • Mac parity/runtime proof and the real Mac ↔ compute01 measurement remain paused under the std-VM freeze. The D348 perf measurement remains pending because only CPUs 1–3 are currently effective, which cannot provide two disjoint two-CPU sets for both test sides. I have not changed masks or started a benchmark.
**J288 rebase and compute01 status.** - Branch `mesh-mtu-381` is rebased on current `origin/dev` `0aad23e1c02778705c1b69abb4f92edaab3072b9`; pushed head: `a2958f4bde195c6788482abd54a9d7e7e4f9024f`. - The quick pre-check passed on the previous base (`2ca748b8c`), with 55 changed-package tests passing; it must be rerun after this rebase. Previous log: `/home/kayg/.cache/codex-scratch/j288/evidence/verify-quick-r6.log`. - The compute01 worktree is synced to the pushed head. Its four build slots are occupied with two queued jobs; I have not queued another run. I will rerun quick, then the 1280/1500 namespace proof and fast gate when a slot is free. - Mac parity/runtime proof and the real Mac ↔ compute01 measurement remain paused under the std-VM freeze. The D348 perf measurement remains pending because only CPUs 1–3 are currently effective, which cannot provide two disjoint two-CPU sets for both test sides. I have not changed masks or started a benchmark.
kayg commented 2026-11-06 05:28:57 +00:00
Author
Owner
Copy link

Perf D348 readiness recheck. A fresh read after the temporary 0-5 observation shows perf.slice back at AllowedCPUs=1-3, effective CPUs 1-3, out of six online CPUs. Receipt: /home/kayg/.cache/codex-scratch/j288/evidence/perf-d348-readiness-recheck.log (2026-11-06 05:28 UTC). #260 records J88i's D348 validation still in progress and no calibration/counting hold yet, so I did not start a benchmark. The normal-MTU comparison remains pending until that pinning setup is proven and the perf window is released.

**Perf D348 readiness recheck.** A fresh read after the temporary `0-5` observation shows `perf.slice` back at `AllowedCPUs=1-3`, effective CPUs `1-3`, out of six online CPUs. Receipt: `/home/kayg/.cache/codex-scratch/j288/evidence/perf-d348-readiness-recheck.log` (2026-11-06 05:28 UTC). #260 records J88i's D348 validation still in progress and no calibration/counting hold yet, so I did not start a benchmark. The normal-MTU comparison remains pending until that pinning setup is proven and the perf window is released.
perf-d348-readiness-recheck.log
54 B
kayg commented 2026-11-06 05:37:55 +00:00
Author
Owner
Copy link

J288 quick rerun queued on compute01. scripts/verify quick is now waiting in build-slot at queue position 3 (ticket PID 419151); log: claude@compute01:~/.cache/codex-scratch/j288/evidence/verify-quick-post-rebase.log. The rebase moved the branch to a2958f4bde195c6788482abd54a9d7e7e4f9024f; this run is against that pushed head and uses /home/claude/worktrees/j288/target, which will also be used for the final gate. No duplicate run was started after the PID-receipt race.

**J288 quick rerun queued on compute01.** `scripts/verify quick` is now waiting in build-slot at queue position 3 (ticket PID `419151`); log: `claude@compute01:~/.cache/codex-scratch/j288/evidence/verify-quick-post-rebase.log`. The rebase moved the branch to `a2958f4bde195c6788482abd54a9d7e7e4f9024f`; this run is against that pushed head and uses `/home/claude/worktrees/j288/target`, which will also be used for the final gate. No duplicate run was started after the PID-receipt race.
kayg commented 2026-11-06 05:47:46 +00:00
Author
Owner
Copy link

J288 compute01 queue wait. The post-rebase scripts/verify quick run remains at build-slot queue position 3 after about ten minutes; compilation and tests have not started. Log: claude@compute01:~/.cache/codex-scratch/j288/evidence/verify-quick-post-rebase.log. I am leaving the run in FIFO order and have not queued the namespace proof or fast gate behind it.

**J288 compute01 queue wait.** The post-rebase `scripts/verify quick` run remains at build-slot queue position 3 after about ten minutes; compilation and tests have not started. Log: `claude@compute01:~/.cache/codex-scratch/j288/evidence/verify-quick-post-rebase.log`. I am leaving the run in FIFO order and have not queued the namespace proof or fast gate behind it.
kayg commented 2026-11-06 05:52:38 +00:00
Author
Owner
Copy link

J288 quick rerun launch failure, not a source failure. The rebased run acquired a compute01 slot, then stopped before any verification row because the nested login shell cleared the Nix cargo path: FileNotFoundError: [Errno 2] No such file or directory: 'cargo'. Log: claude@compute01:~/.cache/codex-scratch/j288/evidence/verify-quick-post-rebase.log. The worktree remains clean at a2958f4bde195c6788482abd54a9d7e7e4f9024f. I am changing the launcher to call nix develop -c env ... scripts/verify quick directly, then will retry through build-slot.

**J288 quick rerun launch failure, not a source failure.** The rebased run acquired a compute01 slot, then stopped before any verification row because the nested login shell cleared the Nix `cargo` path: `FileNotFoundError: [Errno 2] No such file or directory: 'cargo'`. Log: `claude@compute01:~/.cache/codex-scratch/j288/evidence/verify-quick-post-rebase.log`. The worktree remains clean at `a2958f4bde195c6788482abd54a9d7e7e4f9024f`. I am changing the launcher to call `nix develop -c env ... scripts/verify quick` directly, then will retry through build-slot.
kayg commented 2026-11-06 05:53:34 +00:00
Author
Owner
Copy link

J288 quick retry queued. The corrected launcher calls nix develop -c env ... scripts/verify quick without a nested login shell. It is waiting behind four older compute01 tickets (build-slot ticket PID 622901; launcher PID 622900). Log: claude@compute01:~/.cache/codex-scratch/j288/evidence/verify-quick-post-rebase-r2.log. No second run was started; the earlier attempt exited before checks.

**J288 quick retry queued.** The corrected launcher calls `nix develop -c env ... scripts/verify quick` without a nested login shell. It is waiting behind four older compute01 tickets (build-slot ticket PID `622901`; launcher PID `622900`). Log: `claude@compute01:~/.cache/codex-scratch/j288/evidence/verify-quick-post-rebase-r2.log`. No second run was started; the earlier attempt exited before checks.
kayg commented 2026-11-06 06:10:16 +00:00
Author
Owner
Copy link

Correction: the previous quick attempt cannot be used as a code result. It acquired a slot as the branch base was advancing and the compute01 worktree was being reset. Its log reports merge base 0aad23e1c027 and then fails because crates/plugins/mesh/src/mesh_mtu.rs disappeared during fixed test waits; this was a checkout race, not a source assertion. Log: claude@compute01:~/.cache/codex-scratch/j288/evidence/verify-quick-post-rebase-r2.log.

The branch and compute01 worktree are now stable at pushed head 2805422aec86385b0bf5269c49b127b93dcfa643, rebased on current origin/dev 863c8f97fc8d93002d3cba7590004fa61421a6c8. I will rerun quick against this head before continuing.

**Correction: the previous quick attempt cannot be used as a code result.** It acquired a slot as the branch base was advancing and the compute01 worktree was being reset. Its log reports merge base `0aad23e1c027` and then fails because `crates/plugins/mesh/src/mesh_mtu.rs` disappeared during `fixed test waits`; this was a checkout race, not a source assertion. Log: `claude@compute01:~/.cache/codex-scratch/j288/evidence/verify-quick-post-rebase-r2.log`. The branch and compute01 worktree are now stable at pushed head `2805422aec86385b0bf5269c49b127b93dcfa643`, rebased on current `origin/dev` `863c8f97fc8d93002d3cba7590004fa61421a6c8`. I will rerun quick against this head before continuing.
kayg commented 2026-11-06 06:15:52 +00:00
Author
Owner
Copy link

Compute01 queue status correction. A follow-up read shows 34 GB free, back above build-slot's 25 GB floor; all four slots remain busy and J288 ticket PID 807669 remains queued. The current wait is for slots, not disk. Receipt: /home/kayg/.cache/codex-scratch/j288/evidence/compute01-slot-status-diskfloor.log (2026-11-06 06:15 UTC). The short 21 GB reading was transient.

**Compute01 queue status correction.** A follow-up read shows 34 GB free, back above build-slot's 25 GB floor; all four slots remain busy and J288 ticket PID `807669` remains queued. The current wait is for slots, not disk. Receipt: `/home/kayg/.cache/codex-scratch/j288/evidence/compute01-slot-status-diskfloor.log` (2026-11-06 06:15 UTC). The short 21 GB reading was transient.
kayg commented 2026-11-06 06:22:31 +00:00
Author
Owner
Copy link

J288 quick pre-check on rebased head 2805422aec86385b0bf5269c49b127b93dcfa643 (base 863c8f97fc8d93002d3cba7590004fa61421a6c8) passed on compute01.

| Rust gate                 | Result |
|---------------------------|--------|
| scope guard               | PASS   |
| migration names           | PASS   |
| Nix Cargo vendoring       | PASS   |
| fmt                       | PASS   |
| doc coverage ratchet      | PASS   |
| shape lints off (D351)    | PASS   |
| doc comments on diff      | PASS   |
| fixed test waits          | PASS   |
| clippy (workspace)        | PASS   |
| unit tests (changed)      | PASS   |

55 passed; 0 failed.
QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done.
VERIFY_QUICK_EXIT=0

Artifact: claude@compute01:/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-post-rebase-r3.log. Next on compute01: the isolated GotaTun namespace proof at underlay MTUs 1280 and 1500. Mac work remains frozen; D348 measurement remains pending J88i's calibration/window release.

J288 quick pre-check on rebased head `2805422aec86385b0bf5269c49b127b93dcfa643` (base `863c8f97fc8d93002d3cba7590004fa61421a6c8`) passed on compute01. ``` | Rust gate | Result | |---------------------------|--------| | scope guard | PASS | | migration names | PASS | | Nix Cargo vendoring | PASS | | fmt | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | clippy (workspace) | PASS | | unit tests (changed) | PASS | 55 passed; 0 failed. QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done. VERIFY_QUICK_EXIT=0 ``` Artifact: `claude@compute01:/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-post-rebase-r3.log`. Next on compute01: the isolated GotaTun namespace proof at underlay MTUs 1280 and 1500. Mac work remains frozen; D348 measurement remains pending J88i's calibration/window release.
kayg commented 2026-11-06 06:34:49 +00:00
Author
Owner
Copy link

J288 namespace-proof launch failure (not a source result): compute01 build-slot acquired ticket PID 984159 and then nix develop searched from /home/claude, where it found no flake.nix. Cargo build and both namespace runs did not start. Evidence: claude@compute01:/home/claude/.cache/codex-scratch/j288/evidence/mesh-namespace-1280-1500.log.

I’m correcting the launcher to change into /home/claude/worktrees/j288 before invoking build-slot, then I’ll rerun the build and 1280/1500 proofs in FIFO order.

J288 namespace-proof launch failure (not a source result): compute01 build-slot acquired ticket PID `984159` and then `nix develop` searched from `/home/claude`, where it found no `flake.nix`. Cargo build and both namespace runs did not start. Evidence: `claude@compute01:/home/claude/.cache/codex-scratch/j288/evidence/mesh-namespace-1280-1500.log`. I’m correcting the launcher to change into `/home/claude/worktrees/j288` before invoking build-slot, then I’ll rerun the build and 1280/1500 proofs in FIFO order.
kayg commented 2026-11-06 06:41:33 +00:00
Author
Owner
Copy link

J288 namespace regression on the corrected launcher: the kernel_node example built, then the 1280-byte case exited during Node::chunk with Error: Os { code: 22, kind: InvalidInput }. The fixture did not report mesh MTUs, and the 1500-byte case did not run. This is a red result against the new branch, not the original baseline. Evidence: claude@compute01:/home/claude/.cache/codex-scratch/j288/evidence/mesh-namespace-1280-1500-r2.log.

I’m tracing the failing operation before changing code.

J288 namespace regression on the corrected launcher: the `kernel_node` example built, then the 1280-byte case exited during `Node::chunk` with `Error: Os { code: 22, kind: InvalidInput }`. The fixture did not report mesh MTUs, and the 1500-byte case did not run. This is a red result against the new branch, not the original baseline. Evidence: `claude@compute01:/home/claude/.cache/codex-scratch/j288/evidence/mesh-namespace-1280-1500-r2.log`. I’m tracing the failing operation before changing code.
kayg commented 2026-11-06 06:50:24 +00:00
Author
Owner
Copy link

J288 diagnostic narrowed the namespace failure: current_mtu read 1200 and update_mtu(1200) succeeded; start_device(&config) then returned EINVAL. So the route lookup and TUN MTU update passed, and the error is in GotaTun peer/device initialization. Trace artifact: claude@compute01:/home/claude/.cache/codex-scratch/j288/evidence/mesh-namespace-debug-1280.log. I’m tracing that initialization path before changing code.

J288 diagnostic narrowed the namespace failure: `current_mtu` read 1200 and `update_mtu(1200)` succeeded; `start_device(&config)` then returned `EINVAL`. So the route lookup and TUN MTU update passed, and the error is in GotaTun peer/device initialization. Trace artifact: `claude@compute01:/home/claude/.cache/codex-scratch/j288/evidence/mesh-namespace-debug-1280.log`. I’m tracing that initialization path before changing code.
kayg commented 2026-11-06 06:52:09 +00:00
Author
Owner
Copy link

Correction to the diagnostic comment above: the temporary trace printed no start_device failed line, so GotaTun device startup returned successfully. The EINVAL occurs later in the outer Node::apply transaction after peer initialization. I’m adding boundary traces around route and firewall installation to identify the exact failing operation; the earlier attribution to start_device was incorrect.

Correction to the diagnostic comment above: the temporary trace printed no `start_device failed` line, so GotaTun device startup returned successfully. The `EINVAL` occurs later in the outer `Node::apply` transaction after peer initialization. I’m adding boundary traces around route and firewall installation to identify the exact failing operation; the earlier attribution to `start_device` was incorrect.
kayg commented 2026-11-06 06:58:40 +00:00
Author
Owner
Copy link

J288 root cause, reproduced with a disposable user/network namespace on compute01: Linux rejects adding the mesh IPv6 address while the TUN MTU is 1200 (RTNETLINK answers: Invalid argument); adding the IPv6 address at MTU 1280 and lowering the link to 1200 afterwards succeeds. Evidence: claude@compute01:/home/claude/.cache/codex-scratch/j288/evidence/ipv6-minmtu-r2.log.

The namespace fixture trace confirms route-overlap checking and the access fence pass, then the first snapshot fails at Device::addresses before peers are applied. I’ll keep 1280 as the initial interface MTU so dual-stack addresses can be installed, then apply the route-derived MTU after the first address installation. For later peer changes, the existing lower-before-peer-update ordering stays in place. This preserves the route-derived mechanism and its 1200 fallback.

J288 root cause, reproduced with a disposable user/network namespace on compute01: Linux rejects adding the mesh IPv6 address while the TUN MTU is 1200 (`RTNETLINK answers: Invalid argument`); adding the IPv6 address at MTU 1280 and lowering the link to 1200 afterwards succeeds. Evidence: `claude@compute01:/home/claude/.cache/codex-scratch/j288/evidence/ipv6-minmtu-r2.log`. The namespace fixture trace confirms route-overlap checking and the access fence pass, then the first snapshot fails at `Device::addresses` before peers are applied. I’ll keep 1280 as the initial interface MTU so dual-stack addresses can be installed, then apply the route-derived MTU after the first address installation. For later peer changes, the existing lower-before-peer-update ordering stays in place. This preserves the route-derived mechanism and its 1200 fallback.
kayg commented 2026-11-06 07:11:24 +00:00
Author
Owner
Copy link

J288 Linux follow-up plan (2026-11-06): the namespace proof at underlay MTU 1280 is red before the fix because the first snapshot tries to install its IPv6 address on a 1200-byte link. The reproducible kernel result and proposed ordering are recorded above; evidence is /home/claude/.cache/codex-scratch/j288/evidence/ipv6-minmtu-r2.log and the failed 1280/1500 namespace run is /home/claude/.cache/codex-scratch/j288/evidence/mesh-namespace-1280-1500-r2.log on compute01.

Linux fix: keep a fresh GotaTun TUN at least 1280 bytes while addresses are installed, then apply the route-derived MTU; in the kernel path, install addresses before lowering the interface, but tighten the MTU before peer changes. Next I’ll run verify quick on compute01, the 1280/1500 namespace proof, then verify fast, and commit/push the verified fix.

Mac MTU integration/build/runtime proof and the real Mac ↔ compute01 before/after measurement remain pending until the std-VM freeze is lifted. The normal-MTU perf-VM comparison remains pending the D348 calibration/window recorded in #260; I will not change perf CPU masks.

J288 Linux follow-up plan (2026-11-06): the namespace proof at underlay MTU 1280 is red before the fix because the first snapshot tries to install its IPv6 address on a 1200-byte link. The reproducible kernel result and proposed ordering are recorded above; evidence is `/home/claude/.cache/codex-scratch/j288/evidence/ipv6-minmtu-r2.log` and the failed 1280/1500 namespace run is `/home/claude/.cache/codex-scratch/j288/evidence/mesh-namespace-1280-1500-r2.log` on compute01. Linux fix: keep a fresh GotaTun TUN at least 1280 bytes while addresses are installed, then apply the route-derived MTU; in the kernel path, install addresses before lowering the interface, but tighten the MTU before peer changes. Next I’ll run `verify quick` on compute01, the 1280/1500 namespace proof, then `verify fast`, and commit/push the verified fix. Mac MTU integration/build/runtime proof and the real Mac ↔ compute01 before/after measurement remain pending until the std-VM freeze is lifted. The normal-MTU perf-VM comparison remains pending the D348 calibration/window recorded in #260; I will not change perf CPU masks.
kayg commented 2026-11-06 07:32:39 +00:00
Author
Owner
Copy link

J288 Linux ordering fix committed and pushed at e76dce45095... (full head is in the branch). scripts/verify quick passed on compute01 against base 863c8f97fc8d93002d3cba7590004fa61421a6c8; this is the pre-check, not the final gate. The changed-package suite reported 55 passed, 0 failed. Full artifact: /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-mtu-order.log.

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| scope guard               | PASS         |
| migration names           | PASS         |
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| clippy (workspace)        | PASS         |
| unit tests (changed)      | PASS         |
+---------------------------+--------------+

QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done.

The 1280/1500 namespace proof is queued next on compute01. Mac integration/build/runtime proof and the real Mac ↔ compute01 comparison remain pending the std-VM freeze; the D348 perf measurement remains pending the calibration/window recorded in #260.

J288 Linux ordering fix committed and pushed at `e76dce45095...` (full head is in the branch). `scripts/verify quick` passed on compute01 against base `863c8f97fc8d93002d3cba7590004fa61421a6c8`; this is the pre-check, not the final gate. The changed-package suite reported 55 passed, 0 failed. Full artifact: `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-mtu-order.log`. ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | scope guard | PASS | | migration names | PASS | | Nix Cargo vendoring | PASS | | fmt | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | clippy (workspace) | PASS | | unit tests (changed) | PASS | +---------------------------+--------------+ QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done. ``` The 1280/1500 namespace proof is queued next on compute01. Mac integration/build/runtime proof and the real Mac ↔ compute01 comparison remain pending the std-VM freeze; the D348 perf measurement remains pending the calibration/window recorded in #260.
verify-quick-mtu-order.log
16 KiB
kayg commented 2026-11-06 07:45:14 +00:00
Author
Owner
Copy link

D348 status correction from #260 (comments 81846 and 81876): the cpuset/restoration validation passed, but the three-control calibration failed its spread checks (1-flow p99 9.863% vs 3.2%; 8-flow throughput 0.813% vs 0.8%). The owner directed the matched hold to proceed and requires it to be labeled owner-directed and uncalibrated. I’m updating the perf plan accordingly; no perf benchmark has started for J288. I will use the D348 pinned runner and report its calibration limitation alongside the normal-MTU comparison.

D348 status correction from #260 (comments 81846 and 81876): the cpuset/restoration validation passed, but the three-control calibration failed its spread checks (1-flow p99 9.863% vs 3.2%; 8-flow throughput 0.813% vs 0.8%). The owner directed the matched hold to proceed and requires it to be labeled owner-directed and uncalibrated. I’m updating the perf plan accordingly; no perf benchmark has started for J288. I will use the D348 pinned runner and report its calibration limitation alongside the normal-MTU comparison.
kayg commented 2026-11-06 07:56:04 +00:00
Author
Owner
Copy link

Linux kernel-path review found a second IPv6 ordering issue before the 1280/1500 fixture rerun.

Device::create currently activates infraternal0 at the 1200-byte fallback (crates/plugins/mesh/src/wireguard.rs), and Device::addresses leaves it there. The kernel refuses to add the IPv6 address below 1280; the disposable-namespace reproduction is /home/claude/.cache/codex-scratch/j288/evidence/ipv6-minmtu-r2.log. GotaTun now keeps its fresh TUN at 1280 through address installation, but the kernel path does not yet.

I’ll keep the new kernel link at least 1280 until its addresses are installed, then apply the endpoint-route MTU, and extend the namespace proof to exercise both Linux engines. This is within the already-recorded route-derived MTU plan; it does not change the transport. The Mac implementation/build/proof and real Mac ↔ compute01 comparison remain pending the std-VM freeze. Compute01 is currently full, so I will not queue a new run.

Linux kernel-path review found a second IPv6 ordering issue before the 1280/1500 fixture rerun. `Device::create` currently activates `infraternal0` at the 1200-byte fallback (`crates/plugins/mesh/src/wireguard.rs`), and `Device::addresses` leaves it there. The kernel refuses to add the IPv6 address below 1280; the disposable-namespace reproduction is `/home/claude/.cache/codex-scratch/j288/evidence/ipv6-minmtu-r2.log`. GotaTun now keeps its fresh TUN at 1280 through address installation, but the kernel path does not yet. I’ll keep the new kernel link at least 1280 until its addresses are installed, then apply the endpoint-route MTU, and extend the namespace proof to exercise both Linux engines. This is within the already-recorded route-derived MTU plan; it does not change the transport. The Mac implementation/build/proof and real Mac ↔ compute01 comparison remain pending the std-VM freeze. Compute01 is currently full, so I will not queue a new run.
kayg commented 2026-11-06 08:06:57 +00:00
Author
Owner
Copy link

Correction to my previous code-path attribution: Device::create does not set the MTU. The first link is lowered to 1200 by Device::configure after key setup, before the first Node::apply installs addresses; Device::addresses applies the fallback after adding addresses. The confirmed kernel refusal below 1280 is still recorded in /home/claude/.cache/codex-scratch/j288/evidence/ipv6-minmtu-r2.log.

The fix will preserve the link MTU while configuring the kernel key, and, for address changes on an already low-MTU link, raise the previously proven interface index to 1280 immediately before address installation. Node::apply will then apply the route-derived MTU before peer changes. The namespace fixture will exercise the kernel engine first against the current failing state.

Correction to my previous code-path attribution: `Device::create` does not set the MTU. The first link is lowered to 1200 by `Device::configure` after key setup, before the first `Node::apply` installs addresses; `Device::addresses` applies the fallback after adding addresses. The confirmed kernel refusal below 1280 is still recorded in `/home/claude/.cache/codex-scratch/j288/evidence/ipv6-minmtu-r2.log`. The fix will preserve the link MTU while configuring the kernel key, and, for address changes on an already low-MTU link, raise the previously proven interface index to 1280 immediately before address installation. `Node::apply` will then apply the route-derived MTU before peer changes. The namespace fixture will exercise the kernel engine first against the current failing state.
kayg commented 2026-11-06 08:26:57 +00:00
Author
Owner
Copy link

J288 continuation after the std-VM freeze steering:

  • No new work has started on dev-02 or the Mac VM. Mac MTU integration, native build/runtime proof, and the real Mac ↔ compute01 after-measurement remain pending; #379's native build fix is still a prerequisite.
  • Current compute01 status: all four slots are occupied, three jobs are queued, and 100 GB is free. No J288 run is queued. I will wait for an available slot before running the expanded Linux proof or gates.
  • The perf VM still exposes CPUs 1–3 only. The earlier D348 cpuset/restoration check passed, but calibration spread failed its bounds; per the owner's direction, the matched normal-MTU hold may proceed only labeled owner-directed, uncalibrated. No CPU masks will be changed.
  • The original 1280-byte-underlay red result remains documented above, along with J267's real-path baseline: Linux → Mac mesh TCP was 0.5 Mbit/s (repeat 0.2) at 1 flow and 33.5 Mbit/s at 8; plain path was 132.4/145.1 and 374.1/529.4 Mbit/s. The J267 evidence paths are listed in #375.
  • Next: run both Linux engines at 1280 and 1500 underlay MTUs on compute01, then run quick and fast there. After building matched binaries on compute01, run the D348 normal-MTU perf hold and attach its artifacts. The Mac portions remain pending the freeze lift.
J288 continuation after the std-VM freeze steering: - No new work has started on dev-02 or the Mac VM. Mac MTU integration, native build/runtime proof, and the real Mac ↔ compute01 after-measurement remain pending; #379's native build fix is still a prerequisite. - Current compute01 status: all four slots are occupied, three jobs are queued, and 100 GB is free. No J288 run is queued. I will wait for an available slot before running the expanded Linux proof or gates. - The perf VM still exposes CPUs 1–3 only. The earlier D348 cpuset/restoration check passed, but calibration spread failed its bounds; per the owner's direction, the matched normal-MTU hold may proceed only labeled **owner-directed, uncalibrated**. No CPU masks will be changed. - The original 1280-byte-underlay red result remains documented above, along with J267's real-path baseline: Linux → Mac mesh TCP was 0.5 Mbit/s (repeat 0.2) at 1 flow and 33.5 Mbit/s at 8; plain path was 132.4/145.1 and 374.1/529.4 Mbit/s. The J267 evidence paths are listed in #375. - Next: run both Linux engines at 1280 and 1500 underlay MTUs on compute01, then run quick and fast there. After building matched binaries on compute01, run the D348 normal-MTU perf hold and attach its artifacts. The Mac portions remain pending the freeze lift.
kayg commented 2026-11-06 08:35:35 +00:00
Author
Owner
Copy link

J288 Linux review finding: the current ordering fix still leaves the kernel engine below IPv6's interface minimum before the first address is installed.

In wireguard.rs, Device::configure still calls activate(ifindex, Some(FALLBACK_MESH_MTU)) after setting the key. The new ordering in Node::apply moves its later route-derived update after Device::addresses, but that does not restore the MTU which configure already lowered to 1200. On address changes, an already-low interface has the same issue; rollback also reinstalls old addresses before restoring the prior MTU.

The isolated reproduction at /home/claude/.cache/codex-scratch/j288/evidence/ipv6-minmtu-r2.log confirms Linux rejects IPv6 address installation at MTU 1200 and accepts it at 1280. I will run the expanded kernel-engine namespace fixture at underlay MTU 1280 against the current branch before changing production code, then fix initial setup, low-MTU address updates, and rollback ordering if the fixture confirms the path.

J288 Linux review finding: the current ordering fix still leaves the kernel engine below IPv6's interface minimum before the first address is installed. In `wireguard.rs`, `Device::configure` still calls `activate(ifindex, Some(FALLBACK_MESH_MTU))` after setting the key. The new ordering in `Node::apply` moves its later route-derived update after `Device::addresses`, but that does not restore the MTU which `configure` already lowered to 1200. On address changes, an already-low interface has the same issue; rollback also reinstalls old addresses before restoring the prior MTU. The isolated reproduction at `/home/claude/.cache/codex-scratch/j288/evidence/ipv6-minmtu-r2.log` confirms Linux rejects IPv6 address installation at MTU 1200 and accepts it at 1280. I will run the expanded kernel-engine namespace fixture at underlay MTU 1280 against the current branch before changing production code, then fix initial setup, low-MTU address updates, and rollback ordering if the fixture confirms the path.
kayg commented 2026-11-06 08:37:35 +00:00
Author
Owner
Copy link

WireGuard reference check (primary sources): wg-quick documents automatic MTU selection from endpoint addresses or the default route. Its Linux implementation queries endpoint routes and subtracts 80 bytes; the Darwin script uses its default egress MTU minus 80. WireGuardKit's macOS settings use 80 bytes of tunnel overhead. This supports a shared route-derived interface ceiling while keeping platform-specific route lookup and update code; it does not require a tunnel transport change.

The Mac code/build/path measurement remains pending under the current freeze.

WireGuard reference check (primary sources): `wg-quick` documents automatic MTU selection from endpoint addresses or the default route. Its Linux implementation queries endpoint routes and subtracts 80 bytes; the Darwin script uses its default egress MTU minus 80. WireGuardKit's macOS settings use 80 bytes of tunnel overhead. This supports a shared route-derived interface ceiling while keeping platform-specific route lookup and update code; it does not require a tunnel transport change. - https://git.zx2c4.com/wireguard-tools/about/src/man/wg-quick.8 - https://git.zx2c4.com/wireguard-tools/tree/src/wg-quick/linux.bash - https://git.zx2c4.com/wireguard-tools/tree/src/wg-quick/darwin.bash - https://git.zx2c4.com/wireguard-apple/tree/Sources/WireGuardKit/PacketTunnelSettingsGenerator.swift The Mac code/build/path measurement remains pending under the current freeze.
kayg commented 2026-11-06 08:40:36 +00:00
Author
Owner
Copy link

Compute01 capacity update: all four build slots are busy and four jobs are queued ahead of J288. I have not queued J288. The kernel ordering regression fixture remains unrun so it can produce a clean red result against the current production state before I make the fix. No work has started on either frozen std VM.

Compute01 capacity update: all four build slots are busy and four jobs are queued ahead of J288. I have not queued J288. The kernel ordering regression fixture remains unrun so it can produce a clean red result against the current production state before I make the fix. No work has started on either frozen std VM.
kayg commented 2026-11-06 08:43:54 +00:00
Author
Owner
Copy link

J288 scripts/verify quick is now queued through compute01's build-slot at queue position 5 (launcher PID 2215136). It targets the pushed branch head e76dce4502ce00c3cc0ff5ad078a79ed36fc4ad1 plus the two local namespace-fixture edits, using /home/claude/worktrees/j288/target. Log: /home/claude/.cache/codex-scratch/j288/logs/verify-quick-current.log. It has not started yet. The next run is the kernel-engine 1280-MTU red proof; no work is queued on dev-02 or the Mac VM.

J288 `scripts/verify quick` is now queued through compute01's build-slot at queue position 5 (launcher PID 2215136). It targets the pushed branch head `e76dce4502ce00c3cc0ff5ad078a79ed36fc4ad1` plus the two local namespace-fixture edits, using `/home/claude/worktrees/j288/target`. Log: `/home/claude/.cache/codex-scratch/j288/logs/verify-quick-current.log`. It has not started yet. The next run is the kernel-engine 1280-MTU red proof; no work is queued on dev-02 or the Mac VM.
kayg commented 2026-11-06 08:55:32 +00:00
Author
Owner
Copy link

J288 kernel fixture red confirmed the missed ordering. On the current branch, the kernel_node example built and the isolated 1280-byte-underlay proof failed while applying the first configuration: Error: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }. It exited before the MTU assertion; the full artifact is /home/claude/.cache/codex-scratch/j288/evidence/mesh-kernel-red-1280.log.

This matches the earlier Linux IPv6 reproduction: key configuration lowered the new kernel link to the 1200 fallback before Device::addresses tried to add its IPv6 address. I will preserve the existing/default link MTU during key setup and temporarily raise any already-low mesh link to 1280 by its interface index while installing addresses. The address helper also handles rollback/re-enrollment paths that reinstall addresses.

J288 kernel fixture red confirmed the missed ordering. On the current branch, the kernel_node example built and the isolated 1280-byte-underlay proof failed while applying the first configuration: `Error: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }`. It exited before the MTU assertion; the full artifact is `/home/claude/.cache/codex-scratch/j288/evidence/mesh-kernel-red-1280.log`. This matches the earlier Linux IPv6 reproduction: key configuration lowered the new kernel link to the 1200 fallback before `Device::addresses` tried to add its IPv6 address. I will preserve the existing/default link MTU during key setup and temporarily raise any already-low mesh link to 1280 by its interface index while installing addresses. The address helper also handles rollback/re-enrollment paths that reinstall addresses.
kayg commented 2026-11-06 08:59:59 +00:00
Author
Owner
Copy link

Linux ordering fix plus fixture quick pre-check passed on compute01 at the current worktree state: all 10 Rust gate rows PASS, 55 passed / 0 failed, VERIFY_QUICK_EXIT=0. Artifact: /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-after-kernel-fix.log.

Next I will run the isolated 1280/1500 underlay matrix for both GotaTun and kernel WireGuard before committing the regression fixture and ordering fix.

Linux ordering fix plus fixture quick pre-check passed on compute01 at the current worktree state: all 10 Rust gate rows PASS, 55 passed / 0 failed, `VERIFY_QUICK_EXIT=0`. Artifact: `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-after-kernel-fix.log`. Next I will run the isolated 1280/1500 underlay matrix for both GotaTun and kernel WireGuard before committing the regression fixture and ordering fix.
kayg commented 2026-11-06 09:09:36 +00:00
Author
Owner
Copy link

J288 continuation after the owner lifted the std-VM freeze at 14:36 IST:

  • The Mac VM is back on its internal SSD with about 23 GB free. I will hold lockf -k /tmp/mac-bench.lock for the complete Mac build-and-test step, use one build at a time with CARGO_INCREMENTAL=0, and remove the job target before releasing the lock. The Mac MTU integration, native build/runtime proof, and real Mac ↔ compute01 after-measurement are active again.
  • The separate #379 native Mac build fix is now included in this branch's origin/dev base.
  • Linux follow-up: the first 1280-byte GotaTun namespace case stopped with ENODEV before reaching the MTU assertions; the kernel case did not run. Temporary apply-stage tracing is present only in the local worktree. I will rerun that case on compute01 to identify the failing stage, then finish the 1280/1500 matrix for both Linux engines.
  • J267's Linux → Mac baseline remains 0.5 Mbit/s (repeat 0.2) at 1 flow and 33.5 Mbit/s at 8; plain was 132.4/145.1 and 374.1/529.4 Mbit/s. The raw baseline paths remain in #375, comment “J267: measurement”.
  • The perf VM's D348 three-control calibration failed its spread bounds. Per the owner's direction, the matched normal-MTU hold will be labeled owner-directed and uncalibrated; CPU masks will not be changed.
J288 continuation after the owner lifted the std-VM freeze at 14:36 IST: - The Mac VM is back on its internal SSD with about 23 GB free. I will hold `lockf -k /tmp/mac-bench.lock` for the complete Mac build-and-test step, use one build at a time with `CARGO_INCREMENTAL=0`, and remove the job target before releasing the lock. The Mac MTU integration, native build/runtime proof, and real Mac ↔ compute01 after-measurement are active again. - The separate #379 native Mac build fix is now included in this branch's origin/dev base. - Linux follow-up: the first 1280-byte GotaTun namespace case stopped with `ENODEV` before reaching the MTU assertions; the kernel case did not run. Temporary apply-stage tracing is present only in the local worktree. I will rerun that case on compute01 to identify the failing stage, then finish the 1280/1500 matrix for both Linux engines. - J267's Linux → Mac baseline remains 0.5 Mbit/s (repeat 0.2) at 1 flow and 33.5 Mbit/s at 8; plain was 132.4/145.1 and 374.1/529.4 Mbit/s. The raw baseline paths remain in #375, comment “J267: measurement”. - The perf VM's D348 three-control calibration failed its spread bounds. Per the owner's direction, the matched normal-MTU hold will be labeled owner-directed and uncalibrated; CPU masks will not be changed.
kayg commented 2026-11-06 09:12:07 +00:00
Author
Owner
Copy link

J288 finding from the traced GotaTun 1280-byte namespace rerun: the first apply reaches Device::routes, which returns ENODEV (“No such device”). The fence, address installation, and peer configuration boundaries complete; the fixture still fails before its MTU assertion. Evidence: /home/claude/.cache/codex-scratch/j288/evidence/mesh-gotatun-trace-1280.log on compute01. I am tracing the route-install path before changing code.

J288 finding from the traced GotaTun 1280-byte namespace rerun: the first apply reaches `Device::routes`, which returns `ENODEV` (“No such device”). The fence, address installation, and peer configuration boundaries complete; the fixture still fails before its MTU assertion. Evidence: `/home/claude/.cache/codex-scratch/j288/evidence/mesh-gotatun-trace-1280.log` on compute01. I am tracing the route-install path before changing code.
kayg commented 2026-11-06 09:23:10 +00:00
Author
Owner
Copy link

J288 Linux trace finding: the first GotaTun apply adds the IPv4 /32 host route successfully, then the IPv6 /128 route-add returns ENODEV on mesh ifindex 8. Trace: /home/claude/.cache/codex-scratch/j288/evidence/mesh-gotatun-route-trace-1280.log on compute01. The current GotaTun apply lowers the TUN to the derived 1200-byte MTU before Node::apply installs those routes. I will verify the narrow hypothesis that Linux requires the interface to remain at least 1280 through IPv6 route installation, then apply the 1200-byte packet ceiling after route changes.

J288 Linux trace finding: the first GotaTun apply adds the IPv4 /32 host route successfully, then the IPv6 /128 route-add returns `ENODEV` on mesh ifindex 8. Trace: `/home/claude/.cache/codex-scratch/j288/evidence/mesh-gotatun-route-trace-1280.log` on compute01. The current GotaTun apply lowers the TUN to the derived 1200-byte MTU before `Node::apply` installs those routes. I will verify the narrow hypothesis that Linux requires the interface to remain at least 1280 through IPv6 route installation, then apply the 1200-byte packet ceiling after route changes.
kayg commented 2026-11-06 09:30:49 +00:00
Author
Owner
Copy link

J288 root-cause confirmation: a disposable Linux namespace on compute01 installed an IPv6 /128 route on an interface at MTU 1280, then failed to add the same route after lowering that interface to 1200 (RTNETLINK answers: No such device). This matches the fixture's IPv6 route-add failure. Evidence: /home/claude/.cache/codex-scratch/j288/evidence/ipv6-route-mtu-probe-r2.log. The candidate fix keeps the interface at or above 1280 until peer host routes are installed, then applies the route-derived packet MTU.

J288 root-cause confirmation: a disposable Linux namespace on compute01 installed an IPv6 /128 route on an interface at MTU 1280, then failed to add the same route after lowering that interface to 1200 (`RTNETLINK answers: No such device`). This matches the fixture's IPv6 route-add failure. Evidence: `/home/claude/.cache/codex-scratch/j288/evidence/ipv6-route-mtu-probe-r2.log`. The candidate fix keeps the interface at or above 1280 until peer host routes are installed, then applies the route-derived packet MTU.
kayg commented 2026-11-06 09:37:04 +00:00
Author
Owner
Copy link

J288 quick pre-check finding: the route-ordering candidate reached doc comments on diff and stopped because the changed WorkerCommand type and run_worker function lacked D345 documentation. All preceding rows, including fmt, coverage ratchet, and shape lints, passed. Artifact: /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-route-mtu-fix.log on compute01. I am adding the required docs and will rerun quick before the namespace matrix.

J288 quick pre-check finding: the route-ordering candidate reached `doc comments on diff` and stopped because the changed `WorkerCommand` type and `run_worker` function lacked D345 documentation. All preceding rows, including fmt, coverage ratchet, and shape lints, passed. Artifact: `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-route-mtu-fix.log` on compute01. I am adding the required docs and will rerun quick before the namespace matrix.
kayg commented 2026-11-06 09:43:30 +00:00
Author
Owner
Copy link

J288 Linux matrix result after the route-ordering change: the first GotaTun case at underlay MTU 1280 now installs its IPv4 and IPv6 host routes and reports final mesh MTUs [1200, 1200]. The IPv4 mesh ping passes; the IPv6 mesh ping then fails, so the matrix stopped before the kernel and 1500 cases. Initial artifact: /home/claude/.cache/codex-scratch/j288/evidence/mesh-linux-matrix.log on compute01. I added temporary route/address diagnostics to the fixture and will rerun this case to locate the IPv6 failure.

J288 Linux matrix result after the route-ordering change: the first GotaTun case at underlay MTU 1280 now installs its IPv4 and IPv6 host routes and reports final mesh MTUs `[1200, 1200]`. The IPv4 mesh ping passes; the IPv6 mesh ping then fails, so the matrix stopped before the kernel and 1500 cases. Initial artifact: `/home/claude/.cache/codex-scratch/j288/evidence/mesh-linux-matrix.log` on compute01. I added temporary route/address diagnostics to the fixture and will rerun this case to locate the IPv6 failure.
kayg commented 2026-11-06 09:57:25 +00:00
Author
Owner
Copy link

J288 continuation after the std freeze lift: the Mac VM is available again, but J279 currently owns /tmp/mac-bench.lock for an active mesh test build (PID 58278 under lockf). Pending Mac work is the parser red/green test, endpoint-route MTU lookup and utun update, native runtime proof, then the Mac↔compute01 after-measurement under the mesh claim. I will wait for the lock instead of starting or queueing a Mac step.

compute01 has all four build slots occupied on the latest status check (slot 3 is a fast gate); I will rerun only the GotaTun/1280 namespace diagnostic after a slot is free. The Mac target must be deleted before releasing its lock; the VM currently has about 24 GiB free.

J288 continuation after the std freeze lift: the Mac VM is available again, but J279 currently owns /tmp/mac-bench.lock for an active mesh test build (PID 58278 under lockf). Pending Mac work is the parser red/green test, endpoint-route MTU lookup and utun update, native runtime proof, then the Mac↔compute01 after-measurement under the mesh claim. I will wait for the lock instead of starting or queueing a Mac step. compute01 has all four build slots occupied on the latest status check (slot 3 is a fast gate); I will rerun only the GotaTun/1280 namespace diagnostic after a slot is free. The Mac target must be deleted before releasing its lock; the VM currently has about 24 GiB free.
kayg commented 2026-11-06 10:00:29 +00:00
Author
Owner
Copy link

J288 IPv6 diagnostic finding: after the 1280-byte GotaTun namespace apply reaches mesh MTUs [1200, 1200], both namespaces have no IPv6 route to the peer and no IPv6 address on infraternal0; ping fails immediately with Network is unreachable. This places the failure before packet forwarding. Evidence: /home/claude/.cache/codex-scratch/j288/evidence/mesh-gotatun-diagnostic-r3.log on compute01. I am tracing the MTU/address update boundary before changing code.

J288 IPv6 diagnostic finding: after the 1280-byte GotaTun namespace apply reaches mesh MTUs [1200, 1200], both namespaces have no IPv6 route to the peer and no IPv6 address on infraternal0; ping fails immediately with `Network is unreachable`. This places the failure before packet forwarding. Evidence: `/home/claude/.cache/codex-scratch/j288/evidence/mesh-gotatun-diagnostic-r3.log` on compute01. I am tracing the MTU/address update boundary before changing code.
kayg commented 2026-11-06 10:07:40 +00:00
Author
Owner
Copy link

J288 plan update before the next code change: the shared-interface MTU approach needs a Linux IPv6 correction. The 1280-byte GotaTun namespace run reached MTUs [1200, 1200], then both mesh IPv6 addresses and peer routes disappeared; ping failed locally with Network is unreachable (evidence: /home/claude/.cache/codex-scratch/j288/evidence/mesh-gotatun-diagnostic-r3.log). The root cause is that LinuxTun::set_mtu changes both the kernel link and GotaTun watcher; setting the link below IPv6's 1280 minimum removes IPv6 state.

Revised candidate: keep each shared tunnel interface at least 1280, but derive an unlocked MTU metric per peer host route from that peer's endpoint route minus the conservative 80-byte overhead (fallback 1200). This lets IPv4—and, if the Linux route stack accepts a sub-1280 IPv6 route metric, IPv6 too—advertise the smaller packet ceiling without lowering the interface. At normal 1500 underlay the route remains 1420. macOS applies the corresponding host-route metrics and sizes the utun/watcher to the largest peer budget with a 1280 floor.

Trade-offs: route state is more precise than one minimum shared MTU, but route updates must track peer endpoint changes; route MTU remains unlocked so operating-system PMTU updates can refine it. A 1280 floor protects IPv6 link state. Before implementing, I will verify Linux's IPv6 route-metric behavior in a disposable namespace; if it rejects the metric, the IPv6 route will stay at 1280 and the remaining outer-fragment behavior will be called out separately.

J288 plan update before the next code change: the shared-interface MTU approach needs a Linux IPv6 correction. The 1280-byte GotaTun namespace run reached MTUs [1200, 1200], then both mesh IPv6 addresses and peer routes disappeared; `ping` failed locally with `Network is unreachable` (evidence: `/home/claude/.cache/codex-scratch/j288/evidence/mesh-gotatun-diagnostic-r3.log`). The root cause is that `LinuxTun::set_mtu` changes both the kernel link and GotaTun watcher; setting the link below IPv6's 1280 minimum removes IPv6 state. Revised candidate: keep each shared tunnel interface at least 1280, but derive an unlocked MTU metric per peer host route from that peer's endpoint route minus the conservative 80-byte overhead (fallback 1200). This lets IPv4—and, if the Linux route stack accepts a sub-1280 IPv6 route metric, IPv6 too—advertise the smaller packet ceiling without lowering the interface. At normal 1500 underlay the route remains 1420. macOS applies the corresponding host-route metrics and sizes the utun/watcher to the largest peer budget with a 1280 floor. Trade-offs: route state is more precise than one minimum shared MTU, but route updates must track peer endpoint changes; route MTU remains unlocked so operating-system PMTU updates can refine it. A 1280 floor protects IPv6 link state. Before implementing, I will verify Linux's IPv6 route-metric behavior in a disposable namespace; if it rejects the metric, the IPv6 route will stay at 1280 and the remaining outer-fragment behavior will be called out separately.
kayg commented 2026-11-06 10:18:06 +00:00
Author
Owner
Copy link

J288 route-metric probe result: in a disposable Linux namespace with the interface at MTU 1280, ip -6 route add fd04::2/128 dev underlay0 mtu 1200 succeeds and ip -6 route get fd04::2 reports mtu 1200; the IPv6 address remains configured on the interface. IPv4 reports the same route metric. Evidence: /home/claude/.cache/codex-scratch/j288/evidence/ipv6-route-metric-probe.log on compute01.

This supports the revised per-peer host-route MTU plan for both address families while preserving the shared interface's 1280 minimum. I am adding the 1280-underlay fragment-drop regression proof first; its baseline will run against the pre-#381 commit before any route-metric implementation.

J288 route-metric probe result: in a disposable Linux namespace with the interface at MTU 1280, `ip -6 route add fd04::2/128 dev underlay0 mtu 1200` succeeds and `ip -6 route get fd04::2` reports `mtu 1200`; the IPv6 address remains configured on the interface. IPv4 reports the same route metric. Evidence: `/home/claude/.cache/codex-scratch/j288/evidence/ipv6-route-metric-probe.log` on compute01. This supports the revised per-peer host-route MTU plan for both address families while preserving the shared interface's 1280 minimum. I am adding the 1280-underlay fragment-drop regression proof first; its baseline will run against the pre-#381 commit before any route-metric implementation.
kayg commented 2026-11-06 10:39:33 +00:00
Author
Owner
Copy link

J288 pre-fix 1280-byte boundary proof (test-first): the baseline kernel-node fixture kept both mesh links at MTU 1380, with no route MTU. On a 1280-byte underlay, the probe created four outer IPv4 fragments and recorded three reassembly failures; oversized IPv4 and IPv6 probes were lost while small IPv4 and IPv6 probes passed. This demonstrates the packet-loss mechanism at the boundary; it is not a TCP throughput result.

Evidence: /home/claude/.cache/codex-scratch/j288/evidence/mesh-mtu-baseline-boundary.log on compute01.

J288 pre-fix 1280-byte boundary proof (test-first): the baseline kernel-node fixture kept both mesh links at MTU 1380, with no route MTU. On a 1280-byte underlay, the probe created four outer IPv4 fragments and recorded three reassembly failures; oversized IPv4 and IPv6 probes were lost while small IPv4 and IPv6 probes passed. This demonstrates the packet-loss mechanism at the boundary; it is not a TCP throughput result. Evidence: `/home/claude/.cache/codex-scratch/j288/evidence/mesh-mtu-baseline-boundary.log` on compute01.
kayg commented 2026-11-06 10:50:49 +00:00
Author
Owner
Copy link

J288 test-first route/interface MTU check: the compute01 unit test compiled the changed mesh crate and failed as intended because interface_mtu is not implemented yet. The three red assertions cover selecting the largest per-peer budget, preserving the 1280-byte IPv6 interface floor, and using that floor when there are no peers.

Evidence: /home/claude/.cache/codex-scratch/j288/evidence/mesh-mtu-linux-interface-test-red.log on compute01 (cargo exit 101; unresolved interface_mtu at the three assertions).

J288 test-first route/interface MTU check: the compute01 unit test compiled the changed mesh crate and failed as intended because `interface_mtu` is not implemented yet. The three red assertions cover selecting the largest per-peer budget, preserving the 1280-byte IPv6 interface floor, and using that floor when there are no peers. Evidence: `/home/claude/.cache/codex-scratch/j288/evidence/mesh-mtu-linux-interface-test-red.log` on compute01 (cargo exit 101; unresolved `interface_mtu` at the three assertions).
kayg commented 2026-11-06 11:09:38 +00:00
Author
Owner
Copy link

J288 Mac test-target finding: the native cargo command compiled the mesh crate but selected zero tests for the route-parser filter. Inspection shows mesh_mtu is currently compiled only for Linux in lib.rs, so its macOS parser tests and shared route-MTU logic are excluded on macOS. The zero-test run is not passing test evidence. I am widening the module gate to the supported Linux and macOS agent targets, then will rerun the fallback test red/green under the Mac lock.

Evidence: /Users/claude/.cache/codex-scratch/j288/evidence/mesh-mtu-macos-route-test-red.log on the Mac; output says running 0 tests and 108 filtered out.

J288 Mac test-target finding: the native cargo command compiled the mesh crate but selected zero tests for the route-parser filter. Inspection shows `mesh_mtu` is currently compiled only for Linux in `lib.rs`, so its macOS parser tests and shared route-MTU logic are excluded on macOS. The zero-test run is not passing test evidence. I am widening the module gate to the supported Linux and macOS agent targets, then will rerun the fallback test red/green under the Mac lock. Evidence: `/Users/claude/.cache/codex-scratch/j288/evidence/mesh-mtu-macos-route-test-red.log` on the Mac; output says `running 0 tests` and `108 filtered out`.
kayg commented 2026-11-06 11:14:12 +00:00
Author
Owner
Copy link

J288 Mac parser red test after widening the shared MTU module gate: the native test compile failed because parse_macos_route_mtu is missing at both the route lookup call and the fallback assertion. This is the intended pre-implementation failure. Cargo exited 101, and the lock-held script removed ~/worktrees/j288/target before releasing /tmp/mac-bench.lock.

Evidence: /Users/claude/.cache/codex-scratch/j288/evidence/mesh-mtu-macos-route-test-red2.log on the Mac.

J288 Mac parser red test after widening the shared MTU module gate: the native test compile failed because `parse_macos_route_mtu` is missing at both the route lookup call and the fallback assertion. This is the intended pre-implementation failure. Cargo exited 101, and the lock-held script removed `~/worktrees/j288/target` before releasing `/tmp/mac-bench.lock`. Evidence: `/Users/claude/.cache/codex-scratch/j288/evidence/mesh-mtu-macos-route-test-red2.log` on the Mac.
kayg commented 2026-11-06 11:34:03 +00:00
Author
Owner
Copy link

J288 resumed after the owner lifted the std-VM freeze at 14:36 IST. Branch mesh-mtu-381, pushed head e76dce4502ce00c3cc0ff5ad078a79ed36fc4ad1; the current worktree still has uncommitted Linux fixture/kernel-ordering and macOS route-MTU changes.

The Mac VM is back on the internal SSD with about 23 GB free. I will hold lockf -k /tmp/mac-bench.lock across each complete Mac build-and-test step, keep CARGO_INCREMENTAL=0, wait in the foreground if the lock is busy, and delete only ~/worktrees/j288/target before releasing the lock. I will not start concurrent Mac builds.

Remaining sequence: review the current route transaction and rollback paths; finish the Linux 1280/1500 namespace matrix for both engines and quick/fast on compute01; sync the final source to the Mac worktree and run the native mesh tests/runtime proof under the Mac lock; then take the Mac ↔ compute01 matched before/after measurement while holding the mesh claim and Mac lock. The normal-MTU perf hold uses perf-run and the owner-directed, uncalibrated D348 setup already recorded above; I will not change CPU masks. I will push atomic commits as they pass their focused checks and attach the evidence and verbatim final gate output here.

J288 resumed after the owner lifted the std-VM freeze at 14:36 IST. Branch `mesh-mtu-381`, pushed head `e76dce4502ce00c3cc0ff5ad078a79ed36fc4ad1`; the current worktree still has uncommitted Linux fixture/kernel-ordering and macOS route-MTU changes. The Mac VM is back on the internal SSD with about 23 GB free. I will hold `lockf -k /tmp/mac-bench.lock` across each complete Mac build-and-test step, keep `CARGO_INCREMENTAL=0`, wait in the foreground if the lock is busy, and delete only `~/worktrees/j288/target` before releasing the lock. I will not start concurrent Mac builds. Remaining sequence: review the current route transaction and rollback paths; finish the Linux 1280/1500 namespace matrix for both engines and quick/fast on compute01; sync the final source to the Mac worktree and run the native mesh tests/runtime proof under the Mac lock; then take the Mac ↔ compute01 matched before/after measurement while holding the mesh claim and Mac lock. The normal-MTU perf hold uses `perf-run` and the owner-directed, uncalibrated D348 setup already recorded above; I will not change CPU masks. I will push atomic commits as they pass their focused checks and attach the evidence and verbatim final gate output here.
kayg commented 2026-11-06 11:41:29 +00:00
Author
Owner
Copy link

J288 Mac build attempt finding: the first lock-held step synced the rebased worktree, then stopped before compilation because /bin/bash -lc did not expose cargo (/bin/bash: line 14: cargo: command not found). The target cleanup ran before releasing /tmp/mac-bench.lock; 26 GiB remained free. Evidence: claude@mac:~/.cache/codex-scratch/j288/evidence/mac-mtu-shell-path-failure.log. Nix is installed, so the next locked step will run both the focused native test and release build via nix develop.

J288 Mac build attempt finding: the first lock-held step synced the rebased worktree, then stopped before compilation because `/bin/bash -lc` did not expose `cargo` (`/bin/bash: line 14: cargo: command not found`). The target cleanup ran before releasing `/tmp/mac-bench.lock`; 26 GiB remained free. Evidence: `claude@mac:~/.cache/codex-scratch/j288/evidence/mac-mtu-shell-path-failure.log`. Nix is installed, so the next locked step will run both the focused native test and release build via `nix develop`.
kayg commented 2026-11-06 11:49:21 +00:00
Author
Owner
Copy link

J288 rebase and first native Mac proof:

  • The branch is rebased onto origin/dev 74b9c4d8021df6a91afd44e0a750c98652cd116e; pushed branch head has not yet been updated. Current commit is 2465a9346 with the implementation edits still local.
  • Under /tmp/mac-bench.lock and nix develop with CARGO_INCREMENTAL=0, the native route/MTU test filter ran 9 passed, 0 failed. The macOS release build completed; binary SHA-256: eeca389635eaaa769f3eb02f0c792dc636bd31226e0126313a4ea9963dabb4b4.
  • Evidence: claude@mac:~/.cache/codex-scratch/j288/evidence/mac-mtu-unit.log, mac-mtu-build.log, and mac-mtu-binary.sha256. The command removed ~/worktrees/j288/target before unlocking; free space returned to 26 GiB.
  • The native logs exposed two warnings in changed code (an unused parser binding and a Linux-only watcher helper compiled on macOS). Both are fixed locally; I will rerun the native checks and preserve the rebuilt binary outside target/ before Mac runtime and path measurement.
J288 rebase and first native Mac proof: - The branch is rebased onto `origin/dev` `74b9c4d8021df6a91afd44e0a750c98652cd116e`; pushed branch head has not yet been updated. Current commit is `2465a9346` with the implementation edits still local. - Under `/tmp/mac-bench.lock` and `nix develop` with `CARGO_INCREMENTAL=0`, the native route/MTU test filter ran **9 passed, 0 failed**. The macOS release build completed; binary SHA-256: `eeca389635eaaa769f3eb02f0c792dc636bd31226e0126313a4ea9963dabb4b4`. - Evidence: `claude@mac:~/.cache/codex-scratch/j288/evidence/mac-mtu-unit.log`, `mac-mtu-build.log`, and `mac-mtu-binary.sha256`. The command removed `~/worktrees/j288/target` before unlocking; free space returned to 26 GiB. - The native logs exposed two warnings in changed code (an unused parser binding and a Linux-only watcher helper compiled on macOS). Both are fixed locally; I will rerun the native checks and preserve the rebuilt binary outside `target/` before Mac runtime and path measurement.
kayg commented 2026-11-06 11:51:58 +00:00
Author
Owner
Copy link

Branch update: I rebased the existing #381 commits onto origin/dev 74b9c4d8021df6a91afd44e0a750c98652cd116e and pushed the rewritten branch with --force-with-lease. Head is 2465a93461387f9b7555e911ae4f4bf4dabf88df. The nine-file implementation/fixture diff remains local and uncommitted; the Mac logs above include that source as synced for the first native proof, before the two warning-only edits now in the local diff.

Branch update: I rebased the existing #381 commits onto `origin/dev` `74b9c4d8021df6a91afd44e0a750c98652cd116e` and pushed the rewritten branch with `--force-with-lease`. Head is `2465a93461387f9b7555e911ae4f4bf4dabf88df`. The nine-file implementation/fixture diff remains local and uncommitted; the Mac logs above include that source as synced for the first native proof, before the two warning-only edits now in the local diff.
kayg commented 2026-11-06 11:55:16 +00:00
Author
Owner
Copy link

J288 scripts/verify quick was launched through compute01 build-slot (launcher PID 369708) after confirming all four slots were occupied. It is at queue position 2 and has not started compilation. Log: /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-post-rebase.log. I will wait for this run before queuing the namespace matrix; the existing 7.1 GiB ~/worktrees/j288/target is retained for both gates.

J288 `scripts/verify quick` was launched through compute01 build-slot (launcher PID 369708) after confirming all four slots were occupied. It is at queue position 2 and has not started compilation. Log: `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-post-rebase.log`. I will wait for this run before queuing the namespace matrix; the existing 7.1 GiB `~/worktrees/j288/target` is retained for both gates.
kayg commented 2026-11-06 12:03:08 +00:00
Author
Owner
Copy link

Compute01 queue update: J288 verify quick remains at position 2 after about nine minutes. The latest build-slot --status showed all four slots occupied, with two jobs ahead of J288; free disk was 103 GiB. No namespace proof is queued behind it. The queue receipt remains /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-post-rebase.log.

Compute01 queue update: J288 `verify quick` remains at position 2 after about nine minutes. The latest `build-slot --status` showed all four slots occupied, with two jobs ahead of J288; free disk was 103 GiB. No namespace proof is queued behind it. The queue receipt remains `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-post-rebase.log`.
kayg commented 2026-11-06 12:03:39 +00:00
Author
Owner
Copy link

J288 clean native Mac step completed under /tmp/mac-bench.lock with CARGO_INCREMENTAL=0:

  • The route/MTU filter ran 9 passed, 0 failed, with no warnings from changed mesh code.
  • cargo build --locked --release -p infraternal completed. The binary's SHA-256 is eeca389635eaaa769f3eb02f0c792dc636bd31226e0126313a4ea9963dabb4b4; --version printed infraternal 0.1.0 (stable).
  • Evidence on the Mac: ~/.cache/codex-scratch/j288/evidence/mac-mtu-unit-clean.log, mac-mtu-build-clean.log, mac-mtu-binary-clean.sha256, and mac-mtu-runtime-version.log. The binary is preserved as ~/.cache/codex-scratch/j288/evidence/infraternal-mtu381 for the live proof.
  • ~/worktrees/j288/target was removed before releasing the lock; the VM has 25 GiB free. The build log has pre-existing unused-code warnings in the shim/remote CLI, while the changed mesh files are clean.
J288 clean native Mac step completed under `/tmp/mac-bench.lock` with `CARGO_INCREMENTAL=0`: - The route/MTU filter ran **9 passed, 0 failed**, with no warnings from changed mesh code. - `cargo build --locked --release -p infraternal` completed. The binary's SHA-256 is `eeca389635eaaa769f3eb02f0c792dc636bd31226e0126313a4ea9963dabb4b4`; `--version` printed `infraternal 0.1.0 (stable)`. - Evidence on the Mac: `~/.cache/codex-scratch/j288/evidence/mac-mtu-unit-clean.log`, `mac-mtu-build-clean.log`, `mac-mtu-binary-clean.sha256`, and `mac-mtu-runtime-version.log`. The binary is preserved as `~/.cache/codex-scratch/j288/evidence/infraternal-mtu381` for the live proof. - `~/worktrees/j288/target` was removed before releasing the lock; the VM has 25 GiB free. The build log has pre-existing unused-code warnings in the shim/remote CLI, while the changed mesh files are clean.
kayg commented 2026-11-06 12:07:50 +00:00
Author
Owner
Copy link

J288 live-proof block: ~/mesh-vm.claim on compute01 still belongs to J254-linux-proof (claim timestamp 2026-11-06T11:54:42Z). I will not remove or replace it; the live Mac ↔ compute01 mesh setup and after-measurement are waiting for that job to release the claim. The Mac endpoint path is present (route -n get 10.69.69.218 selects utun100, MTU 1280), and J267's measurement harness/raw baselines are under /Users/lume/j267-evidence. The new Mac release binary is built; I have not started a mesh node or altered live routes.

J288 live-proof block: `~/mesh-vm.claim` on compute01 still belongs to `J254-linux-proof` (claim timestamp `2026-11-06T11:54:42Z`). I will not remove or replace it; the live Mac ↔ compute01 mesh setup and after-measurement are waiting for that job to release the claim. The Mac endpoint path is present (`route -n get 10.69.69.218` selects `utun100`, MTU 1280), and J267's measurement harness/raw baselines are under `/Users/lume/j267-evidence`. The new Mac release binary is built; I have not started a mesh node or altered live routes.
kayg commented 2026-11-06 12:16:00 +00:00
Author
Owner
Copy link

J288 verify quick fmt finding (log /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-post-rebase.log): the formatter found four layout differences in my changed mesh files and one in untouched crates/plugins/remote/src/uninstall.rs, then stopped before clippy or tests. I corrected the four mesh hunks. The remaining hunk is in the fetched base: git diff --quiet origin/dev -- crates/plugins/remote/src/uninstall.rs passed, both the worktree and origin/dev file hash to 10bfc08086f06aacd7e01a1338c128fa82dbe2c4f520b99160af022e5c4f7fac, and rustfmt --check --edition 2024 on that exact file reproduces the hunk. I am leaving the unrelated base file unchanged and will rerun quick, then run the focused tests and final gate with this baseline fmt failure identified.

J288 `verify quick` fmt finding (log `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-post-rebase.log`): the formatter found four layout differences in my changed mesh files and one in untouched `crates/plugins/remote/src/uninstall.rs`, then stopped before clippy or tests. I corrected the four mesh hunks. The remaining hunk is in the fetched base: `git diff --quiet origin/dev -- crates/plugins/remote/src/uninstall.rs` passed, both the worktree and `origin/dev` file hash to `10bfc08086f06aacd7e01a1338c128fa82dbe2c4f520b99160af022e5c4f7fac`, and `rustfmt --check --edition 2024` on that exact file reproduces the hunk. I am leaving the unrelated base file unchanged and will rerun quick, then run the focused tests and final gate with this baseline fmt failure identified.
kayg commented 2026-11-06 12:17:44 +00:00
Author
Owner
Copy link

After correcting all four formatter hunks in the changed mesh files, I reran scripts/verify quick through build-slot. It is at queue position 3; all four slots were busy with two jobs ahead. Log: /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-post-fmt.log. The run will stop at the known untouched origin/dev formatting hunk unless upstream changes; no namespace proof is queued behind it.

After correcting all four formatter hunks in the changed mesh files, I reran `scripts/verify quick` through build-slot. It is at queue position 3; all four slots were busy with two jobs ahead. Log: `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-post-fmt.log`. The run will stop at the known untouched `origin/dev` formatting hunk unless upstream changes; no namespace proof is queued behind it.
kayg commented 2026-11-06 12:19:55 +00:00
Author
Owner
Copy link

The post-format rerun of scripts/verify quick finished with VERIFY_QUICK_EXIT=1. Scope guard, migration names and Nix Cargo vendoring passed; fmt stopped on the same single, unmodified origin/dev hunk in crates/plugins/remote/src/uninstall.rs. The exact baseline-file hash and reproducing rustfmt output are documented above. No clippy or unit tests ran after fmt stopped. Log: /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-post-fmt.log. I will run workspace clippy and the changed mesh package tests manually through build-slot, then continue with the namespace matrix.

The post-format rerun of `scripts/verify quick` finished with `VERIFY_QUICK_EXIT=1`. Scope guard, migration names and Nix Cargo vendoring passed; fmt stopped on the same single, unmodified `origin/dev` hunk in `crates/plugins/remote/src/uninstall.rs`. The exact baseline-file hash and reproducing rustfmt output are documented above. No clippy or unit tests ran after fmt stopped. Log: `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-post-fmt.log`. I will run workspace clippy and the changed mesh package tests manually through build-slot, then continue with the namespace matrix.
kayg commented 2026-11-06 12:21:56 +00:00
Author
Owner
Copy link

Because quick stopped at the known origin/dev fmt hunk, I submitted workspace clippy (--workspace --all-targets --locked -- -D warnings) plus all infraternal-plugin-mesh library tests with agent enabled as one build-slot run. It is at queue position 4 (three jobs ahead); no compilation has started. Launcher log: /home/claude/.cache/codex-scratch/j288/evidence/mesh-clippy-launch.log; result log: /home/claude/.cache/codex-scratch/j288/evidence/mesh-clippy-and-unit.log. The namespace matrix remains unqueued until this run completes.

Because quick stopped at the known `origin/dev` fmt hunk, I submitted workspace clippy (`--workspace --all-targets --locked -- -D warnings`) plus all `infraternal-plugin-mesh` library tests with `agent` enabled as one build-slot run. It is at queue position 4 (three jobs ahead); no compilation has started. Launcher log: `/home/claude/.cache/codex-scratch/j288/evidence/mesh-clippy-launch.log`; result log: `/home/claude/.cache/codex-scratch/j288/evidence/mesh-clippy-and-unit.log`. The namespace matrix remains unqueued until this run completes.
kayg commented 2026-11-06 12:23:35 +00:00
Author
Owner
Copy link

D345 doc-comment check found four changed macOS items without docs: remove_routes, PendingWorker::begin, RuntimeState, and RuntimeState::stop. The exact red result is /home/claude/.cache/codex-scratch/j288/evidence/d345-macos-doc-red.log; the same run passed the diff-practices and fixed-sleep checks. I added purpose/invariant/error docs for those four items and will rerun the documentation check before the next build-slot run.

D345 doc-comment check found four changed macOS items without docs: `remove_routes`, `PendingWorker::begin`, `RuntimeState`, and `RuntimeState::stop`. The exact red result is `/home/claude/.cache/codex-scratch/j288/evidence/d345-macos-doc-red.log`; the same run passed the diff-practices and fixed-sleep checks. I added purpose/invariant/error docs for those four items and will rerun the documentation check before the next build-slot run.
kayg commented 2026-11-06 12:24:29 +00:00
Author
Owner
Copy link

D345 follow-up passes after adding the four Mac docs: diff-practices 12/12, documentation coverage ok, and fixed-sleep check ok against merge base 74b9c4d8021d. Evidence: /home/claude/.cache/codex-scratch/j288/evidence/d345-macos-doc-green.log.

D345 follow-up passes after adding the four Mac docs: diff-practices 12/12, documentation coverage `ok`, and fixed-sleep check `ok` against merge base `74b9c4d8021d`. Evidence: `/home/claude/.cache/codex-scratch/j288/evidence/d345-macos-doc-green.log`.
kayg commented 2026-11-06 12:40:32 +00:00
Author
Owner
Copy link

J288 resume status (2026-11-06). Branch mesh-mtu-381 is at 2465a93461387f9b7555e911ae4f4bf4dabf88df; the current Linux/namespace and macOS changes remain local in the J288 worktree.

The prior J288 clippy/unit run was queued on compute01. I stopped only its recorded launcher and build-slot process (PIDs 663845 and 663854); the subsequent slot status no longer listed J288, and other jobs were left alone.

The Mac lock is currently held by another active build, and ~/mesh-vm.claim on compute01 belongs to J254-linux-proof (2026-11-06T11:54:42Z). I have not started a Mac build or changed live mesh state. Pending: rerun the current native Mac tests/build under lockf -k with CARGO_INCREMENTAL=0, remove the J288 target before unlocking, then take the matched Mac ↔ compute01 after-measurement under the claim and Mac lock. J267's before table and raw evidence paths remain recorded in #375; its summary is 0.5 / 0.2 Mbit/s at one Linux→Mac flow and 33.5 Mbit/s at eight.

**J288 resume status (2026-11-06).** Branch `mesh-mtu-381` is at `2465a93461387f9b7555e911ae4f4bf4dabf88df`; the current Linux/namespace and macOS changes remain local in the J288 worktree. The prior J288 clippy/unit run was queued on compute01. I stopped only its recorded launcher and build-slot process (PIDs 663845 and 663854); the subsequent slot status no longer listed J288, and other jobs were left alone. The Mac lock is currently held by another active build, and `~/mesh-vm.claim` on compute01 belongs to `J254-linux-proof` (`2026-11-06T11:54:42Z`). I have not started a Mac build or changed live mesh state. Pending: rerun the current native Mac tests/build under `lockf -k` with `CARGO_INCREMENTAL=0`, remove the J288 target before unlocking, then take the matched Mac ↔ compute01 after-measurement under the claim and Mac lock. J267's before table and raw evidence paths remain recorded in #375; its summary is 0.5 / 0.2 Mbit/s at one Linux→Mac flow and 33.5 Mbit/s at eight.
kayg commented 2026-11-06 12:47:51 +00:00
Author
Owner
Copy link

J288 branch refresh (2026-11-06). Rebased mesh-mtu-381 onto current origin/dev 7b14fbb664cbfb9b2b9b8a56f724d924424accdb and pushed the rewritten branch with --force-with-lease; new head is c6673838d. The nine-file working-tree implementation remains local and uncommitted, so the prior Mac binary is not evidence for this refreshed source. J207's Mac build has released /tmp/mac-bench.lock; I will rerun the focused native tests and release build under the lock, with CARGO_INCREMENTAL=0 and target cleanup before unlock. The compute01 mesh claim still belongs to J254, so no live path measurement has started.

**J288 branch refresh (2026-11-06).** Rebased `mesh-mtu-381` onto current `origin/dev` `7b14fbb664cbfb9b2b9b8a56f724d924424accdb` and pushed the rewritten branch with `--force-with-lease`; new head is `c6673838d`. The nine-file working-tree implementation remains local and uncommitted, so the prior Mac binary is not evidence for this refreshed source. J207's Mac build has released `/tmp/mac-bench.lock`; I will rerun the focused native tests and release build under the lock, with `CARGO_INCREMENTAL=0` and target cleanup before unlock. The compute01 mesh claim still belongs to J254, so no live path measurement has started.
kayg commented 2026-11-06 12:57:43 +00:00
Author
Owner
Copy link

J288 Mac build command finding. Under /tmp/mac-bench.lock, the exact current MTU tests passed 9/9 (~/.cache/codex-scratch/j288/evidence/mac-mtu-current-native.log). My release-build command also enabled the opt-in desktop feature. That pulled apple-metal into the CLI build, whose SwiftPM step failed because the Command Line Tools compiler could not load the arm64 macOS 14.0 standard library while using the Nix 14.4 SDK. The target was removed before unlocking and the Mac returned to 25 GiB free. This is outside the default production CLI feature set; I am retrying the normal cargo build --locked --release -p infraternal path, which includes the mesh agent by default, under a fresh lock-held build step.

**J288 Mac build command finding.** Under `/tmp/mac-bench.lock`, the exact current MTU tests passed 9/9 (`~/.cache/codex-scratch/j288/evidence/mac-mtu-current-native.log`). My release-build command also enabled the opt-in `desktop` feature. That pulled `apple-metal` into the CLI build, whose SwiftPM step failed because the Command Line Tools compiler could not load the arm64 macOS 14.0 standard library while using the Nix 14.4 SDK. The target was removed before unlocking and the Mac returned to 25 GiB free. This is outside the default production CLI feature set; I am retrying the normal `cargo build --locked --release -p infraternal` path, which includes the mesh agent by default, under a fresh lock-held build step.
kayg commented 2026-11-06 13:05:14 +00:00
Author
Owner
Copy link

J288 current-source Mac proof. On the rebased branch worktree, the MTU route/parser tests passed 9/9 under /tmp/mac-bench.lock with CARGO_INCREMENTAL=0; log: /Users/claude/.cache/codex-scratch/j288/evidence/mac-mtu-current-native.log. The default production release build (cargo build --locked --release -p infraternal) passed in a separate lock-held step; binary SHA-256 e1cd1aed7b7b4083aa03fc91615e1f438ad269163a071c9a28971b096e6d9c86, version infraternal 0.1.0 (stable). Build log and hash/version receipts: /Users/claude/.cache/codex-scratch/j288/evidence/mac-mtu-current-release.log, infraternal-mtu381-rebased.sha256, and .version. Both steps removed ~/worktrees/j288/target before unlocking; disk returned to 25 GiB free.

The live Mac ↔ compute01 after-measurement remains pending: compute01's ~/mesh-vm.claim still belongs to J254-linux-proof. J267's measured before rows and the source raw JSON remain available at #375 and /Users/lume/j267-evidence/m-{baseline-netbird,mesh-gotatun,baseline-netbird-2}/.

**J288 current-source Mac proof.** On the rebased branch worktree, the MTU route/parser tests passed **9/9** under `/tmp/mac-bench.lock` with `CARGO_INCREMENTAL=0`; log: `/Users/claude/.cache/codex-scratch/j288/evidence/mac-mtu-current-native.log`. The default production release build (`cargo build --locked --release -p infraternal`) passed in a separate lock-held step; binary SHA-256 `e1cd1aed7b7b4083aa03fc91615e1f438ad269163a071c9a28971b096e6d9c86`, version `infraternal 0.1.0 (stable)`. Build log and hash/version receipts: `/Users/claude/.cache/codex-scratch/j288/evidence/mac-mtu-current-release.log`, `infraternal-mtu381-rebased.sha256`, and `.version`. Both steps removed `~/worktrees/j288/target` before unlocking; disk returned to 25 GiB free. The live Mac ↔ compute01 after-measurement remains pending: compute01's `~/mesh-vm.claim` still belongs to `J254-linux-proof`. J267's measured before rows and the source raw JSON remain available at #375 and `/Users/lume/j267-evidence/m-{baseline-netbird,mesh-gotatun,baseline-netbird-2}/`.
kayg commented 2026-11-06 13:08:46 +00:00
Author
Owner
Copy link

J288 latest-base update. origin/dev advanced with 68ee8d9a5 (dev-fmt-fix), fixing the previously observed upstream-only uninstall.rs rustfmt hunk. I rebased and pushed mesh-mtu-381 again; current branch head is 1a44b8b4f256. The Mac MTU test and default release build passed on the immediately prior base; this rebase changes only unrelated server/uninstall files, and the changed mesh sources are unchanged. The compute01 J288 worktree is now reset to this head plus the current nine-file working-tree implementation; its existing 7.1 GiB target is retained for quick/fast.

Next: verify quick, then the Linux namespace MTU matrix, then verify fast. The live Mac ↔ compute01 measurement remains blocked on J254's ~/mesh-vm.claim.

**J288 latest-base update.** `origin/dev` advanced with `68ee8d9a5` (`dev-fmt-fix`), fixing the previously observed upstream-only `uninstall.rs` rustfmt hunk. I rebased and pushed `mesh-mtu-381` again; current branch head is `1a44b8b4f256`. The Mac MTU test and default release build passed on the immediately prior base; this rebase changes only unrelated server/uninstall files, and the changed mesh sources are unchanged. The compute01 J288 worktree is now reset to this head plus the current nine-file working-tree implementation; its existing 7.1 GiB target is retained for quick/fast. Next: `verify quick`, then the Linux namespace MTU matrix, then `verify fast`. The live Mac ↔ compute01 measurement remains blocked on J254's `~/mesh-vm.claim`.
kayg commented 2026-11-06 13:27:32 +00:00
Author
Owner
Copy link

J288 continuation after the std-VM freeze lifted (14:36 IST):

  • Current-source native Mac MTU/parser tests passed 9/9 and the default macOS release build passed under /tmp/mac-bench.lock, with CARGO_INCREMENTAL=0. The Mac target was removed before unlocking; free space returned to 25 GiB. Evidence: /Users/claude/.cache/codex-scratch/j288/evidence/mac-mtu-current-native.log, mac-mtu-current-release.log, infraternal-mtu381-rebased.sha256, and .version.
  • No J288 quick gate is currently running on compute01. The four slots are busy and nine jobs are queued; I will queue the requested gate on compute01 through build-slot. dev-02 has a free slot but is aarch64-only.
  • The compute01 mesh claim still says J254-linux-proof 2026-11-06T11:54:42Z. I will not override it; the real Mac ↔ compute01 after-measurement remains pending until it is released.
  • I will recheck the D348 perf VM pinning/no-interference readiness before attempting the normal-MTU measurement.
J288 continuation after the std-VM freeze lifted (14:36 IST): - Current-source native Mac MTU/parser tests passed 9/9 and the default macOS release build passed under /tmp/mac-bench.lock, with CARGO_INCREMENTAL=0. The Mac target was removed before unlocking; free space returned to 25 GiB. Evidence: /Users/claude/.cache/codex-scratch/j288/evidence/mac-mtu-current-native.log, mac-mtu-current-release.log, infraternal-mtu381-rebased.sha256, and .version. - No J288 quick gate is currently running on compute01. The four slots are busy and nine jobs are queued; I will queue the requested gate on compute01 through build-slot. dev-02 has a free slot but is aarch64-only. - The compute01 mesh claim still says J254-linux-proof 2026-11-06T11:54:42Z. I will not override it; the real Mac ↔ compute01 after-measurement remains pending until it is released. - I will recheck the D348 perf VM pinning/no-interference readiness before attempting the normal-MTU measurement.
kayg commented 2026-11-06 13:32:25 +00:00
Author
Owner
Copy link

J288 branch refresh and gate queue:

  • Rebased onto origin/dev 48358a5fab (new HEAD 9cadc3ceaf) and pushed mesh-mtu-381 with --force-with-lease. The nine-file working diff was preserved and synced to compute01.
  • verify quick is queued on compute01 via build-slot; launcher PID 1673800. Log: /home/claude/.cache/codex-scratch/j288/logs/verify-quick-after-lift.log.
  • I rechecked the D348 perf VM: AllowedCPUs and effective cpuset remain 1-3, which cannot provide disjoint two-CPU sets for both benchmark sides. I will leave the masks untouched and will not claim a normal-MTU perf hold without D348 readiness.
  • The Mac native test/build logs predate this unrelated upstream rebase; I will rerun the current-branch proof under the Mac lock before live measurement.
J288 branch refresh and gate queue: - Rebased onto origin/dev 48358a5fabab (new HEAD 9cadc3ceaf00067883039e53a1257c5e0c997ed8) and pushed mesh-mtu-381 with --force-with-lease. The nine-file working diff was preserved and synced to compute01. - verify quick is queued on compute01 via build-slot; launcher PID 1673800. Log: /home/claude/.cache/codex-scratch/j288/logs/verify-quick-after-lift.log. - I rechecked the D348 perf VM: AllowedCPUs and effective cpuset remain 1-3, which cannot provide disjoint two-CPU sets for both benchmark sides. I will leave the masks untouched and will not claim a normal-MTU perf hold without D348 readiness. - The Mac native test/build logs predate this unrelated upstream rebase; I will rerun the current-branch proof under the Mac lock before live measurement.
kayg commented 2026-11-06 13:47:17 +00:00
Author
Owner
Copy link

J288 current-base macOS proof completed under the Mac lock on source head 9cadc3ceaf (origin/dev 48358a5fab):

  • Native MTU route/parser tests: 9 passed, 0 failed, 108 filtered.
  • Default release CLI build: passed in 5m 34s; version infraternal 0.1.0 (stable); binary SHA-256 49eccab282edbcb43617d17268d82905f4f952e19b0b06dbd81c26bd16064ed1.
  • Evidence on Mac: /Users/claude/.cache/codex-scratch/j288/evidence/mac-mtu-current-base-native.log, mac-mtu-current-base-release.log, mac-mtu-current-base-source.txt, infraternal-mtu381-current-base.sha256, and infraternal-mtu381-current-base.version. The target is absent and free disk is 24 GiB after the lock-held cleanup. Release warnings were in vendored russh, shim, and remote code; no changed mesh warnings appeared.

The Mac ↔ compute01 live after-measurement still cannot start while compute01 mesh-vm.claim belongs to J254-linux-proof.

J288 current-base macOS proof completed under the Mac lock on source head 9cadc3ceaf00067883039e53a1257c5e0c997ed8 (origin/dev 48358a5fabab): - Native MTU route/parser tests: 9 passed, 0 failed, 108 filtered. - Default release CLI build: passed in 5m 34s; version infraternal 0.1.0 (stable); binary SHA-256 49eccab282edbcb43617d17268d82905f4f952e19b0b06dbd81c26bd16064ed1. - Evidence on Mac: /Users/claude/.cache/codex-scratch/j288/evidence/mac-mtu-current-base-native.log, mac-mtu-current-base-release.log, mac-mtu-current-base-source.txt, infraternal-mtu381-current-base.sha256, and infraternal-mtu381-current-base.version. The target is absent and free disk is 24 GiB after the lock-held cleanup. Release warnings were in vendored russh, shim, and remote code; no changed mesh warnings appeared. The Mac ↔ compute01 live after-measurement still cannot start while compute01 mesh-vm.claim belongs to J254-linux-proof.
kayg commented 2026-11-06 13:48:42 +00:00
Author
Owner
Copy link

J288 quick finding: the first post-rebase quick run stopped at fmt with three rustfmt hunks, all in the changed crates/plugins/mesh/src/macos_node.rs (Tokio import wrapping, route signature wrapping, and a test assertion wrap). Full log: /home/claude/.cache/codex-scratch/j288/logs/verify-quick-after-lift.log. I applied rustfmt to that file locally and am syncing it before rerunning quick. No compilation or unit-test rows ran in that attempt.

J288 quick finding: the first post-rebase quick run stopped at fmt with three rustfmt hunks, all in the changed crates/plugins/mesh/src/macos_node.rs (Tokio import wrapping, route signature wrapping, and a test assertion wrap). Full log: /home/claude/.cache/codex-scratch/j288/logs/verify-quick-after-lift.log. I applied rustfmt to that file locally and am syncing it before rerunning quick. No compilation or unit-test rows ran in that attempt.
kayg commented 2026-11-06 14:04:57 +00:00
Author
Owner
Copy link

J288 quick finding after the formatting fix: fmt, doc coverage, D351 shape checks, D345 doc comments and fixed-wait checks all passed. Workspace clippy then hit E0500 in the new Linux endpoint-only update path at node.rs:1712: the fallible closure captured all of Node for set_mesh_mtu while a borrowed old Config remained live for rollback. Evidence: /home/claude/.cache/codex-scratch/j288/logs/verify-quick-after-fmt.log.

I moved the shared interface MTU update after the engine/route closure returns. That ends the mutable engine-field borrow before the shared update, preserves the same rollback path, and avoids cloning the old Config on every endpoint update. The corrected node.rs is synced to compute01; I am rerunning quick through build-slot.

J288 quick finding after the formatting fix: fmt, doc coverage, D351 shape checks, D345 doc comments and fixed-wait checks all passed. Workspace clippy then hit E0500 in the new Linux endpoint-only update path at node.rs:1712: the fallible closure captured all of Node for set_mesh_mtu while a borrowed old Config remained live for rollback. Evidence: /home/claude/.cache/codex-scratch/j288/logs/verify-quick-after-fmt.log. I moved the shared interface MTU update after the engine/route closure returns. That ends the mutable engine-field borrow before the shared update, preserves the same rollback path, and avoids cloning the old Config on every endpoint update. The corrected node.rs is synced to compute01; I am rerunning quick through build-slot.
kayg commented 2026-11-06 14:22:56 +00:00
Author
Owner
Copy link

J288 continuation after the 14:36 IST freeze lift. Branch mesh-mtu-381 is at 9cadc3ceaf; the working tree includes the pending cross-platform route-MTU implementation. The last quick attempt stopped at clippy with E0500 in node.rs; the follow-up edit is present locally, but the recorded launcher PID is gone and build-slot status has no J288 entry, so I will rerun quick after the focused checks. The Mac target is absent and 25 GiB is free. I will rebuild and test the current source under lockf -k /tmp/mac-bench.lock with CARGO_INCREMENTAL=0 and delete the target before unlocking, then run the real Mac-to-compute01 measurement under ~/mesh-vm.claim plus the Mac lock. Compute01 mesh claim is currently free. The D348 perf VM still exposes AllowedCPUs and cpuset.cpus.effective 1-3, only three CPUs, so it cannot satisfy the required disjoint 2+2 pinning; I will not change its masks.

J288 continuation after the 14:36 IST freeze lift. Branch mesh-mtu-381 is at 9cadc3ceaf00067883039e53a1257c5e0c997ed8; the working tree includes the pending cross-platform route-MTU implementation. The last quick attempt stopped at clippy with E0500 in node.rs; the follow-up edit is present locally, but the recorded launcher PID is gone and build-slot status has no J288 entry, so I will rerun quick after the focused checks. The Mac target is absent and 25 GiB is free. I will rebuild and test the current source under lockf -k /tmp/mac-bench.lock with CARGO_INCREMENTAL=0 and delete the target before unlocking, then run the real Mac-to-compute01 measurement under ~/mesh-vm.claim plus the Mac lock. Compute01 mesh claim is currently free. The D348 perf VM still exposes AllowedCPUs and cpuset.cpus.effective 1-3, only three CPUs, so it cannot satisfy the required disjoint 2+2 pinning; I will not change its masks.
kayg commented 2026-11-06 14:48:22 +00:00
Author
Owner
Copy link

J288 Mac proof on the current cross-platform source completed under /tmp/mac-bench.lock with CARGO_INCREMENTAL=0. The mesh library suite passed 69 tests, 0 failed; the default production release build passed in 5m32s and reported infraternal 0.1.0 (stable). Candidate binary SHA-256: 3a503dd376290820c315b1a7f39632f8ad6ea74323520c9267329f4764e2ca32. Evidence on the Mac: /Users/claude/.cache/codex-scratch/j288/evidence/mac-mtu-lockstep-unit.log, mac-mtu-lockstep-release.log, mac-mtu-lockstep-source.txt, infraternal-mtu381-candidate.sha256, infraternal-mtu381-candidate.version, and mac-mtu-lockstep-cleanup.log. Cleanup reports COMMAND_EXIT=0, TARGET_PRESENT=no; free space returned to 24 GiB before the lock released. Next: validate current Linux changes on compute01, then claim the mesh path and run the live Mac-to-compute01 before/after set.

J288 Mac proof on the current cross-platform source completed under /tmp/mac-bench.lock with CARGO_INCREMENTAL=0. The mesh library suite passed 69 tests, 0 failed; the default production release build passed in 5m32s and reported infraternal 0.1.0 (stable). Candidate binary SHA-256: 3a503dd376290820c315b1a7f39632f8ad6ea74323520c9267329f4764e2ca32. Evidence on the Mac: /Users/claude/.cache/codex-scratch/j288/evidence/mac-mtu-lockstep-unit.log, mac-mtu-lockstep-release.log, mac-mtu-lockstep-source.txt, infraternal-mtu381-candidate.sha256, infraternal-mtu381-candidate.version, and mac-mtu-lockstep-cleanup.log. Cleanup reports COMMAND_EXIT=0, TARGET_PRESENT=no; free space returned to 24 GiB before the lock released. Next: validate current Linux changes on compute01, then claim the mesh path and run the live Mac-to-compute01 before/after set.
kayg commented 2026-11-06 14:54:46 +00:00
Author
Owner
Copy link

J288 quick is waiting through the required compute01 build-slot. At 15:54 UTC it is queue position 12 with all four slots busy; compilation has not started. The foreground launcher and log are /home/claude/.cache/codex-scratch/j288/run-verify-quick.sh and /home/claude/.cache/codex-scratch/j288/logs/verify-quick-current.log. The existing 7.1 GiB target is retained at /home/claude/worktrees/j288/target for quick and fast. I will wait for this quick run before live measurement so build activity does not overlap the throughput sample.

J288 quick is waiting through the required compute01 build-slot. At 15:54 UTC it is queue position 12 with all four slots busy; compilation has not started. The foreground launcher and log are /home/claude/.cache/codex-scratch/j288/run-verify-quick.sh and /home/claude/.cache/codex-scratch/j288/logs/verify-quick-current.log. The existing 7.1 GiB target is retained at /home/claude/worktrees/j288/target for quick and fast. I will wait for this quick run before live measurement so build activity does not overlap the throughput sample.
kayg commented 2026-11-06 15:11:52 +00:00
Author
Owner
Copy link

J288 queue and disk update: the quick launcher is still last in compute01 build-slot and has not started. Free disk is 33 GiB against the 25 GiB admission floor; the existing 7.1 GiB target remains in place for both requested gates. I will keep the same target path and wait for the slot/disk check rather than create another target or move the x86 gate to dev-02.

J288 queue and disk update: the quick launcher is still last in compute01 build-slot and has not started. Free disk is 33 GiB against the 25 GiB admission floor; the existing 7.1 GiB target remains in place for both requested gates. I will keep the same target path and wait for the slot/disk check rather than create another target or move the x86 gate to dev-02.
kayg commented 2026-11-06 15:17:16 +00:00
Author
Owner
Copy link

J288 quick remains queued and is now also below the compute01 disk admission floor: build-slot reports 24 GiB free versus 25 GiB required, with all slots busy. No J288 compilation has started; the 7.1 GiB target stays retained for the quick/fast pair. Log: /home/claude/.cache/codex-scratch/j288/logs/verify-quick-current.log. I will continue waiting for the host to regain a slot and disk headroom.

J288 quick remains queued and is now also below the compute01 disk admission floor: build-slot reports 24 GiB free versus 25 GiB required, with all slots busy. No J288 compilation has started; the 7.1 GiB target stays retained for the quick/fast pair. Log: /home/claude/.cache/codex-scratch/j288/logs/verify-quick-current.log. I will continue waiting for the host to regain a slot and disk headroom.
kayg commented 2026-11-06 15:23:27 +00:00
Author
Owner
Copy link

J288 quick result: scope, migration names, vendoring, fmt, documentation ratchet/comments and fixed-wait checks passed. Workspace Clippy stopped on one changed-code pedantic lint in mesh_mtu.rs:42 ( should use ); unit tests did not run. Full log: /home/claude/.cache/codex-scratch/j288/logs/verify-quick-current.log. I replaced the equivalent expression with and will sync the one-file fix before rerunning quick.

J288 quick result: scope, migration names, vendoring, fmt, documentation ratchet/comments and fixed-wait checks passed. Workspace Clippy stopped on one changed-code pedantic lint in mesh_mtu.rs:42 ( should use ); unit tests did not run. Full log: /home/claude/.cache/codex-scratch/j288/logs/verify-quick-current.log. I replaced the equivalent expression with and will sync the one-file fix before rerunning quick.
kayg commented 2026-11-06 15:23:41 +00:00
Author
Owner
Copy link

J288 quick result: scope, migration names, vendoring, fmt, documentation ratchet/comments and fixed-wait checks passed. Workspace Clippy stopped on one changed-code pedantic lint in mesh_mtu.rs:42: map(...).unwrap_or(...) should use map_or; unit tests did not run. Full log: /home/claude/.cache/codex-scratch/j288/logs/verify-quick-current.log. I replaced the equivalent expression with map_or and will sync the one-file fix before rerunning quick.

J288 quick result: scope, migration names, vendoring, fmt, documentation ratchet/comments and fixed-wait checks passed. Workspace Clippy stopped on one changed-code pedantic lint in mesh_mtu.rs:42: map(...).unwrap_or(...) should use map_or; unit tests did not run. Full log: /home/claude/.cache/codex-scratch/j288/logs/verify-quick-current.log. I replaced the equivalent expression with map_or and will sync the one-file fix before rerunning quick.
kayg commented 2026-11-06 15:34:28 +00:00
Author
Owner
Copy link

J288 verify quick pre-check passed on compute01 after the map_or correction. Scope, migration names, vendoring, fmt, D345/D351 checks, workspace Clippy and changed-package unit tests all passed; unit tests were 55 passed, 0 failed. The verbatim pre-check table and VERIFY_QUICK_EXIT=0 are in /home/claude/.cache/codex-scratch/j288/logs/verify-quick-current-r2.log. This is not the final gate. Next I will run the after-fix 1280-underlay namespace proof for both Linux engines, then commit/push and run verify fast.

J288 verify quick pre-check passed on compute01 after the map_or correction. Scope, migration names, vendoring, fmt, D345/D351 checks, workspace Clippy and changed-package unit tests all passed; unit tests were 55 passed, 0 failed. The verbatim pre-check table and VERIFY_QUICK_EXIT=0 are in /home/claude/.cache/codex-scratch/j288/logs/verify-quick-current-r2.log. This is not the final gate. Next I will run the after-fix 1280-underlay namespace proof for both Linux engines, then commit/push and run verify fast.
kayg commented 2026-11-06 15:40:17 +00:00
Author
Owner
Copy link

J288 namespace-runner attempt stopped before compiling or running the fixture: its outer nix develop was called from /home/claude instead of the worktree and reported that no flake.nix could be found. Evidence: /home/claude/.cache/codex-scratch/j288/evidence/namespace-mtu-after-1280.log (MESH_PROOF_EXIT=1). This is a runner cwd error, not a product result. I will change the runner to cd into ~/worktrees/j288 before nix develop and retry through build-slot.

J288 namespace-runner attempt stopped before compiling or running the fixture: its outer nix develop was called from /home/claude instead of the worktree and reported that no flake.nix could be found. Evidence: /home/claude/.cache/codex-scratch/j288/evidence/namespace-mtu-after-1280.log (MESH_PROOF_EXIT=1). This is a runner cwd error, not a product result. I will change the runner to cd into ~/worktrees/j288 before nix develop and retry through build-slot.
kayg commented 2026-11-06 15:51:41 +00:00
Author
Owner
Copy link

J288 continuation: the std freeze is lifted. One source change (the map_or lint fix in mesh_mtu.rs) landed after the previous Mac run, so the final Mac unit/build proof remains pending. I will hold lockf -k /tmp/mac-bench.lock for the entire remaining Mac build/test/measurement step, keep CARGO_INCREMENTAL=0, and remove the Mac target before unlocking. Compute01 is currently full with six jobs queued; this continuation has not queued new build work. The first namespace attempt failed before testing because nix develop ran outside the worktree (namespace-mtu-after-1280.log); the corrected runner now enters the worktree, and I am waiting to rerun the 1280 boundary proof through build-slot. D348 perf readiness will be checked again before measuring; earlier evidence showed only CPUs 1-3 available, which cannot provide disjoint 2+2 pinning.

J288 continuation: the std freeze is lifted. One source change (the map_or lint fix in mesh_mtu.rs) landed after the previous Mac run, so the final Mac unit/build proof remains pending. I will hold lockf -k /tmp/mac-bench.lock for the entire remaining Mac build/test/measurement step, keep CARGO_INCREMENTAL=0, and remove the Mac target before unlocking. Compute01 is currently full with six jobs queued; this continuation has not queued new build work. The first namespace attempt failed before testing because nix develop ran outside the worktree (namespace-mtu-after-1280.log); the corrected runner now enters the worktree, and I am waiting to rerun the 1280 boundary proof through build-slot. D348 perf readiness will be checked again before measuring; earlier evidence showed only CPUs 1-3 available, which cannot provide disjoint 2+2 pinning.
kayg commented 2026-11-06 16:18:41 +00:00
Author
Owner
Copy link

J288 implementation commit 2171a4b409413f57d2f3d11fb5cba1b7c4733025 is pushed on mesh-mtu-381. Current-source Mac proof is complete under lockf -k /tmp/mac-bench.lock with CARGO_INCREMENTAL=0. The explicit agent-feature suite includes macos_node and mesh_mtu: test result: ok. 117 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.60s. The default release build completed with COMMAND_EXIT=0; candidate SHA-256 c9f48c47986179e14594b24c150b4cb5bcf90229e64f8ea3112f7a2835e477f4; version infraternal 0.1.0 (stable). Evidence: /Users/claude/.cache/codex-scratch/j288/evidence/mac-mtu-final-agent-tests.log, mac-mtu-final-release.log, mac-mtu-final-agent-cleanup.log, infraternal-mtu381-final.sha256, infraternal-mtu381-final.version. Cleanup reports TARGET_PRESENT=no and 24 GiB free before lock release. The after-fix Linux 1280-underlay boundary proof for both engines is now running on compute01.

J288 implementation commit 2171a4b409413f57d2f3d11fb5cba1b7c4733025 is pushed on mesh-mtu-381. Current-source Mac proof is complete under lockf -k /tmp/mac-bench.lock with CARGO_INCREMENTAL=0. The explicit agent-feature suite includes macos_node and mesh_mtu: test result: ok. 117 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.60s. The default release build completed with COMMAND_EXIT=0; candidate SHA-256 c9f48c47986179e14594b24c150b4cb5bcf90229e64f8ea3112f7a2835e477f4; version infraternal 0.1.0 (stable). Evidence: /Users/claude/.cache/codex-scratch/j288/evidence/mac-mtu-final-agent-tests.log, mac-mtu-final-release.log, mac-mtu-final-agent-cleanup.log, infraternal-mtu381-final.sha256, infraternal-mtu381-final.version. Cleanup reports TARGET_PRESENT=no and 24 GiB free before lock release. The after-fix Linux 1280-underlay boundary proof for both engines is now running on compute01.
kayg commented 2026-11-06 16:26:37 +00:00
Author
Owner
Copy link

Correction: compute01 filled before the namespace proof acquired a slot; my queued run was seventh and was canceled before compilation (namespace-mtu-after-1280-r2.log ends with MESH_PROOF_EXIT=143 and contains no build or product result). Dev-02 had one free aarch64 slot, so I checked out the pushed 2171a4b40 source there and started the same proof with CARGO_INCREMENTAL=0; its single target is removed on exit. The x86 verify fast gate and live Mac-to-compute01 measurement remain on their required hosts.

Correction: compute01 filled before the namespace proof acquired a slot; my queued run was seventh and was canceled before compilation (namespace-mtu-after-1280-r2.log ends with MESH_PROOF_EXIT=143 and contains no build or product result). Dev-02 had one free aarch64 slot, so I checked out the pushed 2171a4b40 source there and started the same proof with CARGO_INCREMENTAL=0; its single target is removed on exit. The x86 verify fast gate and live Mac-to-compute01 measurement remain on their required hosts.
kayg commented 2026-11-06 16:27:23 +00:00
Author
Owner
Copy link

J288 1280-byte boundary proof is green on the pushed 2171a4b40 source. Before-fix remains the red fixture run on compute01: interface MTUs 1380/1380, no route metrics, four outer IPv4 fragments and three receiver reassembly failures; small IPv4/IPv6 passed while oversized IPv4/IPv6 were lost. Evidence: /home/claude/.cache/codex-scratch/j288/evidence/mesh-mtu-baseline-boundary.log on compute01. After-fix ran on dev-02 aarch64 through its free build slot: both GotaTun and kernel kept interface MTUs 1280/1280 and installed IPv4+IPv6 peer route MTUs 1200/1200; small probes passed, oversized IPv4 was limited by the route, IPv6 source-fragmented to route MTU, outer fragments=0 and reassembly failures=0. MESH_PROOF_EXIT=0. Evidence: /home/claude/.cache/codex-scratch/j288/evidence/namespace-mtu-after-1280-aarch64.log on dev-02. The target was removed (TARGET_PRESENT=no); free disk 111 GiB.

J288 1280-byte boundary proof is green on the pushed 2171a4b40 source. Before-fix remains the red fixture run on compute01: interface MTUs 1380/1380, no route metrics, four outer IPv4 fragments and three receiver reassembly failures; small IPv4/IPv6 passed while oversized IPv4/IPv6 were lost. Evidence: /home/claude/.cache/codex-scratch/j288/evidence/mesh-mtu-baseline-boundary.log on compute01. After-fix ran on dev-02 aarch64 through its free build slot: both GotaTun and kernel kept interface MTUs 1280/1280 and installed IPv4+IPv6 peer route MTUs 1200/1200; small probes passed, oversized IPv4 was limited by the route, IPv6 source-fragmented to route MTU, outer fragments=0 and reassembly failures=0. MESH_PROOF_EXIT=0. Evidence: /home/claude/.cache/codex-scratch/j288/evidence/namespace-mtu-after-1280-aarch64.log on dev-02. The target was removed (TARGET_PRESENT=no); free disk 111 GiB.
kayg commented 2026-11-06 16:33:17 +00:00
Author
Owner
Copy link

J288 continuation after the std freeze lift. Current pushed head is 2171a4b406; the Mac current-source test/build and 1280-underlay proof are complete, with their targets removed. I found origin/dev has advanced beyond this branch, so I will rebase and rerun the required final gate on the refreshed head. Next I will run the live after-measurement on the J267 Mac to compute01 path under ~/mesh-vm.claim and the Mac lock, using J267's recorded plain and pre-fix mesh runs on #375 as the before baseline. The D348 perf VM still exposes only CPUs 1-3; I will recheck its readiness and will not change masks or claim a valid normal-MTU hold without the required disjoint 2+2 setup. Existing baseline and Mac proof evidence remain at the paths recorded in the prior #381 comments.

J288 continuation after the std freeze lift. Current pushed head is 2171a4b406b5b0a62fc33a0b54c30e7da4f12d88; the Mac current-source test/build and 1280-underlay proof are complete, with their targets removed. I found origin/dev has advanced beyond this branch, so I will rebase and rerun the required final gate on the refreshed head. Next I will run the live after-measurement on the J267 Mac to compute01 path under ~/mesh-vm.claim and the Mac lock, using J267's recorded plain and pre-fix mesh runs on #375 as the before baseline. The D348 perf VM still exposes only CPUs 1-3; I will recheck its readiness and will not change masks or claim a valid normal-MTU hold without the required disjoint 2+2 setup. Existing baseline and Mac proof evidence remain at the paths recorded in the prior #381 comments.
kayg commented 2026-11-06 16:37:29 +00:00
Author
Owner
Copy link

J288 branch refresh is pushed at a6704428c8, based on current origin/dev 37c93637e4. I collapsed the four incremental feature commits into one atomic #381 commit; the implementation tree is unchanged. Compute01's checkout is synced to that head and its 7.3 GiB target is retained. The next action is verify quick on compute01, then fast. The latest D348 readiness recheck still shows only CPUs 1-3 effective on the perf VM, so the normal-MTU performance run remains blocked by the required disjoint 2+2 setup.

J288 branch refresh is pushed at a6704428c8241c9ae38cd02413af3366a41c985c, based on current origin/dev 37c93637e49a415578416be037f972b9a716701c. I collapsed the four incremental feature commits into one atomic #381 commit; the implementation tree is unchanged. Compute01's checkout is synced to that head and its 7.3 GiB target is retained. The next action is verify quick on compute01, then fast. The latest D348 readiness recheck still shows only CPUs 1-3 effective on the perf VM, so the normal-MTU performance run remains blocked by the required disjoint 2+2 setup.
kayg commented 2026-11-06 16:55:06 +00:00
Author
Owner
Copy link

Mac recheck on refreshed head: the mesh library tests pass (117 passed, 0 failed) with --lib --features agent; evidence: /Users/claude/.cache/codex-scratch/j288/evidence/mac-mtu-current-reverify-tests.log. The first broad cargo test --features agent invocation was invalid on macOS because it also compiled the Linux-only tests/access_updates.rs target; its E0432/E0433 output is in mac-mtu-final-tests.log.

The refreshed default release CLI build fails in upstream crates/infraternal-agent/src/logins.rs:424 because nix::SockFlag::SOCK_CLOEXEC is unavailable on macOS. This file changed in the current origin/dev history (D354); the mesh files did not change between the previously tested base and current dev. Full compiler output: /Users/claude/.cache/codex-scratch/j288/evidence/mac-mtu-current-reverify-release.log. Target cleanup succeeded under the Mac lock (TARGET_PRESENT=no; mac-mtu-current-reverify-cleanup.log). I will use the already-built candidate from the same mesh source tree for the live path measurement: /tmp/infraternal-mtu381-final, SHA-256 c9f48c47986179e14594b24c150b4cb5bcf90229e64f8ea3112f7a2835e477f4; its prior current-source test/build receipts are in the previous #381 comment.

Mac recheck on refreshed head: the mesh library tests pass (117 passed, 0 failed) with `--lib --features agent`; evidence: `/Users/claude/.cache/codex-scratch/j288/evidence/mac-mtu-current-reverify-tests.log`. The first broad `cargo test --features agent` invocation was invalid on macOS because it also compiled the Linux-only `tests/access_updates.rs` target; its E0432/E0433 output is in `mac-mtu-final-tests.log`. The refreshed default release CLI build fails in upstream `crates/infraternal-agent/src/logins.rs:424` because `nix::SockFlag::SOCK_CLOEXEC` is unavailable on macOS. This file changed in the current `origin/dev` history (D354); the mesh files did not change between the previously tested base and current dev. Full compiler output: `/Users/claude/.cache/codex-scratch/j288/evidence/mac-mtu-current-reverify-release.log`. Target cleanup succeeded under the Mac lock (`TARGET_PRESENT=no`; `mac-mtu-current-reverify-cleanup.log`). I will use the already-built candidate from the same mesh source tree for the live path measurement: `/tmp/infraternal-mtu381-final`, SHA-256 `c9f48c47986179e14594b24c150b4cb5bcf90229e64f8ea3112f7a2835e477f4`; its prior current-source test/build receipts are in the previous #381 comment.
kayg commented 2026-11-06 17:06:17 +00:00
Author
Owner
Copy link

J288 verify quick passed on current head a6704428c8241c9ae38cd02413af3366a41c985c (merge base 37c93637e49a415578416be037f972b9a716701c): all scope, migration, vendoring, fmt, D345/D351, workspace Clippy and changed-package test rows passed; mesh package tests were 55 passed, 0 failed. Receipt with the exact summary table and VERIFY_QUICK_EXIT=0: claude@compute01:~/.cache/codex-scratch/j288/evidence/verify-quick-final-receipt.txt. This is the pre-check only. I will run verify fast after the Mac ↔ compute01 after-sample.

J288 `verify quick` passed on current head `a6704428c8241c9ae38cd02413af3366a41c985c` (merge base `37c93637e49a415578416be037f972b9a716701c`): all scope, migration, vendoring, fmt, D345/D351, workspace Clippy and changed-package test rows passed; mesh package tests were 55 passed, 0 failed. Receipt with the exact summary table and `VERIFY_QUICK_EXIT=0`: `claude@compute01:~/.cache/codex-scratch/j288/evidence/verify-quick-final-receipt.txt`. This is the pre-check only. I will run `verify fast` after the Mac ↔ compute01 after-sample.
kayg commented 2026-11-06 17:13:01 +00:00
Author
Owner
Copy link

Correction to the D348 status above: the current perf.slice mask is still 1-3, but J317 on #260 confirms the approved J88i runner temporarily expands the measured scope to CPUs 0-5 under the shared perf lock, assigns sides 1-2 and 3-4, and restores/verifies the original mask. The current blocker is J88i's explicit perf-lock handoff; its latest #260 update says the handoff has not yet been posted. I will not bypass that lock or change CPU masks. Fresh readiness receipt: /home/kayg/.cache/codex-scratch/j288/evidence/perf-d348-readiness-final.log (2026-11-06 17:11:54 UTC).

Correction to the D348 status above: the current `perf.slice` mask is still `1-3`, but J317 on #260 confirms the approved J88i runner temporarily expands the measured scope to CPUs 0-5 under the shared perf lock, assigns sides 1-2 and 3-4, and restores/verifies the original mask. The current blocker is J88i's explicit perf-lock handoff; its latest #260 update says the handoff has not yet been posted. I will not bypass that lock or change CPU masks. Fresh readiness receipt: `/home/kayg/.cache/codex-scratch/j288/evidence/perf-d348-readiness-final.log` (2026-11-06 17:11:54 UTC).
kayg commented 2026-11-06 17:45:20 +00:00
Author
Owner
Copy link

Resume status: the Mac controller and Linux runner drafts are syntax-checked and use J267s same path, four iperf flow cases, a target-bound server, and a lockf-protected Mac run. I have not started the live test. compute01 ~/mesh-vm.claim is currently held by J279 (2026-11-06T16:15:33Z); I will wait for its release before changing mesh state. The fast gate is compiling on compute01 at PID 279492. D348 is still blocked for J288: J88is latest #260 note offers the next perf window to J316, not this job; I will not take the perf lock or alter CPU masks.

Resume status: the Mac controller and Linux runner drafts are syntax-checked and use J267s same path, four iperf flow cases, a target-bound server, and a lockf-protected Mac run. I have not started the live test. compute01 `~/mesh-vm.claim` is currently held by J279 (2026-11-06T16:15:33Z); I will wait for its release before changing mesh state. The fast gate is compiling on compute01 at PID 279492. D348 is still blocked for J288: J88is latest #260 note offers the next perf window to J316, not this job; I will not take the perf lock or alter CPU masks.
kayg commented 2026-11-06 18:04:51 +00:00
Author
Owner
Copy link

Mac-path readiness finding: the returned Mac VM has 23 GiB free, the J288 worktree target is absent, the cached candidate hash matches c9f48c47986179e14594b24c150b4cb5bcf90229e64f8ea3112f7a2835e477f4, and J267s measure.sh/summ.py are present. This VM has no J288 scripts checkout, so I staged the branchs measurement fixture files in /tmp under mac-bench.lock. The current controller and fixture hashes, disk receipt, and preflight checks are recorded in compute01: /home/claude/.cache/codex-scratch/j288/evidence/mac-controller-sync-verify.log. I have not configured the mesh: J279 still owns ~/mesh-vm.claim, and verify fast remains in the workspace test phase.

Mac-path readiness finding: the returned Mac VM has 23 GiB free, the J288 worktree target is absent, the cached candidate hash matches c9f48c47986179e14594b24c150b4cb5bcf90229e64f8ea3112f7a2835e477f4, and J267s measure.sh/summ.py are present. This VM has no J288 scripts checkout, so I staged the branchs measurement fixture files in /tmp under mac-bench.lock. The current controller and fixture hashes, disk receipt, and preflight checks are recorded in compute01: /home/claude/.cache/codex-scratch/j288/evidence/mac-controller-sync-verify.log. I have not configured the mesh: J279 still owns ~/mesh-vm.claim, and verify fast remains in the workspace test phase.
kayg commented 2026-11-06 18:13:12 +00:00
Author
Owner
Copy link

Update: J279 released ~/mesh-vm.claim during the foreground poll; J288 has not taken it or changed any live mesh state. The verify fast run remains active on compute01, so I will wait for it to finish before measuring. Next: preserve its release binary, run the required real-root TUN proof through build-slot, then claim the live mesh for the Mac-to-compute01 sample under the Mac lock.

Update: J279 released ~/mesh-vm.claim during the foreground poll; J288 has not taken it or changed any live mesh state. The verify fast run remains active on compute01, so I will wait for it to finish before measuring. Next: preserve its release binary, run the required real-root TUN proof through build-slot, then claim the live mesh for the Mac-to-compute01 sample under the Mac lock.
kayg commented 2026-11-06 18:21:51 +00:00
Author
Owner
Copy link

verify fast privileged integration finding: two existing route-restart namespace cases fail on this head. orphan_route_retirement_preserves_foreign_routes fails its route-presence vector assertion; restart_retires_orphan_peer_host_routes reports a removed peer route still present with metrics: [{"mtu":1200}]. Full evidence is in /home/claude/.cache/codex-scratch/j288/logs/verify-fast-a670442.log. I am tracing route install/retirement and the test fixtures before changing anything. The gate is still running through later rows; no live mesh state has been touched.

verify fast privileged integration finding: two existing route-restart namespace cases fail on this head. `orphan_route_retirement_preserves_foreign_routes` fails its route-presence vector assertion; `restart_retires_orphan_peer_host_routes` reports a removed peer route still present with `metrics: [{"mtu":1200}]`. Full evidence is in `/home/claude/.cache/codex-scratch/j288/logs/verify-fast-a670442.log`. I am tracing route install/retirement and the test fixtures before changing anything. The gate is still running through later rows; no live mesh state has been touched.
kayg commented 2026-11-06 18:25:44 +00:00
Author
Owner
Copy link

Root cause identified: Device::routes now installs an RTA_METRICS attribute containing one RTAX_MTU, but retire_routes treats every unrecognized route attribute as foreign. The orphan-route cleanup therefore skips the exact MTU-tagged routes this change creates; the failed restart test exposes the retained 1200-byte route. After the running gate exits, I will teach retirement to recognize only our exact nested metric shape (one valid mesh MTU, no extra metrics) while continuing to reject other route modifiers, then rerun the focused regression and gates.

Root cause identified: `Device::routes` now installs an `RTA_METRICS` attribute containing one `RTAX_MTU`, but `retire_routes` treats every unrecognized route attribute as foreign. The orphan-route cleanup therefore skips the exact MTU-tagged routes this change creates; the failed restart test exposes the retained 1200-byte route. After the running gate exits, I will teach retirement to recognize only our exact nested metric shape (one valid mesh MTU, no extra metrics) while continuing to reject other route modifiers, then rerun the focused regression and gates.
kayg commented 2026-11-06 18:36:26 +00:00
Author
Owner
Copy link

Mac work is resumed after the std freeze lifted at 14:36 IST. Branch mesh-mtu-381 is at a6704428c8241c9ae38cd02413af3366a41c985c; the cached Mac candidate is SHA-256 c9f48c47986179e14594b24c150b4cb5bcf90229e64f8ea3112f7a2835e477f4. Pending: real Mac ↔ compute01 plain-before / mesh-after / plain-after samples on J267's NetBird path. I will keep the compute01 mesh claim and Mac lockf -k /tmp/mac-bench.lock for the full run, set CARGO_INCREMENTAL=0, and confirm the J288 Mac target is absent before releasing the lock.

The original verify fast is still compiling on compute01 (PID 279492; log /home/claude/.cache/codex-scratch/j288/logs/verify-fast-a670442.log), so I have not started live mesh setup. D348 perf-VM measurement remains pending the J317/J88i matrix and lock handoff recorded on #260; I will not bypass that schedule.

Mac work is resumed after the std freeze lifted at 14:36 IST. Branch `mesh-mtu-381` is at `a6704428c8241c9ae38cd02413af3366a41c985c`; the cached Mac candidate is SHA-256 `c9f48c47986179e14594b24c150b4cb5bcf90229e64f8ea3112f7a2835e477f4`. Pending: real Mac ↔ compute01 plain-before / mesh-after / plain-after samples on J267's NetBird path. I will keep the compute01 mesh claim and Mac `lockf -k /tmp/mac-bench.lock` for the full run, set `CARGO_INCREMENTAL=0`, and confirm the J288 Mac target is absent before releasing the lock. The original `verify fast` is still compiling on compute01 (PID 279492; log `/home/claude/.cache/codex-scratch/j288/logs/verify-fast-a670442.log`), so I have not started live mesh setup. D348 perf-VM measurement remains pending the J317/J88i matrix and lock handoff recorded on #260; I will not bypass that schedule.
kayg commented 2026-11-06 18:48:11 +00:00
Author
Owner
Copy link

The first full verify fast on a6704428c8241c9ae38cd02413af3366a41c985c exited 1. As predicted, privileged route cleanup failed both kernel namespace cases: orphan_route_retirement_preserves_foreign_routes left both MTU-tagged peer routes (left: [1, 1, 1, 1, 1, 1, 1], expected [0, 0, 1, 1, 1, 1, 1]), and restart_retires_orphan_peer_host_routes found 10.254.0.2 still present with metrics: [{"mtu":1200}]. Full evidence: /home/claude/.cache/codex-scratch/j288/logs/verify-fast-a670442.log.

Verbatim final gate summary:

+---------------------------+--------------+
| Rust gate | Result |
+---------------------------+--------------+
| Nix Cargo vendoring | PASS |
| fmt | PASS |
| migration names | PASS |
| migration tooling tests | PASS |
| VT native fmt | PASS |
| CLI plugin compile-out | PASS |
| clippy | PASS |
| SSH runtime test helper | PASS |
| real-binary fixture paths | PASS |
| Traefik component manifest | PASS |
| release component publication | PASS |
| test | PASS |
| SSH inode reuse fixture | PASS |
| shim process-per-user | PASS |
| cargo-deny | PASS |
| gate architecture | PASS |
| doc coverage ratchet | PASS |
| shape lints off (D351) | PASS |
| doc comments on diff | PASS |
| fixed test waits | PASS |
| privileged integration | FAIL (1) |
| probe release exclusion | PASS |
| nightly fuzz reporter | PASS |
| static builds | PASS |
+---------------------------+--------------+

VERIFY_FAST_EXIT=1

The root cause is the route metric shape already recorded in comment 82709. I’ll add a focused metrics-shape regression, fix retirement, and rerun the gates. ~/mesh-vm.claim is currently held by J287 (2026-11-06T18:42:51Z); I will wait for its release before live measurement.

The first full `verify fast` on `a6704428c8241c9ae38cd02413af3366a41c985c` exited 1. As predicted, privileged route cleanup failed both kernel namespace cases: `orphan_route_retirement_preserves_foreign_routes` left both MTU-tagged peer routes (`left: [1, 1, 1, 1, 1, 1, 1]`, expected `[0, 0, 1, 1, 1, 1, 1]`), and `restart_retires_orphan_peer_host_routes` found `10.254.0.2` still present with `metrics: [{"mtu":1200}]`. Full evidence: `/home/claude/.cache/codex-scratch/j288/logs/verify-fast-a670442.log`. Verbatim final gate summary: +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | Nix Cargo vendoring | PASS | | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | FAIL (1) | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | static builds | PASS | +---------------------------+--------------+ VERIFY_FAST_EXIT=1 The root cause is the route metric shape already recorded in comment 82709. I’ll add a focused metrics-shape regression, fix retirement, and rerun the gates. `~/mesh-vm.claim` is currently held by J287 (`2026-11-06T18:42:51Z`); I will wait for its release before live measurement.
kayg commented 2026-11-06 18:52:18 +00:00
Author
Owner
Copy link

Route-retirement fix and its metric-shape regression are in the worktree. verify quick is queued through compute01 build-slot at position 15 (launcher PID 1509922); log: /home/claude/.cache/codex-scratch/j288/logs/verify-quick-route-cleanup.log. I’ll commit/push after it passes. The live mesh claim remains held by J287, so no mesh setup has started.

Route-retirement fix and its metric-shape regression are in the worktree. `verify quick` is queued through compute01 build-slot at position 15 (launcher PID 1509922); log: `/home/claude/.cache/codex-scratch/j288/logs/verify-quick-route-cleanup.log`. I’ll commit/push after it passes. The live mesh claim remains held by J287, so no mesh setup has started.
kayg commented 2026-11-06 19:21:54 +00:00
Author
Owner
Copy link

J288 verify quick has waited in the compute01 build-slot queue for 30 minutes. It remains queued at PID 1509924, currently second behind PID 1508793; no test has started. Log: /home/claude/.cache/codex-scratch/j288/logs/verify-quick-route-cleanup.log. The target stays /home/claude/worktrees/j288/target, shared with the prior gate. I’m keeping the slot request and polling; the next step after the pre-check passes is commit/push and the final fast gate.

J288 `verify quick` has waited in the compute01 build-slot queue for 30 minutes. It remains queued at PID 1509924, currently second behind PID 1508793; no test has started. Log: `/home/claude/.cache/codex-scratch/j288/logs/verify-quick-route-cleanup.log`. The target stays `/home/claude/worktrees/j288/target`, shared with the prior gate. I’m keeping the slot request and polling; the next step after the pre-check passes is commit/push and the final fast gate.
kayg commented 2026-11-06 19:31:11 +00:00
Author
Owner
Copy link

The route-retirement fix passed verify quick on the current worktree (a6704428c8241c9ae38cd02413af3366a41c985c): changed mesh package tests were 55 passed, 0 failed, and all pre-check rows passed. Evidence: /home/claude/.cache/codex-scratch/j288/logs/verify-quick-route-cleanup.log.

Verbatim quick summary:

+---------------------------+--------------+
| Rust gate | Result |
+---------------------------+--------------+
| scope guard | PASS |
| migration names | PASS |
| Nix Cargo vendoring | PASS |
| fmt | PASS |
| doc coverage ratchet | PASS |
| shape lints off (D351) | PASS |
| doc comments on diff | PASS |
| fixed test waits | PASS |
| clippy (workspace) | PASS |
| unit tests (changed) | PASS |
+---------------------------+--------------+

QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done.

I’m committing and pushing the focused route cleanup slice now, then I will run the full gate.

The route-retirement fix passed `verify quick` on the current worktree (`a6704428c8241c9ae38cd02413af3366a41c985c`): changed mesh package tests were 55 passed, 0 failed, and all pre-check rows passed. Evidence: `/home/claude/.cache/codex-scratch/j288/logs/verify-quick-route-cleanup.log`. Verbatim quick summary: +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | scope guard | PASS | | migration names | PASS | | Nix Cargo vendoring | PASS | | fmt | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | clippy (workspace) | PASS | | unit tests (changed) | PASS | +---------------------------+--------------+ QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done. I’m committing and pushing the focused route cleanup slice now, then I will run the full gate.
kayg commented 2026-11-06 19:32:54 +00:00
Author
Owner
Copy link

Focused route-retirement fix committed and pushed as 28486bbe4bf92ee64e5dfe24b06d633d1339ef65 on mesh-mtu-381. The change accepts only one well-formed nested MTU metric in the supported range, continues to accept legacy routes without metrics, and preserves routes with any additional or foreign modifier. verify quick passed before commit with 55 changed-package tests; receipt: /home/claude/.cache/codex-scratch/j288/logs/verify-quick-route-cleanup.log. I’m now queuing verify fast on compute01.

Focused route-retirement fix committed and pushed as `28486bbe4bf92ee64e5dfe24b06d633d1339ef65` on `mesh-mtu-381`. The change accepts only one well-formed nested MTU metric in the supported range, continues to accept legacy routes without metrics, and preserves routes with any additional or foreign modifier. `verify quick` passed before commit with 55 changed-package tests; receipt: `/home/claude/.cache/codex-scratch/j288/logs/verify-quick-route-cleanup.log`. I’m now queuing `verify fast` on compute01.
kayg commented 2026-11-06 20:09:37 +00:00
Author
Owner
Copy link

verify fast for 28486bbe4bf92ee64e5dfe24b06d633d1339ef65 has now run for about 35 minutes including queue wait. The privileged integration harness passed its 58 checks and detected areas=install,mesh; the full workspace compile is still running. Current log: /home/claude/.cache/codex-scratch/j288/logs/verify-fast-28486bbe.log. No final gate result yet.

`verify fast` for `28486bbe4bf92ee64e5dfe24b06d633d1339ef65` has now run for about 35 minutes including queue wait. The privileged integration harness passed its 58 checks and detected `areas=install,mesh`; the full workspace compile is still running. Current log: `/home/claude/.cache/codex-scratch/j288/logs/verify-fast-28486bbe.log`. No final gate result yet.
kayg commented 2026-11-06 20:23:45 +00:00
Author
Owner
Copy link

J288 finding: verify fast exited 1 on 28486bbe4bf92ee64e5dfe24b06d633d1339ef65. The exact patch issues are a rustfmt hunk in the nested route-metrics test helper, an unannotated integer array causing E0689 at wireguard.rs:1472 (which also stops clippy and tests), and the missing D345 doc comment on retire_routes at wireguard.rs:749. Static builds passed: x86_64 27,806,800 bytes (+5,388,544); aarch64 25,015,640 bytes (+4,562,360). The namespace proof harness passed its 58 checks and selected install,mesh; the later privileged integration test row failed as a consequence of the compile error. Full log: /home/claude/.cache/codex-scratch/j288/logs/verify-fast-28486bbe.log on compute01.

I will fix only the test type, formatting and D345 comment, rerun verify quick, push the correction, then rerun verify fast on compute01. The Mac build/runtime step and live Mac ↔ compute01 measurement remain next; the D348 normal-MTU perf measurement is still blocked by the invalid calibration and missing lock handoff recorded on #260, so I will not start or alter perf-VM state.

J288 finding: `verify fast` exited 1 on `28486bbe4bf92ee64e5dfe24b06d633d1339ef65`. The exact patch issues are a rustfmt hunk in the nested route-metrics test helper, an unannotated integer array causing E0689 at `wireguard.rs:1472` (which also stops clippy and tests), and the missing D345 doc comment on `retire_routes` at `wireguard.rs:749`. Static builds passed: x86_64 27,806,800 bytes (+5,388,544); aarch64 25,015,640 bytes (+4,562,360). The namespace proof harness passed its 58 checks and selected `install,mesh`; the later privileged integration test row failed as a consequence of the compile error. Full log: `/home/claude/.cache/codex-scratch/j288/logs/verify-fast-28486bbe.log` on compute01. I will fix only the test type, formatting and D345 comment, rerun `verify quick`, push the correction, then rerun `verify fast` on compute01. The Mac build/runtime step and live Mac ↔ compute01 measurement remain next; the D348 normal-MTU perf measurement is still blocked by the invalid calibration and missing lock handoff recorded on #260, so I will not start or alter perf-VM state.
kayg commented 2026-11-06 20:45:35 +00:00
Author
Owner
Copy link

J288 progress after the Mac VM returned:

  • I checked the old Mac candidate provenance before using it. The binary hash was stable, but its head receipt said a670442 while its source/test receipts said 2171a4b, so I am not using that candidate as final evidence.
  • A fresh Mac build and Mac mesh library test are now running from exact branch head 28486bbe4bf92ee64e5dfe24b06d633d1339ef65, under lockf -k /tmp/mac-bench.lock with CARGO_INCREMENTAL=0. The job target is ~/worktrees/j288-mac-final-28486bbe/target; the locked script removes it and the temporary worktree before releasing the lock. Log: /Users/claude/.cache/codex-scratch/j288/evidence/mac-build-test-28486bbe.log on the Mac.
  • The corrected verify quick was queued on compute01 at position 21 before admission froze. It has not started. I will not resubmit or start compute01 work until the maintenance restart is complete and the owner posts that compute01 is back; the queued run being cut is not a test failure.
  • The D348 perf-VM run remains pending: #260 still lacks the validated calibration and lock handoff, so I have not started a counted run or changed CPU masks.
J288 progress after the Mac VM returned: - I checked the old Mac candidate provenance before using it. The binary hash was stable, but its head receipt said `a670442` while its source/test receipts said `2171a4b`, so I am not using that candidate as final evidence. - A fresh Mac build and Mac mesh library test are now running from exact branch head `28486bbe4bf92ee64e5dfe24b06d633d1339ef65`, under `lockf -k /tmp/mac-bench.lock` with `CARGO_INCREMENTAL=0`. The job target is `~/worktrees/j288-mac-final-28486bbe/target`; the locked script removes it and the temporary worktree before releasing the lock. Log: `/Users/claude/.cache/codex-scratch/j288/evidence/mac-build-test-28486bbe.log` on the Mac. - The corrected `verify quick` was queued on compute01 at position 21 before admission froze. It has not started. I will not resubmit or start compute01 work until the maintenance restart is complete and the owner posts that compute01 is back; the queued run being cut is not a test failure. - The D348 perf-VM run remains pending: #260 still lacks the validated calibration and lock handoff, so I have not started a counted run or changed CPU masks.
kayg commented 2026-11-06 20:55:10 +00:00
Author
Owner
Copy link

Mac result from a fresh run against exact branch head 28486bbe4bf92ee64e5dfe24b06d633d1339ef65:

  • cargo test --locked -j3 --release --no-default-features --features agent -p infraternal-plugin-mesh --lib passed: 69 passed, 0 failed. Log: /Users/claude/.cache/codex-scratch/j288/evidence/mac-build-test-28486bbe.log.
  • The subsequent full cargo build --locked -j3 --release -p infraternal stops in unrelated existing macOS code at crates/infraternal-agent/src/logins.rs:424: SockFlag::SOCK_CLOEXEC is unavailable on macOS. This is the open #379 prerequisite; I did not bundle its fix into #381. The cleanup removed the target and temporary worktree before releasing the lock (TARGET_PRESENT=no, COMMAND_EXIT=101 in the same log); the Mac is back to 23 GB free.
  • The earlier cached Mac candidate was built from source head 2171a4b, not a670442 as its separate head receipt claimed. I compared its source snapshot: macos_node.rs is byte-identical to current #381; mesh_mtu.rs differs only in the visibility/doc line for MIN_MESH_MTU. Candidate SHA-256 remains c9f48c47986179e14594b24c150b4cb5bcf90229e64f8ea3112f7a2835e477f4. I will record its exact source provenance if used for the real path run.

The corrected verify quick is still queued at compute01 position 21 and has not started; admission is frozen for maintenance. The D348 perf run remains pending the valid calibration and lock handoff in #260.

Mac result from a fresh run against exact branch head `28486bbe4bf92ee64e5dfe24b06d633d1339ef65`: - `cargo test --locked -j3 --release --no-default-features --features agent -p infraternal-plugin-mesh --lib` passed: 69 passed, 0 failed. Log: `/Users/claude/.cache/codex-scratch/j288/evidence/mac-build-test-28486bbe.log`. - The subsequent full `cargo build --locked -j3 --release -p infraternal` stops in unrelated existing macOS code at `crates/infraternal-agent/src/logins.rs:424`: `SockFlag::SOCK_CLOEXEC` is unavailable on macOS. This is the open #379 prerequisite; I did not bundle its fix into #381. The cleanup removed the target and temporary worktree before releasing the lock (`TARGET_PRESENT=no`, `COMMAND_EXIT=101` in the same log); the Mac is back to 23 GB free. - The earlier cached Mac candidate was built from source head `2171a4b`, not `a670442` as its separate head receipt claimed. I compared its source snapshot: `macos_node.rs` is byte-identical to current #381; `mesh_mtu.rs` differs only in the visibility/doc line for `MIN_MESH_MTU`. Candidate SHA-256 remains `c9f48c47986179e14594b24c150b4cb5bcf90229e64f8ea3112f7a2835e477f4`. I will record its exact source provenance if used for the real path run. The corrected `verify quick` is still queued at compute01 position 21 and has not started; admission is frozen for maintenance. The D348 perf run remains pending the valid calibration and lock handoff in #260.
kayg commented 2026-11-06 21:03:51 +00:00
Author
Owner
Copy link

J288 maintenance/rebase update:

  • While compute01 admission is frozen, I fetched the control-host repository and rebased mesh-mtu-381 onto the newer origin/dev fba745051416c60a8523f0fe529c2aee6bbe0dd0. Current rebased head is a9a81cae1 (route-retirement fix); the small compiler/doc/format correction is still unstaged. The remote job branch remains at its pre-rebase head until the correction passes quick and is committed.
  • The quick run queued before maintenance remains at position 21 and has not started. I will resync the rebased source and submit quick once, in the foreground, after the #402 back notice. The queued attempt will be treated as cut by maintenance, not as a gate failure.
  • Mac source files in the current base are unchanged from the earlier exact-head Mac test. The current Mac CLI build still fails at the open #379 SockFlag::SOCK_CLOEXEC issue; no #379 changes are included here.
  • D348 remains blocked by the missing passing calibration and lock handoff on #260; no perf-VM run has started.
J288 maintenance/rebase update: - While compute01 admission is frozen, I fetched the control-host repository and rebased `mesh-mtu-381` onto the newer `origin/dev` `fba745051416c60a8523f0fe529c2aee6bbe0dd0`. Current rebased head is `a9a81cae1` (route-retirement fix); the small compiler/doc/format correction is still unstaged. The remote job branch remains at its pre-rebase head until the correction passes quick and is committed. - The quick run queued before maintenance remains at position 21 and has not started. I will resync the rebased source and submit quick once, in the foreground, after the #402 back notice. The queued attempt will be treated as cut by maintenance, not as a gate failure. - Mac source files in the current base are unchanged from the earlier exact-head Mac test. The current Mac CLI build still fails at the open #379 `SockFlag::SOCK_CLOEXEC` issue; no #379 changes are included here. - D348 remains blocked by the missing passing calibration and lock handoff on #260; no perf-VM run has started.
kayg commented 2026-11-06 21:05:36 +00:00
Author
Owner
Copy link

Correction to the rebase status: the branch rewrite is now pushed to origin/mesh-mtu-381 with --force-with-lease, head a9a81cae17fb7050cf6b451a1fa6e59ceb5ca2fd, based on origin/dev fba745051416c60a8523f0fe529c2aee6bbe0dd0. The two existing #381 commits were rebased; no new production behavior was added. The small fast-gate correction remains uncommitted pending the fresh quick run.

Correction to the rebase status: the branch rewrite is now pushed to `origin/mesh-mtu-381` with `--force-with-lease`, head `a9a81cae17fb7050cf6b451a1fa6e59ceb5ca2fd`, based on `origin/dev` `fba745051416c60a8523f0fe529c2aee6bbe0dd0`. The two existing #381 commits were rebased; no new production behavior was added. The small fast-gate correction remains uncommitted pending the fresh quick run.
kayg commented 2026-11-06 21:17:00 +00:00
Author
Owner
Copy link

The earlier scripts/verify quick was queued on compute01 at position 21 and never started. The maintenance restart cut the SSH/build-slot waiter; this was not a test failure. Following the 02:45 IST notice on #402, I am resubmitting it once in the foreground.

The earlier `scripts/verify quick` was queued on compute01 at position 21 and never started. The maintenance restart cut the SSH/build-slot waiter; this was not a test failure. Following the 02:45 IST notice on #402, I am resubmitting it once in the foreground.
kayg commented 2026-11-06 21:57:50 +00:00
Author
Owner
Copy link

Post-restart quick resubmission on compute01 completed successfully: all Rust quick rows passed and the changed mesh plugin tests reported 55 passed, 0 failed. The tested tree is now committed and pushed as 779950c0f5. Full receipt: /home/kayg/.cache/codex-scratch/j288/evidence/verify-quick-final.log. The same compute01 target is retained for verify fast.

Post-restart quick resubmission on compute01 completed successfully: all Rust quick rows passed and the changed mesh plugin tests reported 55 passed, 0 failed. The tested tree is now committed and pushed as 779950c0f5. Full receipt: /home/kayg/.cache/codex-scratch/j288/evidence/verify-quick-final.log. The same compute01 target is retained for verify fast.
kayg commented 2026-11-06 22:11:54 +00:00
Author
Owner
Copy link

Blocker update: the post-commit scripts/verify fast run was submitted once in the foreground on compute01 at 22:58 IST and is waiting at build-slot queue position 20. All five slots were busy and earlier jobs were ahead; no fast checks have started, so this is not a failure. It will reuse /home/claude/.cache/codex-scratch/j288/target, the same target used by quick.

Blocker update: the post-commit `scripts/verify fast` run was submitted once in the foreground on compute01 at 22:58 IST and is waiting at build-slot queue position 20. All five slots were busy and earlier jobs were ahead; no fast checks have started, so this is not a failure. It will reuse `/home/claude/.cache/codex-scratch/j288/target`, the same target used by quick.
kayg commented 2026-11-06 23:59:11 +00:00
Author
Owner
Copy link

Gate finding: scripts/verify fast failed the session_retention test bounded_owned_exit_reconnects_as_ended_with_status. Its __shim child exited before the expected socket path appeared, so the fixture assertion at scripts/verify-session-retention.py:50 failed. The test and fixture are unchanged versus origin/dev; all five compute01 slots were occupied during this run. Fast is still running later rows, so no final gate result yet. Full log: /home/kayg/.cache/codex-scratch/j288/evidence/verify-fast-final.log. I am investigating before any retry or code change.

Gate finding: `scripts/verify fast` failed the `session_retention` test `bounded_owned_exit_reconnects_as_ended_with_status`. Its `__shim` child exited before the expected socket path appeared, so the fixture assertion at `scripts/verify-session-retention.py:50` failed. The test and fixture are unchanged versus origin/dev; all five compute01 slots were occupied during this run. Fast is still running later rows, so no final gate result yet. Full log: /home/kayg/.cache/codex-scratch/j288/evidence/verify-fast-final.log. I am investigating before any retry or code change.
kayg commented 2026-11-07 00:08:27 +00:00
Author
Owner
Copy link

In-scope fast-gate finding: privileged integration test orphan_route_retirement_preserves_foreign_routes failed at test 65/101. It expected the two stale MTU-tagged owned routes to be removed while five other routes remained ([0,0,1,1,1,1,1]), but observed [1,1,1,1,1,1,1]. The full fast run continues. I am tracing the fixture and route deletion path before changing code. Log: /home/kayg/.cache/codex-scratch/j288/evidence/verify-fast-final.log.

In-scope fast-gate finding: privileged integration test `orphan_route_retirement_preserves_foreign_routes` failed at test 65/101. It expected the two stale MTU-tagged owned routes to be removed while five other routes remained (`[0,0,1,1,1,1,1]`), but observed `[1,1,1,1,1,1,1]`. The full fast run continues. I am tracing the fixture and route deletion path before changing code. Log: /home/kayg/.cache/codex-scratch/j288/evidence/verify-fast-final.log.
kayg commented 2026-11-07 00:25:19 +00:00
Author
Owner
Copy link

J288 Mac verification update. At pushed head 779950c0f5, the Mac arm64 run cargo test --locked -p infraternal-plugin-mesh passed: 131 passed, 0 failed, 1 ignored. It ran under lockf -k /tmp/mac-bench.lock with CARGO_INCREMENTAL=0; its target was removed before releasing the lock, leaving 25 GiB free. Log: claude@infraternal-macos-01-std:~/.cache/codex-scratch/j288/evidence/mac-mesh-tests-779950c0.log. The existing foreground verify fast on compute01 is still running; I have started no new compute01 work during maintenance. The real Mac↔compute01 after-measurement and D348 perf run remain pending.

J288 Mac verification update. At pushed head 779950c0f5da7771388804bb697a7a51955566d8, the Mac arm64 run `cargo test --locked -p infraternal-plugin-mesh` passed: 131 passed, 0 failed, 1 ignored. It ran under `lockf -k /tmp/mac-bench.lock` with `CARGO_INCREMENTAL=0`; its target was removed before releasing the lock, leaving 25 GiB free. Log: `claude@infraternal-macos-01-std:~/.cache/codex-scratch/j288/evidence/mac-mesh-tests-779950c0.log`. The existing foreground `verify fast` on compute01 is still running; I have started no new compute01 work during maintenance. The real Mac↔compute01 after-measurement and D348 perf run remain pending.
kayg commented 2026-11-07 00:28:39 +00:00
Author
Owner
Copy link

J288 fast-gate route finding. Both privileged route-retirement tests leave the orphan host routes present. A raw rtnetlink dump from an isolated Linux namespace confirms the kernel returns RTA_METRICS as type 0x0008 with nested=false, while preserving the nested RTAX_MTU=1200 payload. mesh_route_metrics_are_owned currently requires the outer nested flag, so it rejects the route it installed. Evidence: /home/kayg/.cache/codex-scratch/j288/evidence/route-metrics-flag-normalization.log (dev-02 kernel, no mesh interface or host route changes). I added a focused unit regression test for this normalized dump; dev-02 build-slot is currently occupied, so I have not run it or changed the matcher yet.

J288 fast-gate route finding. Both privileged route-retirement tests leave the orphan host routes present. A raw rtnetlink dump from an isolated Linux namespace confirms the kernel returns `RTA_METRICS` as type `0x0008` with `nested=false`, while preserving the nested `RTAX_MTU=1200` payload. `mesh_route_metrics_are_owned` currently requires the outer nested flag, so it rejects the route it installed. Evidence: `/home/kayg/.cache/codex-scratch/j288/evidence/route-metrics-flag-normalization.log` (dev-02 kernel, no mesh interface or host route changes). I added a focused unit regression test for this normalized dump; dev-02 build-slot is currently occupied, so I have not run it or changed the matcher yet.
kayg commented 2026-11-07 00:39:57 +00:00
Author
Owner
Copy link

J288 route-matcher regression test RED. With the agent feature enabled, wireguard::route_metric_tests::route_retirement_accepts_kernel_normalized_metrics fails at assert!(owns_encoded_metrics(&encoded)) (0 passed, 1 failed, exit 101), matching the raw kernel dump. Evidence: claude@infraternal-dev-02:~/.cache/codex-scratch/j288/evidence/route-metric-normalization-red-exact.log. The preceding verify quick passed 55 default-feature tests but does not compile this agent-gated Linux module; it was a pre-check, not evidence for this regression.

J288 route-matcher regression test RED. With the `agent` feature enabled, `wireguard::route_metric_tests::route_retirement_accepts_kernel_normalized_metrics` fails at `assert!(owns_encoded_metrics(&encoded))` (0 passed, 1 failed, exit 101), matching the raw kernel dump. Evidence: `claude@infraternal-dev-02:~/.cache/codex-scratch/j288/evidence/route-metric-normalization-red-exact.log`. The preceding `verify quick` passed 55 default-feature tests but does not compile this agent-gated Linux module; it was a pre-check, not evidence for this regression.
kayg commented 2026-11-07 00:50:49 +00:00
Author
Owner
Copy link

J288 namespace rerun on dev-02 after the matcher change: both affected tests now pass in real disposable network namespaces: orphan_route_retirement_preserves_foreign_routes ([65/101]) and restart_retires_orphan_peer_host_routes ([77/101]). The overall scripts/verify-privileged run exited 1 because 18 other nftables-related cases returned No such file or directory inside the sandbox; this is an intermediate aarch64 run, not the final x86 gate. Full log: claude@infraternal-dev-02:~/.cache/codex-scratch/j288/evidence/verify-privileged-route-green.log. I am preserving those failures for diagnosis rather than treating the suite as green.

J288 namespace rerun on dev-02 after the matcher change: both affected tests now pass in real disposable network namespaces: `orphan_route_retirement_preserves_foreign_routes` ([65/101]) and `restart_retires_orphan_peer_host_routes` ([77/101]). The overall `scripts/verify-privileged` run exited 1 because 18 other nftables-related cases returned `No such file or directory` inside the sandbox; this is an intermediate aarch64 run, not the final x86 gate. Full log: `claude@infraternal-dev-02:~/.cache/codex-scratch/j288/evidence/verify-privileged-route-green.log`. I am preserving those failures for diagnosis rather than treating the suite as green.
kayg commented 2026-11-07 00:52:43 +00:00
Author
Owner
Copy link

J288 pushed route cleanup fix 5ae1d1f0b2e619cd400724f6ec5570e165ff7b70 on mesh-mtu-381. The exact normalized-metrics unit test is red before the matcher change and green after it (route-metric-normalization-red-exact.log, route-metric-normalization-green.log); the updated dev-02 verify quick passes (55 changed-feature tests, 0 failed; verify-quick-route-green.log). In the disposable namespace sweep, both orphan-route integration tests pass; the wider sweep still exits 1 on 18 nftables-related missing-file errors (verify-privileged-route-green.log). The pre-maintenance compute01 fast run is still the existing run on the previous head. I will not submit compute01 work until the owner posts the restart confirmation; final fast and real-path measurement remain pending.

J288 pushed route cleanup fix `5ae1d1f0b2e619cd400724f6ec5570e165ff7b70` on `mesh-mtu-381`. The exact normalized-metrics unit test is red before the matcher change and green after it (`route-metric-normalization-red-exact.log`, `route-metric-normalization-green.log`); the updated dev-02 `verify quick` passes (55 changed-feature tests, 0 failed; `verify-quick-route-green.log`). In the disposable namespace sweep, both orphan-route integration tests pass; the wider sweep still exits 1 on 18 nftables-related missing-file errors (`verify-privileged-route-green.log`). The pre-maintenance compute01 fast run is still the existing run on the previous head. I will not submit compute01 work until the owner posts the restart confirmation; final fast and real-path measurement remain pending.
kayg commented 2026-11-07 01:07:08 +00:00
Author
Owner
Copy link

Update after the #402 compute01 restart: the pre-maintenance scripts/verify fast run did not complete before the VM stopped. The SSH/build-slot run was cut by the restart, so I am excluding it as a test result rather than treating it as a code failure.

The #402 02:45 IST notice confirms compute01 is back with five slots. I am submitting the final verify fast once, in the foreground, on mesh-mtu-381 at 5ae1d1f0b2e619cd400724f6ec5570e165ff7b70. The same head's verify quick passed on dev-02; evidence: /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-route-green.log.

The Mac mesh library tests passed 131, 0 failed, 1 ignored under the Mac lock with CARGO_INCREMENTAL=0; evidence: /Users/claude/.cache/codex-scratch/j288/evidence/mac-mesh-tests-779950c0.log. Next I will run the real Mac↔compute01 measurement under ~/mesh-vm.claim and the Mac lock. D348 perf-VM measurement remains blocked by the missing passing calibration and lock handoff recorded on #260; I will not start a counted run without them.

Update after the #402 compute01 restart: the pre-maintenance `scripts/verify fast` run did not complete before the VM stopped. The SSH/build-slot run was cut by the restart, so I am excluding it as a test result rather than treating it as a code failure. The #402 02:45 IST notice confirms compute01 is back with five slots. I am submitting the final `verify fast` once, in the foreground, on `mesh-mtu-381` at `5ae1d1f0b2e619cd400724f6ec5570e165ff7b70`. The same head's `verify quick` passed on dev-02; evidence: `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-route-green.log`. The Mac mesh library tests passed 131, 0 failed, 1 ignored under the Mac lock with `CARGO_INCREMENTAL=0`; evidence: `/Users/claude/.cache/codex-scratch/j288/evidence/mac-mesh-tests-779950c0.log`. Next I will run the real Mac↔compute01 measurement under `~/mesh-vm.claim` and the Mac lock. D348 perf-VM measurement remains blocked by the missing passing calibration and lock handoff recorded on #260; I will not start a counted run without them.
kayg commented 2026-11-07 01:22:21 +00:00
Author
Owner
Copy link

Gate queue update: the single post-restart scripts/verify fast for 5ae1d1f0b2e619cd400724f6ec5570e165ff7b70 is waiting at compute01 build-slot position 21. No checks have started and I have not submitted a duplicate. The exact Nix/Cargo preflight passed (cargo 1.99.0).

Before queueing, I confirmed no active process had a cwd or open file under either J288 target. I kept /home/claude/.cache/codex-scratch/j288/target (44 GiB), removed the inactive J288 worktree target (62 GiB), and restored the required ~/worktrees/j288/target symlink. Free disk is 92 GiB. The foreground SSH session is still attached; full output is being saved to /home/claude/.cache/codex-scratch/j288/evidence/verify-fast-final-5ae1d1f.log.

Gate queue update: the single post-restart `scripts/verify fast` for `5ae1d1f0b2e619cd400724f6ec5570e165ff7b70` is waiting at compute01 build-slot position 21. No checks have started and I have not submitted a duplicate. The exact Nix/Cargo preflight passed (`cargo 1.99.0`). Before queueing, I confirmed no active process had a cwd or open file under either J288 target. I kept `/home/claude/.cache/codex-scratch/j288/target` (44 GiB), removed the inactive J288 worktree target (62 GiB), and restored the required `~/worktrees/j288/target` symlink. Free disk is 92 GiB. The foreground SSH session is still attached; full output is being saved to `/home/claude/.cache/codex-scratch/j288/evidence/verify-fast-final-5ae1d1f.log`.
kayg commented 2026-11-07 02:04:05 +00:00
Author
Owner
Copy link

Resource checkpoint: compute01 still has five active slots and my same foreground fast-gate request has not started. Free disk is now 32 GiB (build-slot floor 25 GiB), down from 92 GiB at submission as other jobs use the VM. The J288 cache is already consolidated to one target; I will not delete or borrow another job's target.

Resource checkpoint: compute01 still has five active slots and my same foreground fast-gate request has not started. Free disk is now 32 GiB (build-slot floor 25 GiB), down from 92 GiB at submission as other jobs use the VM. The J288 cache is already consolidated to one target; I will not delete or borrow another job's target.
kayg commented 2026-11-07 02:07:14 +00:00
Author
Owner
Copy link

Gate admission is now blocked by compute01 disk capacity. build-slot --status shows one free slot but only 8.6 GiB free, below its 25 GiB floor, so the same foreground request remains queued and no checks have started. J288 has only the retained 44 GiB scratch target (the inactive duplicate was removed before submission); I will not delete that cache or another job's target while the final gate still needs it. The build-slot waiter will recheck admission as space recovers.

Gate admission is now blocked by compute01 disk capacity. `build-slot --status` shows one free slot but only 8.6 GiB free, below its 25 GiB floor, so the same foreground request remains queued and no checks have started. J288 has only the retained 44 GiB scratch target (the inactive duplicate was removed before submission); I will not delete that cache or another job's target while the final gate still needs it. The build-slot waiter will recheck admission as space recovers.
kayg commented 2026-11-07 02:26:04 +00:00
Author
Owner
Copy link

Disk-floor recheck: compute01 is back below the build-slot admission floor at 20 GiB free. Five slots are occupied; the same verify fast request remains queued and has not started. I am preserving the single 44 GiB cache and waiting for capacity to recover; I will not bypass the gate or remove another job's target.

Disk-floor recheck: compute01 is back below the build-slot admission floor at 20 GiB free. Five slots are occupied; the same `verify fast` request remains queued and has not started. I am preserving the single 44 GiB cache and waiting for capacity to recover; I will not bypass the gate or remove another job's target.
kayg commented 2026-11-07 02:28:56 +00:00
Author
Owner
Copy link

Latest admission check: compute01 now has 8.5 GiB free, still below the 25 GiB floor; all five slots are busy. The one final fast-gate waiter remains attached and has not started. No J288 target or other job's target has been deleted since the prior update.

Latest admission check: compute01 now has 8.5 GiB free, still below the 25 GiB floor; all five slots are busy. The one final fast-gate waiter remains attached and has not started. No J288 target or other job's target has been deleted since the prior update.
kayg commented 2026-11-07 02:55:26 +00:00
Author
Owner
Copy link

Gate admission update: the single post-restart scripts/verify fast run is now admitted on compute01 and compiling. No result is claimed yet. Foreground session remains attached; full output is being written to /home/claude/.cache/codex-scratch/j288/evidence/verify-fast-final-5ae1d1f.log.

Gate admission update: the single post-restart `scripts/verify fast` run is now admitted on compute01 and compiling. No result is claimed yet. Foreground session remains attached; full output is being written to `/home/claude/.cache/codex-scratch/j288/evidence/verify-fast-final-5ae1d1f.log`.
kayg commented 2026-11-07 03:27:20 +00:00
Author
Owner
Copy link

Post-restart verify fast on compute01 exited 1 (log: /home/claude/.cache/codex-scratch/j288/evidence/verify-fast-final-5ae1d1f.log). It found one contradictory test assertion: route_retirement_accepts_only_our_exact_mtu_metric rejects the normalized RTA_METRICS shape accepted by the adjacent kernel-normalization test and matcher (wireguard.rs:1508). The static-build helper also failed with OSError: [Errno 28] No space left on device; build-slot status now reports 15 GB free against its 25 GB floor. I am correcting only the stale test case, then will run the focused check and quick/fast gates on dev-02, which currently has one free slot and 58 GB free. The interrupted pre-restart run remains recorded separately.

Post-restart `verify fast` on compute01 exited 1 (log: `/home/claude/.cache/codex-scratch/j288/evidence/verify-fast-final-5ae1d1f.log`). It found one contradictory test assertion: `route_retirement_accepts_only_our_exact_mtu_metric` rejects the normalized `RTA_METRICS` shape accepted by the adjacent kernel-normalization test and matcher (`wireguard.rs:1508`). The static-build helper also failed with `OSError: [Errno 28] No space left on device`; build-slot status now reports 15 GB free against its 25 GB floor. I am correcting only the stale test case, then will run the focused check and quick/fast gates on dev-02, which currently has one free slot and 58 GB free. The interrupted pre-restart run remains recorded separately.
kayg commented 2026-11-07 03:37:55 +00:00
Author
Owner
Copy link

The stale foreign-case assertion is removed in 36e7a6308f and pushed to mesh-mtu-381. Focused Linux route metric test: 1 passed, 162 filtered out (evidence: /home/claude/.cache/codex-scratch/j288/evidence/route-metric-fix-focused.log). scripts/verify quick: PASS; 55 changed-package tests passed (evidence: /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-postfix.log). Running the final verify fast next.

The stale foreign-case assertion is removed in 36e7a6308f5561181b56cf44b67f479a33d6d0f6 and pushed to mesh-mtu-381. Focused Linux route metric test: 1 passed, 162 filtered out (evidence: `/home/claude/.cache/codex-scratch/j288/evidence/route-metric-fix-focused.log`). `scripts/verify quick`: PASS; 55 changed-package tests passed (evidence: `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-postfix.log`). Running the final `verify fast` next.
kayg commented 2026-11-07 03:56:28 +00:00
Author
Owner
Copy link

The final verify fast on dev-02 completed at 36e7a6308f but is not green (full log: /home/claude/.cache/codex-scratch/j288/evidence/verify-fast-final-36e7a63.log). static builds passed, including cross-compiled x86_64 (27,803,408 bytes; +5,385,152) and native aarch64 (25,006,408 bytes; +4,553,128). The workspace test row failed on an unrelated SSH streaming bounded-wait fixture. The privileged integration row failed in 17 nf_tables fixtures with No such file or directory in the dev-02 namespace environment; adjacent nft cases passed. This host result does not replace the compute01 gate, where those fixtures ran on the intended host. The J288 target reached 42 GB and free disk fell to 18 GB; I am deleting that target before continuing.

The final `verify fast` on dev-02 completed at 36e7a6308f5561181b56cf44b67f479a33d6d0f6 but is not green (full log: `/home/claude/.cache/codex-scratch/j288/evidence/verify-fast-final-36e7a63.log`). `static builds` passed, including cross-compiled x86_64 (27,803,408 bytes; +5,385,152) and native aarch64 (25,006,408 bytes; +4,553,128). The workspace test row failed on an unrelated SSH streaming bounded-wait fixture. The privileged integration row failed in 17 `nf_tables` fixtures with `No such file or directory` in the dev-02 namespace environment; adjacent `nft` cases passed. This host result does not replace the compute01 gate, where those fixtures ran on the intended host. The J288 target reached 42 GB and free disk fell to 18 GB; I am deleting that target before continuing.
kayg commented 2026-11-07 03:58:42 +00:00
Author
Owner
Copy link

The dev-02 fast result is red due to missing-file failures in its nf_tables namespace fixtures (17 privileged probes) plus the SSH streaming bounded-wait fixture; its full evidence is at /home/claude/.cache/codex-scratch/j288/evidence/verify-fast-final-36e7a63.log. I removed J288’s 42 GB dev-02 target; free space is back to 59 GB. For a host-appropriate final gate, I am rerunning verify fast on compute01 at 36e7a6308. All five slots are busy and eight jobs are ahead, so it is queued through build-slot and I will wait in the foreground.

The dev-02 fast result is red due to missing-file failures in its nf_tables namespace fixtures (17 privileged probes) plus the SSH streaming bounded-wait fixture; its full evidence is at `/home/claude/.cache/codex-scratch/j288/evidence/verify-fast-final-36e7a63.log`. I removed J288’s 42 GB dev-02 target; free space is back to 59 GB. For a host-appropriate final gate, I am rerunning `verify fast` on compute01 at 36e7a6308. All five slots are busy and eight jobs are ahead, so it is queued through build-slot and I will wait in the foreground.
kayg commented 2026-11-07 04:08:00 +00:00
Author
Owner
Copy link

D348 normal-MTU perf-VM measurement remains blocked. #260 has no passing calibration or J288 lock-window handoff, and the latest isolation audit found unrelated runnable work on measured CPUs. I will not count samples or change masks without those prerequisites. The real Mac↔compute01 J267-path after-measurement is still pending the final compute gate and live-mesh claim.

D348 normal-MTU perf-VM measurement remains blocked. #260 has no passing calibration or J288 lock-window handoff, and the latest isolation audit found unrelated runnable work on measured CPUs. I will not count samples or change masks without those prerequisites. The real Mac↔compute01 J267-path after-measurement is still pending the final compute gate and live-mesh claim.
kayg commented 2026-11-07 04:28:04 +00:00
Author
Owner
Copy link

Progress: the post-fix compute01 verify fast at 36e7a6308 remains in the foreground build-slot FIFO, now position 3 after about 27 minutes; 75 GB is free. Log path: /home/claude/.cache/codex-scratch/j288/evidence/verify-fast-final-compute-36e7a63.log. The dev-02 target is deleted; no live mesh claim or real-path measurement has started while the gate waits.

Progress: the post-fix compute01 `verify fast` at 36e7a6308 remains in the foreground build-slot FIFO, now position 3 after about 27 minutes; 75 GB is free. Log path: `/home/claude/.cache/codex-scratch/j288/evidence/verify-fast-final-compute-36e7a63.log`. The dev-02 target is deleted; no live mesh claim or real-path measurement has started while the gate waits.
kayg commented 2026-11-07 04:38:21 +00:00
Author
Owner
Copy link

Capacity update for the queued compute01 gate: it remains at queue position 3, but free space has fallen from 47 GB to 16 GB, below the 25 GB build-slot floor. It has not started. I am retaining the existing target cache (91 GB) so the pending gate can reuse it; no other target or job is being deleted. I will continue waiting for the slot and disk floor.

Capacity update for the queued compute01 gate: it remains at queue position 3, but free space has fallen from 47 GB to 16 GB, below the 25 GB build-slot floor. It has not started. I am retaining the existing target cache (91 GB) so the pending gate can reuse it; no other target or job is being deleted. I will continue waiting for the slot and disk floor.
kayg commented 2026-11-07 04:41:12 +00:00
Author
Owner
Copy link

Capacity changed again while queued: compute01 is now at 6 GB free (25 GB build-slot floor), all five slots remain occupied, and the J288 gate is still position 3. The gate has not started. I am leaving the existing target untouched until admission is possible without a forced rebuild.

Capacity changed again while queued: compute01 is now at 6 GB free (25 GB build-slot floor), all five slots remain occupied, and the J288 gate is still position 3. The gate has not started. I am leaving the existing target untouched until admission is possible without a forced rebuild.
kayg commented 2026-11-07 05:26:45 +00:00
Author
Owner
Copy link

verify fast at 36e7a6308f5561181b56cf44b67f479a33d6d0f6 completed on compute01 with exit 1. This run completed; the earlier pre-fix run was cut by the VM restart and is not counted as a gate failure.

Gate summary:

Rust gate Result
Nix Cargo vendoring PASS
fmt PASS
migration names PASS
migration tooling tests PASS
VT native fmt PASS
CLI plugin compile-out PASS
clippy PASS
SSH runtime test helper PASS
real-binary fixture paths PASS
Traefik component manifest PASS
release component publication PASS
test FAIL (101)
SSH inode reuse fixture PASS
shim process-per-user PASS
cargo-deny PASS
gate architecture PASS
doc coverage ratchet PASS
shape lints off (D351) PASS
doc comments on diff PASS
fixed test waits PASS
privileged integration PASS
probe release exclusion PASS
nightly fuzz reporter PASS
static builds FAIL (1)

The visible static failure is scripts/test-rustc-paths.py:28 (AssertionError for the expected /infraternal/source.rs path). I have not inspected compute01 again or submitted another gate; I’ll wait for the compute01 back, 5 slots notice, then inspect the saved log and resubmit once in the foreground as directed. Saved log: /home/claude/.cache/codex-scratch/j288/evidence/verify-fast-final-compute-36e7a63.log.

Still pending: compute01 real-root proof and live Mac↔compute01 after measurement; the D348 perf-VM measurement remains blocked because #260’s required calibration/lock handoff is absent. The Mac unit tests already passed under the Mac lock, and the target was removed.

`verify fast` at `36e7a6308f5561181b56cf44b67f479a33d6d0f6` completed on compute01 with exit 1. This run completed; the earlier pre-fix run was cut by the VM restart and is not counted as a gate failure. Gate summary: | Rust gate | Result | |---|---| | Nix Cargo vendoring | PASS | | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | FAIL (101) | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | PASS | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | static builds | FAIL (1) | The visible static failure is `scripts/test-rustc-paths.py:28` (`AssertionError` for the expected `/infraternal/source.rs` path). I have not inspected compute01 again or submitted another gate; I’ll wait for the `compute01 back, 5 slots` notice, then inspect the saved log and resubmit once in the foreground as directed. Saved log: `/home/claude/.cache/codex-scratch/j288/evidence/verify-fast-final-compute-36e7a63.log`. Still pending: compute01 real-root proof and live Mac↔compute01 after measurement; the D348 perf-VM measurement remains blocked because #260’s required calibration/lock handoff is absent. The Mac unit tests already passed under the Mac lock, and the target was removed.
kayg commented 2026-11-07 05:45:17 +00:00
Author
Owner
Copy link

The post-restart verify fast has been resubmitted once in the foreground on compute01 at 36e7a6308f5561181b56cf44b67f479a33d6d0f6. Its build-slot waiter reported queue position 8 with all five slots busy, so the gate has not started yet. I’m keeping this one submission attached and will record its result after it exits.

The post-restart `verify fast` has been resubmitted once in the foreground on compute01 at `36e7a6308f5561181b56cf44b67f479a33d6d0f6`. Its build-slot waiter reported queue position 8 with all five slots busy, so the gate has not started yet. I’m keeping this one submission attached and will record its result after it exits.
kayg commented 2026-11-07 08:11:57 +00:00
Author
Owner
Copy link

Post-restart verify fast at 36e7a6308f5561181b56cf44b67f479a33d6d0f6 completed. All test and privileged rows passed. The only failing row is the static-build source-path helper:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| privileged integration    | PASS         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

The failure is reproducible on unmodified origin/dev 3644950b8e4b5dddeefba350f97a3892189ca193 with CARGO_TARGET_DIR=/home/claude/worktrees/j288/target (the symlink layout used by the gate): scripts/test-rustc-paths.py:28 asserts /infraternal/source.rs. The same baseline helper passes with a physical target directory. The helper, rustc wrapper, remap flags, and toolchain file are unchanged from origin/dev.

Evidence logs on compute01:

  • verify fast: /home/claude/.cache/codex-scratch/j288/evidence/verify-fast-post-restart-36e7a63.log
  • matching origin/dev baseline proof: /home/claude/.cache/codex-scratch/j288/evidence/baseline-rustc-paths-origin-dev-symlink.log
  • control run with a physical target: /home/claude/.cache/codex-scratch/j288/evidence/baseline-rustc-paths-origin-dev.log

The earlier pre-fix compute01 gate interrupted by the VM restart was recorded separately. The after-measurement on the live Mac↔compute01 path and the required real-root proof are still pending; D348 perf-VM remains blocked on #260's missing J288 calibration/lock handoff.

Post-restart `verify fast` at `36e7a6308f5561181b56cf44b67f479a33d6d0f6` completed. All test and privileged rows passed. The only failing row is the static-build source-path helper: ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | Nix Cargo vendoring | PASS | | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | PASS | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` The failure is reproducible on unmodified `origin/dev` `3644950b8e4b5dddeefba350f97a3892189ca193` with `CARGO_TARGET_DIR=/home/claude/worktrees/j288/target` (the symlink layout used by the gate): `scripts/test-rustc-paths.py:28` asserts `/infraternal/source.rs`. The same baseline helper passes with a physical target directory. The helper, rustc wrapper, remap flags, and toolchain file are unchanged from `origin/dev`. Evidence logs on compute01: - `verify fast`: `/home/claude/.cache/codex-scratch/j288/evidence/verify-fast-post-restart-36e7a63.log` - matching `origin/dev` baseline proof: `/home/claude/.cache/codex-scratch/j288/evidence/baseline-rustc-paths-origin-dev-symlink.log` - control run with a physical target: `/home/claude/.cache/codex-scratch/j288/evidence/baseline-rustc-paths-origin-dev.log` The earlier pre-fix compute01 gate interrupted by the VM restart was recorded separately. The after-measurement on the live Mac↔compute01 path and the required real-root proof are still pending; D348 perf-VM remains blocked on #260's missing J288 calibration/lock handoff.
kayg commented 2026-11-07 08:41:10 +00:00
Author
Owner
Copy link

Real-root TUN proof passed on compute01 for the submitted commit:

root clone at 36e7a6308f5561181b56cf44b67f479a33d6d0f6 in /root/tun-proof.NySJY7
elf-runpath-guard: /root/tun-proof.NySJY7/target/debug/deps/gotatun_tun_owner-fb440a76d1f8ef18: interpreter and 1 search path(s) are root-safe
running as root (uid 0) in a fresh network namespace

running 1 test
test an_unprivileged_user_cannot_attach_the_persisted_tun ...
running 1 test
Cannot find device "infraternal0"
ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.90s

TUN_REAL_ROOT_PROOF_EXIT=0 commit=36e7a6308f5561181b56cf44b67f479a33d6d0f6

The proof script's cleanup trap removed its root-owned clone, target, and private run directory on exit. The live mesh measurement remains pending.

Real-root TUN proof passed on compute01 for the submitted commit: ``` root clone at 36e7a6308f5561181b56cf44b67f479a33d6d0f6 in /root/tun-proof.NySJY7 elf-runpath-guard: /root/tun-proof.NySJY7/target/debug/deps/gotatun_tun_owner-fb440a76d1f8ef18: interpreter and 1 search path(s) are root-safe running as root (uid 0) in a fresh network namespace running 1 test test an_unprivileged_user_cannot_attach_the_persisted_tun ... running 1 test Cannot find device "infraternal0" ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.90s TUN_REAL_ROOT_PROOF_EXIT=0 commit=36e7a6308f5561181b56cf44b67f479a33d6d0f6 ``` The proof script's cleanup trap removed its root-owned clone, target, and private run directory on exit. The live mesh measurement remains pending.
kayg commented 2026-11-07 08:49:51 +00:00
Author
Owner
Copy link

J288 progress, 2026-11-07 14:17 IST: compute01 maintenance is scheduled for 14:15–14:45 IST. I have not started any new compute01 work while waiting for the owner’s explicit ‘compute01 back, 5 slots’ notice. The live Mac↔compute01 before/after run remains pending because it needs that host and the live-mesh claim. I’m preparing the Mac measurement controller locally; the D348 perf-VM run still needs a calibrated lock handoff from #260.

J288 progress, 2026-11-07 14:17 IST: compute01 maintenance is scheduled for 14:15–14:45 IST. I have not started any new compute01 work while waiting for the owner’s explicit ‘compute01 back, 5 slots’ notice. The live Mac↔compute01 before/after run remains pending because it needs that host and the live-mesh claim. I’m preparing the Mac measurement controller locally; the D348 perf-VM run still needs a calibrated lock handoff from #260.
kayg commented 2026-11-07 09:02:40 +00:00
Author
Owner
Copy link

J288 Mac staging finding: under lockf -k /tmp/mac-bench.lock, I found a pre-existing /tmp/j288-mtu381-scripts-a670442 on the Mac (owned by lume, dated Nov 6); I left it untouched. The safe controller now uses a unique fixture directory and evidence root, refuses pre-existing paths, verifies the package SHA-256, and cleans only the fixture directory it creates. It and the package are staged on the Mac; controller SHA-256 6b9f007f468f87829e02f9f36e991aae4640492000852cb5e46570119fbb4868, archive d97838850325f142ae19eab71174c51b25aa08303ead6bc68c4b0bd880bfc6b5. The candidate binary hash matches the expected c9f48c47986179e14594b24c150b4cb5bcf90229e64f8ea3112f7a2835e477f4. The Mac disk showed 24 GiB free; no Mac build target was created. Current Mac MTU source fingerprints match commit a6704428c; since then the Mac implementation only changed the visibility/comment of MIN_MESH_MTU, while route-netlink changes are Linux-only. The controller is not running yet because the live measurement depends on compute01, and I am waiting for the owner’s explicit return notice.

J288 Mac staging finding: under `lockf -k /tmp/mac-bench.lock`, I found a pre-existing `/tmp/j288-mtu381-scripts-a670442` on the Mac (owned by lume, dated Nov 6); I left it untouched. The safe controller now uses a unique fixture directory and evidence root, refuses pre-existing paths, verifies the package SHA-256, and cleans only the fixture directory it creates. It and the package are staged on the Mac; controller SHA-256 `6b9f007f468f87829e02f9f36e991aae4640492000852cb5e46570119fbb4868`, archive `d97838850325f142ae19eab71174c51b25aa08303ead6bc68c4b0bd880bfc6b5`. The candidate binary hash matches the expected `c9f48c47986179e14594b24c150b4cb5bcf90229e64f8ea3112f7a2835e477f4`. The Mac disk showed 24 GiB free; no Mac build target was created. Current Mac MTU source fingerprints match commit `a6704428c`; since then the Mac implementation only changed the visibility/comment of `MIN_MESH_MTU`, while route-netlink changes are Linux-only. The controller is not running yet because the live measurement depends on compute01, and I am waiting for the owner’s explicit return notice.
kayg commented 2026-11-07 09:16:35 +00:00
Author
Owner
Copy link

J288 status, 2026-11-07 14:46 IST: the announced maintenance window has passed. A read-only ssh ... true to compute01 succeeds, but I have not started any compute01 work because the requested “compute01 back, 5 slots” notice is not on #402 yet. The Mac after-measurement and compute-side setup remain pending that signal.

J288 status, 2026-11-07 14:46 IST: the announced maintenance window has passed. A read-only `ssh ... true` to compute01 succeeds, but I have not started any compute01 work because the requested “compute01 back, 5 slots” notice is not on #402 yet. The Mac after-measurement and compute-side setup remain pending that signal.
kayg commented 2026-11-07 09:40:06 +00:00
Author
Owner
Copy link

J288 resumed per owner correction: #402 comment 83012 records compute01 back with 5 slots at 02:45 IST; there has been no later freeze. No D348 or perf-VM window is needed for this live functional comparison. I am proceeding under the compute01 live-mesh claim and Mac lock with three repetitions for each direction (Linux→Mac and Mac→Linux), flow count (1 and 8), and path (mesh and plain), comparing shipped dev with head 36e7a6308f. I will post the full table and packet/ICMP observations after the runs.

J288 resumed per owner correction: #402 comment 83012 records compute01 back with 5 slots at 02:45 IST; there has been no later freeze. No D348 or perf-VM window is needed for this live functional comparison. I am proceeding under the compute01 live-mesh claim and Mac lock with three repetitions for each direction (Linux→Mac and Mac→Linux), flow count (1 and 8), and path (mesh and plain), comparing shipped dev with head 36e7a6308f5561181b56cf44b67f479a33d6d0f6. I will post the full table and packet/ICMP observations after the runs.
kayg commented 2026-11-07 09:41:13 +00:00
Author
Owner
Copy link

J288 resumed measurement is waiting on the live-mesh claim. The atomic claim attempt found ~/mesh-vm.claim already held by J207 (since 2026-11-07T06:55:47Z); I did not overwrite it or touch live mesh state. I am waiting for the claim to be released, and no measurement run has started.

J288 resumed measurement is waiting on the live-mesh claim. The atomic claim attempt found `~/mesh-vm.claim` already held by J207 (since 2026-11-07T06:55:47Z); I did not overwrite it or touch live mesh state. I am waiting for the claim to be released, and no measurement run has started.
kayg commented 2026-11-07 09:44:42 +00:00
Author
Owner
Copy link

Gate tracking: opened #421 for the reproducible verify fast static-build source-path failure. Attached the post-restart gate log, clean origin/dev symlink-target failure, and physical-target passing control. This is now tracked separately and does not block the live MTU comparison.

Gate tracking: opened #421 for the reproducible `verify fast` static-build source-path failure. Attached the post-restart gate log, clean `origin/dev` symlink-target failure, and physical-target passing control. This is now tracked separately and does not block the live MTU comparison.
kayg commented 2026-11-07 10:27:31 +00:00
Author
Owner
Copy link

J288 measurement controller audit (2026-11-07): the staged live controller currently records one sample per path and does not run the baseline mesh pair. The retained J267 table has one mesh sample and two plain samples. Neither satisfies the corrected request for three repetitions across both shipped-dev and fixed-head paths.

I am replacing it with 12 measurement labels: shipped-dev and fixed-head × plain and mesh × 3 repetitions. Each label runs 1-flow and 8-flow TCP in both directions. I will also record route MTUs and maximum-size IPv4 DF pings in both directions to check the 1200-byte mesh route and detect ICMP/PMTU black-holing.

No live test has started. The compute01 mesh claim is still held by J207, and the Mac lock is busy. The compute01 Nix launcher check passed (cargo 1.99.0).

J288 measurement controller audit (2026-11-07): the staged live controller currently records one sample per path and does not run the baseline mesh pair. The retained J267 table has one mesh sample and two plain samples. Neither satisfies the corrected request for three repetitions across both shipped-dev and fixed-head paths. I am replacing it with 12 measurement labels: shipped-dev and fixed-head × plain and mesh × 3 repetitions. Each label runs 1-flow and 8-flow TCP in both directions. I will also record route MTUs and maximum-size IPv4 DF pings in both directions to check the 1200-byte mesh route and detect ICMP/PMTU black-holing. No live test has started. The compute01 mesh claim is still held by J207, and the Mac lock is busy. The compute01 Nix launcher check passed (`cargo 1.99.0`).
kayg commented 2026-11-07 10:54:04 +00:00
Author
Owner
Copy link

J288 measurement progress (2026-11-07 16:53 IST): the 12-label before/after matrix is staged. The Linux x86_64 binaries for shipped-dev 9b0c731e8 and fixed head 36e7a6308 are compiling sequentially in compute01 build-slot, with CARGO_INCREMENTAL=0 and one target. Live build log: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/build-linux-live-binaries.log.

The Mac baseline build command is waiting in the foreground for /tmp/mac-bench.lock, currently held by J340's Mac check. The build script will remove its target before releasing the lock. The live mesh claim remains held by J207, so no live mesh state or measurement has started.

The functional Mac↔compute01 matrix does not use the perf lock or D348 calibration. The separate normal-MTU perf-VM comparison remains unavailable while J317/J279 own the perf window.

J288 measurement progress (2026-11-07 16:53 IST): the 12-label before/after matrix is staged. The Linux x86_64 binaries for shipped-dev `9b0c731e8` and fixed head `36e7a6308` are compiling sequentially in compute01 build-slot, with `CARGO_INCREMENTAL=0` and one target. Live build log: `/home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/build-linux-live-binaries.log`. The Mac baseline build command is waiting in the foreground for `/tmp/mac-bench.lock`, currently held by J340's Mac check. The build script will remove its target before releasing the lock. The live mesh claim remains held by J207, so no live mesh state or measurement has started. The functional Mac↔compute01 matrix does not use the perf lock or D348 calibration. The separate normal-MTU perf-VM comparison remains unavailable while J317/J279 own the perf window.
kayg commented 2026-11-07 11:04:18 +00:00
Author
Owner
Copy link

Update (2026-11-07 16:30 IST): the shipped-dev Mac binary build completed under the Mac lock, and its target was deleted before releasing the lock. Binary SHA-256: 0071d6bb2e4efee33ef88bd0d935979d858fe70e17744d7ed318c96512f06c51. Build log: /Users/claude/.cache/codex-scratch/j288/evidence/mac-build-dev-baseline-9b0c731.log.

The fixed Mac binary is present at /tmp/infraternal-mtu381-final (SHA-256 c9f48c47986179e14594b24c150b4cb5bcf90229e64f8ea3112f7a2835e477f4). Existing source-provenance evidence shows the Mac node source matches the current branch; the helper difference is visibility/documentation only. Linux before/after binaries are still compiling in compute01 build-slot; log: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/build-linux-live-binaries.log.

No live measurement has started: J207 still holds ~/mesh-vm.claim and J340 currently holds /tmp/mac-bench.lock. The existing foreground waiters remain active.

Update (2026-11-07 16:30 IST): the shipped-dev Mac binary build completed under the Mac lock, and its target was deleted before releasing the lock. Binary SHA-256: 0071d6bb2e4efee33ef88bd0d935979d858fe70e17744d7ed318c96512f06c51. Build log: /Users/claude/.cache/codex-scratch/j288/evidence/mac-build-dev-baseline-9b0c731.log. The fixed Mac binary is present at /tmp/infraternal-mtu381-final (SHA-256 c9f48c47986179e14594b24c150b4cb5bcf90229e64f8ea3112f7a2835e477f4). Existing source-provenance evidence shows the Mac node source matches the current branch; the helper difference is visibility/documentation only. Linux before/after binaries are still compiling in compute01 build-slot; log: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/build-linux-live-binaries.log. No live measurement has started: J207 still holds ~/mesh-vm.claim and J340 currently holds /tmp/mac-bench.lock. The existing foreground waiters remain active.
kayg commented 2026-11-07 11:13:18 +00:00
Author
Owner
Copy link

Linux x86_64 before/after binaries are built and verified from the exact sources requested. Shipped-dev 9b0c731e89b70408ac719ef3a10cb1bf36bebf99: SHA-256 655a5306928cc8bc11c04cb314b938e478dc249f8c97826b484fb8e71b5e214d. Fixed head 36e7a6308f5561181b56cf44b67f479a33d6d0f6: SHA-256 f6b944948f6b76a5db3c33eda1996665a26aab16ef85e2b3be75586c02a31a1d. Build log: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/build-linux-live-binaries.log. The target was removed and the compute01 worktree is clean at the fixed head.

The Mac controller and fixture archive are staged at their unique expected paths; checksums match the local inputs and the report directory is absent. No live measurement has started yet: J207 still owns the mesh claim, compute01's five build slots are busy with queued work, and J340 currently holds the Mac lock.

Linux x86_64 before/after binaries are built and verified from the exact sources requested. Shipped-dev `9b0c731e89b70408ac719ef3a10cb1bf36bebf99`: SHA-256 `655a5306928cc8bc11c04cb314b938e478dc249f8c97826b484fb8e71b5e214d`. Fixed head `36e7a6308f5561181b56cf44b67f479a33d6d0f6`: SHA-256 `f6b944948f6b76a5db3c33eda1996665a26aab16ef85e2b3be75586c02a31a1d`. Build log: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/build-linux-live-binaries.log. The target was removed and the compute01 worktree is clean at the fixed head. The Mac controller and fixture archive are staged at their unique expected paths; checksums match the local inputs and the report directory is absent. No live measurement has started yet: J207 still owns the mesh claim, compute01's five build slots are busy with queued work, and J340 currently holds the Mac lock.
kayg commented 2026-11-07 11:15:04 +00:00
Author
Owner
Copy link

Correction to my previous status: the Mac lock is currently free (checked at 17:44 IST); the sentence saying J340 held it was stale by the time I posted. J207 still owns ~/mesh-vm.claim. The Linux binaries remain ready, and no live mesh measurement has started.

Correction to my previous status: the Mac lock is currently free (checked at 17:44 IST); the sentence saying J340 held it was stale by the time I posted. J207 still owns ~/mesh-vm.claim. The Linux binaries remain ready, and no live mesh measurement has started.
kayg commented 2026-11-07 11:21:46 +00:00
Author
Owner
Copy link

Live preflight finding: the J288 waiter acquired ~/mesh-vm.claim at 2026-11-07T11:16:23Z. The compute01 runner then found no Linux iperf3 executable in PATH, the Nix profile, /usr, or the prepared scratch paths. No interface or iperf server was started. I released the claim immediately after that preflight (CLAIM_RELEASED) and am preparing an iperf3 binary before reclaiming for the matrix.

The Mac lock was free at the last check; the controller and fixture archive remain staged at their unique /tmp paths with verified hashes.

Live preflight finding: the J288 waiter acquired ~/mesh-vm.claim at 2026-11-07T11:16:23Z. The compute01 runner then found no Linux iperf3 executable in PATH, the Nix profile, /usr, or the prepared scratch paths. No interface or iperf server was started. I released the claim immediately after that preflight (`CLAIM_RELEASED`) and am preparing an iperf3 binary before reclaiming for the matrix. The Mac lock was free at the last check; the controller and fixture archive remain staged at their unique /tmp paths with verified hashes.
kayg commented 2026-11-07 11:29:09 +00:00
Author
Owner
Copy link

Resolved the iperf preflight without a build: compute01 already has iperf3 3.21 at /nix/store/i6jh66v9gq8jz1d06z7463c9l8s9mjf1-iperf-3.21/bin/iperf3. --version confirms 3.21. I canceled the unnecessary build-slot fetch waiter before it acquired a slot; no package build ran. The verified fixture archive is extracted into the unique Linux fixture directory, and the run/evidence paths are confirmed absent.

J328 currently owns ~/mesh-vm.claim. My foreground waiter is ready to reclaim it; the Mac lock was free at the last check.

Resolved the iperf preflight without a build: compute01 already has iperf3 3.21 at /nix/store/i6jh66v9gq8jz1d06z7463c9l8s9mjf1-iperf-3.21/bin/iperf3. `--version` confirms 3.21. I canceled the unnecessary build-slot fetch waiter before it acquired a slot; no package build ran. The verified fixture archive is extracted into the unique Linux fixture directory, and the run/evidence paths are confirmed absent. J328 currently owns ~/mesh-vm.claim. My foreground waiter is ready to reclaim it; the Mac lock was free at the last check.
kayg commented 2026-11-07 11:29:54 +00:00
Author
Owner
Copy link

The J288 foreground waiter acquired ~/mesh-vm.claim at 2026-11-07T11:29:02Z. Starting the live Mac↔compute01 matrix now with the verified Linux shipped-dev/fixed-head binaries, Mac shipped-dev/fixed binary hashes, and iperf3 3.21. The Mac controller acquires and holds /tmp/mac-bench.lock for the full before/after run.

Linux evidence destination: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/run/evidence; runner log: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/live-matrix-runner.log.

The J288 foreground waiter acquired ~/mesh-vm.claim at 2026-11-07T11:29:02Z. Starting the live Mac↔compute01 matrix now with the verified Linux shipped-dev/fixed-head binaries, Mac shipped-dev/fixed binary hashes, and iperf3 3.21. The Mac controller acquires and holds /tmp/mac-bench.lock for the full before/after run. Linux evidence destination: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/run/evidence; runner log: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/live-matrix-runner.log.
kayg commented 2026-11-07 11:37:23 +00:00
Author
Owner
Copy link

Live-run diagnostic and correction:

  • The first matrix invocation failed at argument validation before claim verification or any process/TUN startup. Root cause: it concatenated four valid SHA-256 strings and matched the 64-character validator. Evidence: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/live-matrix-runner.log.
  • I corrected only the scratch runner. With four valid hashes it now advances to the claim check; a malformed hash is still rejected. Evidence: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/digest-validator-valid-args.log and digest-validator-invalid-args.log.
  • While J288 held the mesh claim, a read-only preflight showed a preexisting infraternal0 interface, UP/NO-CARRIER, MTU 1380, with address 10.254.0.1/32. The runner did not create it. Evidence: /home/kayg/.cache/codex-scratch/j288/evidence/live-matrix-20261107/preflight-existing-interface-20261107.log. I have not removed or changed it because its ownership is not established.

I released the J288 claim after that preflight. The live matrix is waiting for the interface owner to clean up before retrying; the Mac controller did not start and no Mac lock was taken.

Live-run diagnostic and correction: - The first matrix invocation failed at argument validation before claim verification or any process/TUN startup. Root cause: it concatenated four valid SHA-256 strings and matched the 64-character validator. Evidence: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/live-matrix-runner.log. - I corrected only the scratch runner. With four valid hashes it now advances to the claim check; a malformed hash is still rejected. Evidence: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/digest-validator-valid-args.log and digest-validator-invalid-args.log. - While J288 held the mesh claim, a read-only preflight showed a preexisting `infraternal0` interface, UP/NO-CARRIER, MTU 1380, with address 10.254.0.1/32. The runner did not create it. Evidence: /home/kayg/.cache/codex-scratch/j288/evidence/live-matrix-20261107/preflight-existing-interface-20261107.log. I have not removed or changed it because its ownership is not established. I released the J288 claim after that preflight. The live matrix is waiting for the interface owner to clean up before retrying; the Mac controller did not start and no Mac lock was taken.
kayg commented 2026-11-07 11:40:10 +00:00
Author
Owner
Copy link

The scratch-runner digest fix is in place and its valid/invalid argument checks are recorded. A fresh, read-only check under the J288 claim at 2026-11-07T11:39:18Z confirmed the preexisting infraternal0 still exists at MTU 1380 with the same alias. I released the claim immediately afterward. I have not stopped or deleted it. The live matrix remains pending cleanup by whoever owns that interface; no Mac controller or measurement is running.

The scratch-runner digest fix is in place and its valid/invalid argument checks are recorded. A fresh, read-only check under the J288 claim at 2026-11-07T11:39:18Z confirmed the preexisting `infraternal0` still exists at MTU 1380 with the same alias. I released the claim immediately afterward. I have not stopped or deleted it. The live matrix remains pending cleanup by whoever owns that interface; no Mac controller or measurement is running.
kayg commented 2026-11-07 11:41:24 +00:00
Author
Owner
Copy link

The preexisting infraternal0 is still present after releasing the J288 claim. Its alias/address do not identify an owner in the J288 evidence, and the task rules prohibit deleting an interface J288 did not create. Whoever owns it should clean it using their own run's cleanup, or identify its owner. J288 will retry as soon as the interface is gone; all binaries and runner dependencies are ready.

The preexisting `infraternal0` is still present after releasing the J288 claim. Its alias/address do not identify an owner in the J288 evidence, and the task rules prohibit deleting an interface J288 did not create. Whoever owns it should clean it using their own run's cleanup, or identify its owner. J288 will retry as soon as the interface is gone; all binaries and runner dependencies are ready.
kayg commented 2026-11-07 11:48:57 +00:00
Author
Owner
Copy link

Additional read-only ownership check under the J288 claim at 2026-11-07T11:47:14Z: infraternal0 is a persistent TUN (tun vnet_hdr persist user 0 group 0). No J288 agent process was present, and the visible J207 process had no open /dev/net/tun descriptor. This still does not establish who created the persistent interface, so I released the claim without cleanup. Evidence: /home/kayg/.cache/codex-scratch/j288/evidence/live-matrix-20261107/preflight-interface-owner-20261107.md.

Additional read-only ownership check under the J288 claim at 2026-11-07T11:47:14Z: `infraternal0` is a persistent TUN (`tun vnet_hdr persist user 0 group 0`). No J288 agent process was present, and the visible J207 process had no open /dev/net/tun descriptor. This still does not establish who created the persistent interface, so I released the claim without cleanup. Evidence: /home/kayg/.cache/codex-scratch/j288/evidence/live-matrix-20261107/preflight-interface-owner-20261107.md.
kayg commented 2026-11-07 12:02:35 +00:00
Author
Owner
Copy link

Resumed after kayg's #402 cleanup notice. J288 acquired ~/mesh-vm.claim at 2026-11-07T12:02:07Z; preflight confirms infraternal0 is absent and TCP/45269 is free. Starting the full 12-label shipped-dev/fixed-head × plain/mesh × 3-repetition matrix now. The Mac controller takes /tmp/mac-bench.lock for the complete Mac build/test step. I will release the mesh claim after cleanup, within the 60-minute window.

Resumed after kayg's #402 cleanup notice. J288 acquired ~/mesh-vm.claim at 2026-11-07T12:02:07Z; preflight confirms infraternal0 is absent and TCP/45269 is free. Starting the full 12-label shipped-dev/fixed-head × plain/mesh × 3-repetition matrix now. The Mac controller takes /tmp/mac-bench.lock for the complete Mac build/test step. I will release the mesh claim after cleanup, within the 60-minute window.
kayg commented 2026-11-07 12:20:37 +00:00
Author
Owner
Copy link

Live-mesh measurement update (2026-11-07, 17:44 IST)

The claim was held from 17:32 to 17:44 IST and has been released. The Mac route to compute01 reported MTU 1280 on utun100. The runner completed the three shipped-dev plain repetitions, then failed while starting the Linux test binary (binary closed connection), before configuring a mesh peer. This attempt therefore has no mesh or fixed-head results.

Partial shipped-dev plain throughput (Mbit/s; down is Linux→Mac, up is Mac→Linux):

Rep 1 flow down 1 flow up 8 flows down 8 flows up
1 127.4 75.1 424.8 154.0
2 114.8 57.3 73.2 140.9
3 101.1 56.3 82.5 88.2

Failure report: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/run-final/evidence/mesh-mtu-live-matrix-report.json. It records the 1280-MTU route and cleanup (infraternal0 absent, Mac mesh interface absent, no new utun). The runner also terminated the Mac controller and released its lock. I am restarting the full matrix from a clean evidence directory after a no-mesh startup check.

Live-mesh measurement update (2026-11-07, 17:44 IST) The claim was held from 17:32 to 17:44 IST and has been released. The Mac route to compute01 reported MTU 1280 on utun100. The runner completed the three shipped-dev plain repetitions, then failed while starting the Linux test binary (`binary closed connection`), before configuring a mesh peer. This attempt therefore has no mesh or fixed-head results. Partial shipped-dev plain throughput (Mbit/s; down is Linux→Mac, up is Mac→Linux): | Rep | 1 flow down | 1 flow up | 8 flows down | 8 flows up | |---|---:|---:|---:|---:| | 1 | 127.4 | 75.1 | 424.8 | 154.0 | | 2 | 114.8 | 57.3 | 73.2 | 140.9 | | 3 | 101.1 | 56.3 | 82.5 | 88.2 | Failure report: `/home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/run-final/evidence/mesh-mtu-live-matrix-report.json`. It records the 1280-MTU route and cleanup (`infraternal0` absent, Mac mesh interface absent, no new utun). The runner also terminated the Mac controller and released its lock. I am restarting the full matrix from a clean evidence directory after a no-mesh startup check.
kayg commented 2026-11-07 12:25:17 +00:00
Author
Owner
Copy link

Live-mesh retry update (2026-11-07, 17:53 IST)

The second attempt exited before mesh setup. Its report and mac-ssh.stderr show that the Mac controller refused the already-used evidence root. I had staged a rerun controller under a different filename while the Linux runner still invoked the original filename, so the old script ran. The controller also could not remove the staged /tmp files because it did not own them; I will clean those files explicitly and stage the corrected script at the exact path the runner uses.

Evidence: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/run-rerun/evidence/mesh-mtu-live-matrix-report.json and /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/run-rerun/evidence/mac-ssh.stderr. The report confirms Linux infraternal0 absent and no listener on port 45269. No mesh configuration or throughput sample ran. The claim acquired at 17:52 IST was released at 17:53 IST.

Live-mesh retry update (2026-11-07, 17:53 IST) The second attempt exited before mesh setup. Its report and `mac-ssh.stderr` show that the Mac controller refused the already-used evidence root. I had staged a rerun controller under a different filename while the Linux runner still invoked the original filename, so the old script ran. The controller also could not remove the staged `/tmp` files because it did not own them; I will clean those files explicitly and stage the corrected script at the exact path the runner uses. Evidence: `/home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/run-rerun/evidence/mesh-mtu-live-matrix-report.json` and `/home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/run-rerun/evidence/mac-ssh.stderr`. The report confirms Linux `infraternal0` absent and no listener on port 45269. No mesh configuration or throughput sample ran. The claim acquired at 17:52 IST was released at 17:53 IST.
kayg commented 2026-11-07 12:30:59 +00:00
Author
Owner
Copy link

Live-mesh claim update (2026-11-07, 18:00 IST)

J254 acquired ~/mesh-vm.claim at 18:00 IST while I was correcting the runner. My noclobber claim attempt did not replace it. The runner then rejected the J254 claim before launching the Mac controller or configuring a mesh interface. I am leaving J254's claim untouched and waiting in the foreground to re-claim after it is released.

Evidence: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/run-rerun2/evidence/mesh-mtu-live-matrix-report.json records the other job's claim and infraternal0 absent; the compute port 45269 is also free. No throughput sample ran in this attempt.

Live-mesh claim update (2026-11-07, 18:00 IST) J254 acquired `~/mesh-vm.claim` at 18:00 IST while I was correcting the runner. My `noclobber` claim attempt did not replace it. The runner then rejected the J254 claim before launching the Mac controller or configuring a mesh interface. I am leaving J254's claim untouched and waiting in the foreground to re-claim after it is released. Evidence: `/home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/run-rerun2/evidence/mesh-mtu-live-matrix-report.json` records the other job's claim and `infraternal0` absent; the compute port 45269 is also free. No throughput sample ran in this attempt.
kayg commented 2026-11-07 12:44:00 +00:00
Author
Owner
Copy link

Live-mesh state update (2026-11-07, 18:13 IST)

After I released my claim at 18:07 IST, J254 re-claimed at 18:10 IST. The compute host now has a persistent infraternal0 TUN interface at MTU 1380 with a GotaTun alias. lsof -t /dev/net/tun reported no open process, and port 45269 is free. My preceding aborted run had confirmed infraternal0 absent; I did not create or touch the current interface.

Evidence: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/unowned-interface-after-j254-20261107-1243.txt. I am leaving the interface and J254's claim untouched and will resume the matrix when the claim is free and the interface has been cleaned up.

Live-mesh state update (2026-11-07, 18:13 IST) After I released my claim at 18:07 IST, J254 re-claimed at 18:10 IST. The compute host now has a persistent `infraternal0` TUN interface at MTU 1380 with a GotaTun alias. `lsof -t /dev/net/tun` reported no open process, and port 45269 is free. My preceding aborted run had confirmed `infraternal0` absent; I did not create or touch the current interface. Evidence: `/home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/unowned-interface-after-j254-20261107-1243.txt`. I am leaving the interface and J254's claim untouched and will resume the matrix when the claim is free and the interface has been cleaned up.
kayg commented 2026-11-07 13:46:54 +00:00
Author
Owner
Copy link

Live-mesh measurement update (2026-11-07, 19:13 IST)

The corrected output root captured all three shipped-dev plain repetitions, but the Linux agent again closed its fixture connection before mesh setup. The Mac route was MTU 1280. The runner cleaned up; Linux infraternal0 and port 45269 were absent, and the Mac controller reported no mesh interface or new utun. The claim held from 19:07 to 19:13 IST and has been released.

Partial shipped-dev plain throughput (Mbit/s; down is Linux→Mac, up is Mac→Linux):

Rep 1 flow down 1 flow up 8 flows down 8 flows up
1 99.5 105.5 218.8 130.7
2 58.8 74.4 158.7 123.0
3 47.5 102.1 495.1 103.5

Failure report: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/run-final-live/evidence/mesh-mtu-live-matrix-report.json. A no-mesh control handshake immediately afterward passed without creating an interface: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/diagnostics/linux-shipped-dev-control-no-mesh-20261107-1344.txt. I will retain Linux stderr logs in the next attempt and retry from a fresh evidence root.

Live-mesh measurement update (2026-11-07, 19:13 IST) The corrected output root captured all three shipped-dev plain repetitions, but the Linux agent again closed its fixture connection before mesh setup. The Mac route was MTU 1280. The runner cleaned up; Linux `infraternal0` and port 45269 were absent, and the Mac controller reported no mesh interface or new utun. The claim held from 19:07 to 19:13 IST and has been released. Partial shipped-dev plain throughput (Mbit/s; down is Linux→Mac, up is Mac→Linux): | Rep | 1 flow down | 1 flow up | 8 flows down | 8 flows up | |---|---:|---:|---:|---:| | 1 | 99.5 | 105.5 | 218.8 | 130.7 | | 2 | 58.8 | 74.4 | 158.7 | 123.0 | | 3 | 47.5 | 102.1 | 495.1 | 103.5 | Failure report: `/home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/run-final-live/evidence/mesh-mtu-live-matrix-report.json`. A no-mesh control handshake immediately afterward passed without creating an interface: `/home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/diagnostics/linux-shipped-dev-control-no-mesh-20261107-1344.txt`. I will retain Linux stderr logs in the next attempt and retry from a fresh evidence root.
kayg commented 2026-11-07 13:57:10 +00:00
Author
Owner
Copy link

Live-mesh runner diagnosis (2026-11-07, 19:27 IST)

The Linux startup stderr identifies the failure: infraternal agent failed: path must be shorter than SUN_LEN. The runner put the agent's state beneath a deep evidence/scratch directory, making its AF_UNIX socket path too long before mesh setup. This attempt therefore has no mesh result; the three shipped-dev plain samples are recorded, and should not be compared with mesh data.

Evidence: report /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/run-final-attempt2/evidence/mesh-mtu-live-matrix-report.json; startup stderr /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/run-final-attempt2/evidence/linux-agent-node-a7f2rvy5.stderr. The report confirms no Linux interface; the Mac controller report records no mesh interface or new utun. The claim is absent now. Next run will keep evidence in the existing directory and put runtime scratch under a short /tmp path.

Live-mesh runner diagnosis (2026-11-07, 19:27 IST) The Linux startup stderr identifies the failure: `infraternal agent failed: path must be shorter than SUN_LEN`. The runner put the agent's state beneath a deep evidence/scratch directory, making its AF_UNIX socket path too long before mesh setup. This attempt therefore has no mesh result; the three shipped-dev plain samples are recorded, and should not be compared with mesh data. Evidence: report `/home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/run-final-attempt2/evidence/mesh-mtu-live-matrix-report.json`; startup stderr `/home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/run-final-attempt2/evidence/linux-agent-node-a7f2rvy5.stderr`. The report confirms no Linux interface; the Mac controller report records no mesh interface or new utun. The claim is absent now. Next run will keep evidence in the existing directory and put runtime scratch under a short `/tmp` path.
kayg commented 2026-11-07 16:11:49 +00:00
Author
Owner
Copy link

J288 live Mac ↔ compute01 matrix, attempt 8 (2026-11-07). Underlay was NetBird (wt0/utun100) at MTU 1280. Three repetitions per path/flow/build; all rates are Mbit/s. “Down” means Linux → Mac; “up” means Mac → Linux. Values are median (raw repetitions), except where fewer than two samples were valid.

Build Path Linux → Mac, 1 flow Linux → Mac, 8 flows Mac → Linux, 1 flow Mac → Linux, 8 flows
shipped dev plain 110.4 (110.4, 114.3, 70.4) 178.2 (497.1, 178.2, 139.3) 96.8 (79.9, 101.8, 96.8) 135.5 (289.8, 135.5, 84.2)
shipped dev mesh 0.4 (0.3, 0.4, 0.4) n=1: 58.0 (ERR, ERR, 58.0) 67.7 (63.9, 96.5, 67.7) n=1: 65.7 (ERR, ERR, 65.7)
fixed Linux head 36e7a6308 plain 105.8 (106.0, 105.8, 71.3) 235.3 (250.4, 211.9, 235.3) 99.4 (79.8, 99.7, 99.4) 162.9 (162.9, 193.5, 93.0)
fixed Linux head 36e7a6308 mesh 78.0 (64.3, 78.0, 84.0) 259.0 (124.8, 267.1, 259.0) 96.1 (60.2, 97.6, 96.1) 131.9 (99.2, 131.9, 155.2)

The shipped-dev mesh 8-flow runs had iperf control-channel errors in two of three repetitions in each direction; those are recorded as ERR, not throughput. With the fix, Linux → Mac mesh is 78.0 Mbit/s at 1 flow versus 105.8 plain (1.36× lower), and 259.0 at 8 flows versus 235.3 plain. Mac → Linux remains near plain at 1 flow (96.1 vs 99.4) and 8 flows (131.9 vs 162.9).

The fixed Linux binary was built from 36e7a6308f5561181b56cf44b67f479a33d6d0f6. The fixed Mac binary reports source 2171a4b409413f57d2f3d11fb5cba1b7c4733025, an equivalent route-MTU implementation commit. Comparing that source to 36e7a6308, the only mesh-source differences are Linux route handling and the visibility of a constant used by that Linux code; macos_node.rs and the macOS route-MTU implementation are unchanged. An exact-head Mac rebuild was attempted under the Mac lock, but the unrelated infraternal-agent/src/logins.rs:424 SockFlag::SOCK_CLOEXEC compile error stopped it. The Mac source provenance is therefore disclosed rather than labeled as an exact-head build.

Packet-size evidence: both peers installed a 1200-byte route MTU over the 1280-byte underlay; the Linux mesh link stayed at 1280 for IPv6. 1200-byte inner IPv4 probes passed 3/3 in both directions. Linux rejected a 1201-byte DF probe locally with sendmsg: Message too long; Mac's 1201-byte probes received replies. The run showed no PMTU black-holing at these boundaries.

Cleanup passed at the end of this attempt: the report records no Linux mesh interface, and the Mac controller records no mesh interface or new utun and stopped: true. A later state check sees the next job's claim (J254, 2026-11-07T15:35:27Z) and its infraternal0; I left that state untouched.

Evidence:

  • Linux report and raw runner output: claude@compute01:/home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/run-final-attempt8/evidence/mesh-mtu-live-matrix-report.json
  • Mac raw iperf/ping samples: lume@mac:/Users/lume/.cache/codex-scratch/j288/live-matrix-20261107-attempt8/evidence/
  • Mac cleanup/controller report: .../evidence/mac-controller-report.json
  • Exact-head Mac build failure: lume@mac:/Users/lume/.cache/codex-scratch/j288/build-mac-36e7.log
**J288 live Mac ↔ compute01 matrix, attempt 8 (2026-11-07).** Underlay was NetBird (`wt0`/`utun100`) at MTU 1280. Three repetitions per path/flow/build; all rates are Mbit/s. “Down” means Linux → Mac; “up” means Mac → Linux. Values are median (raw repetitions), except where fewer than two samples were valid. | Build | Path | Linux → Mac, 1 flow | Linux → Mac, 8 flows | Mac → Linux, 1 flow | Mac → Linux, 8 flows | |---|---|---:|---:|---:|---:| | shipped dev | plain | 110.4 (110.4, 114.3, 70.4) | 178.2 (497.1, 178.2, 139.3) | 96.8 (79.9, 101.8, 96.8) | 135.5 (289.8, 135.5, 84.2) | | shipped dev | mesh | 0.4 (0.3, 0.4, 0.4) | n=1: 58.0 (ERR, ERR, 58.0) | 67.7 (63.9, 96.5, 67.7) | n=1: 65.7 (ERR, ERR, 65.7) | | fixed Linux head `36e7a6308` | plain | 105.8 (106.0, 105.8, 71.3) | 235.3 (250.4, 211.9, 235.3) | 99.4 (79.8, 99.7, 99.4) | 162.9 (162.9, 193.5, 93.0) | | fixed Linux head `36e7a6308` | mesh | 78.0 (64.3, 78.0, 84.0) | 259.0 (124.8, 267.1, 259.0) | 96.1 (60.2, 97.6, 96.1) | 131.9 (99.2, 131.9, 155.2) | The shipped-dev mesh 8-flow runs had iperf control-channel errors in two of three repetitions in each direction; those are recorded as `ERR`, not throughput. With the fix, Linux → Mac mesh is 78.0 Mbit/s at 1 flow versus 105.8 plain (1.36× lower), and 259.0 at 8 flows versus 235.3 plain. Mac → Linux remains near plain at 1 flow (96.1 vs 99.4) and 8 flows (131.9 vs 162.9). The fixed Linux binary was built from `36e7a6308f5561181b56cf44b67f479a33d6d0f6`. The fixed Mac binary reports source `2171a4b409413f57d2f3d11fb5cba1b7c4733025`, an equivalent route-MTU implementation commit. Comparing that source to `36e7a6308`, the only mesh-source differences are Linux route handling and the visibility of a constant used by that Linux code; `macos_node.rs` and the macOS route-MTU implementation are unchanged. An exact-head Mac rebuild was attempted under the Mac lock, but the unrelated `infraternal-agent/src/logins.rs:424` `SockFlag::SOCK_CLOEXEC` compile error stopped it. The Mac source provenance is therefore disclosed rather than labeled as an exact-head build. Packet-size evidence: both peers installed a 1200-byte route MTU over the 1280-byte underlay; the Linux mesh link stayed at 1280 for IPv6. 1200-byte inner IPv4 probes passed 3/3 in both directions. Linux rejected a 1201-byte DF probe locally with `sendmsg: Message too long`; Mac's 1201-byte probes received replies. The run showed no PMTU black-holing at these boundaries. Cleanup passed at the end of this attempt: the report records no Linux mesh interface, and the Mac controller records no mesh interface or new utun and `stopped: true`. A later state check sees the next job's claim (`J254`, 2026-11-07T15:35:27Z) and its `infraternal0`; I left that state untouched. Evidence: - Linux report and raw runner output: `claude@compute01:/home/claude/.cache/codex-scratch/j288/evidence/live-matrix-20261107/run-final-attempt8/evidence/mesh-mtu-live-matrix-report.json` - Mac raw iperf/ping samples: `lume@mac:/Users/lume/.cache/codex-scratch/j288/live-matrix-20261107-attempt8/evidence/` - Mac cleanup/controller report: `.../evidence/mac-controller-report.json` - Exact-head Mac build failure: `lume@mac:/Users/lume/.cache/codex-scratch/j288/build-mac-36e7.log`
kayg commented 2026-11-07 16:21:54 +00:00
Author
Owner
Copy link

Gate status at 36e7a6308f5561181b56cf44b67f479a33d6d0f6. The latest verify quick receipt records VERIFY_QUICK_EXIT=0 and 55 changed mesh tests passed; that receipt is from a6704428c8241c9ae38cd02413af3366a41c985c. The final verify fast ran at this issue's head 36e7a6308; its summary is:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| privileged integration    | PASS         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

The static failure is scripts/test-rustc-paths.py:28, which asserts the wrong source path under a symlinked target. It reproduces on unmodified origin/dev; the same baseline helper passes with a physical target. Issue #421 has the fast log, symlink-target control, and physical-target control attached. Evidence on compute01: /home/claude/.cache/codex-scratch/j288/evidence/verify-fast-post-restart-36e7a63.log, /home/claude/.cache/codex-scratch/j288/evidence/baseline-rustc-paths-origin-dev-symlink.log, and /home/claude/.cache/codex-scratch/j288/evidence/baseline-rustc-paths-origin-dev.log.

The separate normal-MTU perf-VM comparison has not run. At the latest check, the perf lock was free but J317 had announced its corrected D348 comparison sequence; perf.slice was restored to AllowedCPUs=EffectiveCPUs=1-3. I did not acquire or queue a competing run, and changed no perf CPU masks. The existing J288 direction allows an owner-directed, uncalibrated matched hold after D348 readiness; this measurement remains pending that shared window.

**Gate status at `36e7a6308f5561181b56cf44b67f479a33d6d0f6`.** The latest `verify quick` receipt records `VERIFY_QUICK_EXIT=0` and 55 changed mesh tests passed; that receipt is from `a6704428c8241c9ae38cd02413af3366a41c985c`. The final `verify fast` ran at this issue's head `36e7a6308`; its summary is: ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | Nix Cargo vendoring | PASS | | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | PASS | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` The static failure is `scripts/test-rustc-paths.py:28`, which asserts the wrong source path under a symlinked target. It reproduces on unmodified `origin/dev`; the same baseline helper passes with a physical target. Issue #421 has the fast log, symlink-target control, and physical-target control attached. Evidence on compute01: `/home/claude/.cache/codex-scratch/j288/evidence/verify-fast-post-restart-36e7a63.log`, `/home/claude/.cache/codex-scratch/j288/evidence/baseline-rustc-paths-origin-dev-symlink.log`, and `/home/claude/.cache/codex-scratch/j288/evidence/baseline-rustc-paths-origin-dev.log`. The separate normal-MTU perf-VM comparison has not run. At the latest check, the perf lock was free but J317 had announced its corrected D348 comparison sequence; `perf.slice` was restored to `AllowedCPUs=EffectiveCPUs=1-3`. I did not acquire or queue a competing run, and changed no perf CPU masks. The existing J288 direction allows an owner-directed, uncalibrated matched hold after D348 readiness; this measurement remains pending that shared window.
kayg commented 2026-11-07 16:23:53 +00:00
Author
Owner
Copy link

D348 perf status follow-up: J288 has no run queued. A later nonblocking check found the shared perf lock held and perf.slice temporarily on CPUs 0–5. I have not touched the perf VM or its CPU masks. The separate normal-MTU comparison remains pending until this shared hold is released.

D348 perf status follow-up: J288 has no run queued. A later nonblocking check found the shared perf lock held and `perf.slice` temporarily on CPUs 0–5. I have not touched the perf VM or its CPU masks. The separate normal-MTU comparison remains pending until this shared hold is released.
kayg commented 2026-11-07 16:28:57 +00:00
Author
Owner
Copy link

J350 independent correctness review start. Reviewing mesh-mtu-381 at 36e7a6308f5561181b56cf44b67f479a33d6d0f6 against origin/dev 6b1bdb81b3d8eb90716643e7d1bd03f61ce37827; scratch branch review-j350. Plan (~2 hours plus shared-host queues): inspect MTU derivation/refresh and route ownership; rerun privileged route tests as root; merge dev in scratch and build/test mesh on Mac under lock with target cleanup; audit raw matrix and #421 red/control evidence. No merge to dev, deploy, or push. Per common brief reviewer rule, no full quick/fast rerun; inspect author gate and run focused checks. Evidence: ~/.cache/codex-scratch/j350/evidence/.

J350 independent correctness review start. Reviewing `mesh-mtu-381` at `36e7a6308f5561181b56cf44b67f479a33d6d0f6` against `origin/dev` `6b1bdb81b3d8eb90716643e7d1bd03f61ce37827`; scratch branch `review-j350`. Plan (~2 hours plus shared-host queues): inspect MTU derivation/refresh and route ownership; rerun privileged route tests as root; merge dev in scratch and build/test mesh on Mac under lock with target cleanup; audit raw matrix and #421 red/control evidence. No merge to dev, deploy, or push. Per common brief reviewer rule, no full quick/fast rerun; inspect author gate and run focused checks. Evidence: `~/.cache/codex-scratch/j350/evidence/`.
kayg commented 2026-11-07 16:42:50 +00:00
Author
Owner
Copy link

J350 progress: the scratch merge of reviewed head 36e7a6308 with dev 6b1bdb81b is 8c646d886713c21ef4461d472d82ea6b2468d27d; all mesh production files are unchanged by that merge. On Mac, under lockf -k /tmp/mac-bench.lock, 69 agent-feature mesh library tests pass and cargo build --locked -j3 --release -p infraternal passes. Binary SHA-256 30522c9ba591baa11836ac76db9f558c38cfd8364edf33c1f6a2104f88e061eb; target cleanup completed before unlock. Receipt: /Users/lume/.cache/codex-scratch/j350/evidence/mac-nix.log. The first plain-shell build stopped at a checksum-tool mismatch; the repository Nix shell resolves it.

The three requested route-retirement integration tests pass as uid 0 through cua + sudo -n, each in a fresh dev-02 network namespace. This includes foreign interface/protocol/prefix/table/local routes and gateway routes. Normalized-metrics tests pass (2/2), MTU unit tests pass (6/6). Receipts on control host: ~/.cache/codex-scratch/j350/evidence/root-routes.log and linux-unit.log.

I recalculated every matrix median from the Mac raw end.sum_received.bits_per_second values; J288's matrix matches, including the four explicit ERR cells. Current-dev 6b1bdb81b independently reproduces #421: symlink target exit 1, physical target exit 0, identical wrapper/remap/toolchain inputs. Receipts: matrix-audit.txt, path-controls.log, and the copied author fast log in the same evidence directory.

Remaining: focused adversarial tests for MTU refresh, egress-link upper bound, and relay framing. The first is at the front of dev-02's slot queue; the relay run is queued separately on the same single-slot host and same target. No live mesh interfaces or services have been changed. Verdict pending those results.

J350 progress: the scratch merge of reviewed head `36e7a6308` with dev `6b1bdb81b` is `8c646d886713c21ef4461d472d82ea6b2468d27d`; all mesh production files are unchanged by that merge. On Mac, under `lockf -k /tmp/mac-bench.lock`, 69 agent-feature mesh library tests pass and `cargo build --locked -j3 --release -p infraternal` passes. Binary SHA-256 `30522c9ba591baa11836ac76db9f558c38cfd8364edf33c1f6a2104f88e061eb`; target cleanup completed before unlock. Receipt: `/Users/lume/.cache/codex-scratch/j350/evidence/mac-nix.log`. The first plain-shell build stopped at a checksum-tool mismatch; the repository Nix shell resolves it. The three requested route-retirement integration tests pass as uid 0 through `cua` + `sudo -n`, each in a fresh dev-02 network namespace. This includes foreign interface/protocol/prefix/table/local routes and gateway routes. Normalized-metrics tests pass (2/2), MTU unit tests pass (6/6). Receipts on control host: `~/.cache/codex-scratch/j350/evidence/root-routes.log` and `linux-unit.log`. I recalculated every matrix median from the Mac raw `end.sum_received.bits_per_second` values; J288's matrix matches, including the four explicit ERR cells. Current-dev `6b1bdb81b` independently reproduces #421: symlink target exit 1, physical target exit 0, identical wrapper/remap/toolchain inputs. Receipts: `matrix-audit.txt`, `path-controls.log`, and the copied author fast log in the same evidence directory. Remaining: focused adversarial tests for MTU refresh, egress-link upper bound, and relay framing. The first is at the front of dev-02's slot queue; the relay run is queued separately on the same single-slot host and same target. No live mesh interfaces or services have been changed. Verdict pending those results.
kayg commented 2026-11-07 16:51:40 +00:00
Author
Owner
Copy link

J350 confirmed findings at reviewed head 36e7a6308 (production mesh code unchanged in scratch merge 8c646d886):

P2: refresh all affected route metrics, not just changed peer records (node.rs:1887-1919). Root reproducer: start with underlay MTU 1500, peer 2 route MTU 1420; narrow the underlay to 1280, then successfully apply a delta adding peer 3. Peer 3 receives MTU 1200, while unchanged peer 2 remains at 1420. The shared interface re-derivation does not refresh that peer's route. This can bring back oversized encapsulated traffic after a route change even when a later snapshot applies successfully. full_peer_update_refreshes_existing_peer_route_mtu fails as expected, exit 101.

P2: bound an explicit route MTU by the egress link MTU before subtracting overhead (wireguard.rs:1116-1119). Linux permits an explicit route metric above the output interface MTU. On a 1280-byte dummy link, an endpoint route MTU 9000 results in mesh route MTU 8920. endpoint_route_metric_is_bounded_by_egress_link_mtu fails as expected, exit 101. This requires local/stale route state, not a peer-supplied raw MTU; I am not claiming arbitrary remote control of the metric.

Both ran as uid 0 through cua + sudo -n, each inside its own network namespace, with cleanup before the assertions. Full receipt: control host /home/kayg/.cache/codex-scratch/j350/evidence/mtu-repro-root.log. Regression tests are committed on the review-only branch review-j350; no production code or dev branch changed. Relay framing proof is still pending its build slot. Final verdict will follow that run.

J350 confirmed findings at reviewed head `36e7a6308` (production mesh code unchanged in scratch merge `8c646d886`): **P2: refresh all affected route metrics, not just changed peer records** (`node.rs:1887-1919`). Root reproducer: start with underlay MTU 1500, peer 2 route MTU 1420; narrow the underlay to 1280, then successfully apply a delta adding peer 3. Peer 3 receives MTU 1200, while unchanged peer 2 remains at 1420. The shared interface re-derivation does not refresh that peer's route. This can bring back oversized encapsulated traffic after a route change even when a later snapshot applies successfully. `full_peer_update_refreshes_existing_peer_route_mtu` fails as expected, exit 101. **P2: bound an explicit route MTU by the egress link MTU before subtracting overhead** (`wireguard.rs:1116-1119`). Linux permits an explicit route metric above the output interface MTU. On a 1280-byte dummy link, an endpoint route MTU 9000 results in mesh route MTU 8920. `endpoint_route_metric_is_bounded_by_egress_link_mtu` fails as expected, exit 101. This requires local/stale route state, not a peer-supplied raw MTU; I am not claiming arbitrary remote control of the metric. Both ran as uid 0 through `cua` + `sudo -n`, each inside its own network namespace, with cleanup before the assertions. Full receipt: control host `/home/kayg/.cache/codex-scratch/j350/evidence/mtu-repro-root.log`. Regression tests are committed on the review-only branch `review-j350`; no production code or dev branch changed. Relay framing proof is still pending its build slot. Final verdict will follow that run.
kayg commented 2026-11-07 16:54:34 +00:00
Author
Owner
Copy link

J350 independent correctness review: REQUEST CHANGES.

Reviewed mesh-mtu-381 at 36e7a6308f5561181b56cf44b67f479a33d6d0f6 against origin/dev 6b1bdb81b3d8eb90716643e7d1bd03f61ce37827. Native checks use scratch merge 8c646d886713c21ef4461d472d82ea6b2468d27d; its production mesh files are byte-identical to the submitted head. No merge to dev, deployment, or production service changes.

Findings (no P3 findings):

  • P1: the new direct-path MTU budget breaks relay datagrams, including normal 1500-byte paths. mesh_mtu.rs:41-44,65-70 derives only the direct WireGuard budget. The relay handoff in node.rs:734 / gotatun_node.rs:661 updates the endpoint without refreshing the route, and relay_path.rs:500 drops SendDatagramError::TooLarge. The live QUIC regression test negotiated an actual datagram maximum of 1434 on a 1500-byte IPv4 path: a frame for the old 1380 inner budget (1416 bytes) is accepted, while the new 1420 budget (at least 1456 bytes before any padding) returns TooLarge. On a 1280 path, the maximum is 1214 and the new 1200 budget needs at least 1236, also TooLarge. This reproduces the packet-budget failure using the real relay/TLS/QUIC code; it is not a full mesh throughput rerun. QUIC framing and the relay's 4-byte link id need a separate budget based on the actual connection's max_datagram_size, including path changes. Artifact: relay.log; test narrow_path_mesh_budget_fits_relay_datagram fails as expected.
  • P2: successful full snapshot application leaves an unchanged peer with a stale route MTU. node.rs:1887-1919 derives the shared link budget from every peer but rewrites routes only for changed peers. An unchanged peer can retain its old budget even after a successful snapshot application observes the new path MTU. The endpoint-only path does rewrite all peer routes; the access-only fast path does not. There is no live route-change monitor, as disclosed in J288's initial plan. Root test: underlay 1500→1280, then add peer 3; existing peer 2 retains MTU 1420 while new peer 3 gets 1200. full_peer_update_refreshes_existing_peer_route_mtu fails, exit 101 (mtu-repro-root.log). Refresh every affected route when re-deriving, including unchanged peers; the interface floor alone cannot reserve encapsulation overhead.
  • P2: an explicit/stale route metric can exceed the actual egress link MTU. wireguard.rs:1116-1119 returns a route metric without bounding it by the output interface MTU. An explicit/stale route MTU can exceed the link MTU; subtracting 80 from that value is unsafe. Root test: endpoint route metric 9000 on a 1280-byte dummy link installs mesh route MTU 8920. endpoint_route_metric_is_bounded_by_egress_link_mtu fails, exit 101 (mtu-repro-root.log). This is a local/stale route-state case, not arbitrary remote MTU injection. Query both and use the smaller effective underlay ceiling before subtracting overhead.

Checks and evidence:

  • Subtracting 80 is conservative for IPv4 outer overhead 60 and correct for IPv6 overhead 80. IPv4 loses 20 bytes of possible budget. The shared link's 1280 floor preserves IPv6 addresses; per-peer route metrics may be smaller. Inputs are kernel/local route state, not a peer-supplied MTU field; the numeric conversion is bounded to 68..65535. That numerical bound does not establish that the value fits the actual path. No new authentication/secret-handling flaw found.
  • Route retirement: normalized outer RTA_METRICS is accepted with or without NLA_F_NESTED; exactly one nonnested four-byte RTAX_MTU is required. Extra/duplicate/malformed metrics remain foreign. Deletion also requires the recorded address inventory, matching interface index, main table 254, static protocol, host prefix, family-specific scope and no gateway/other modifiers. Three integration tests passed as uid 0 through cua + sudo -n, in separate dev-02 network namespaces: restart_retires_orphan_peer_host_routes, restart_preserves_operator_gateway_host_route, and orphan_route_retirement_preserves_foreign_routes. The latter preserves foreign interface/protocol/prefix/table/local routes. Artifact: root-routes.log. MTU unit tests 6/6 and metric-shape tests 2/2 pass (linux-unit.log).
  • macOS: under lockf -k /tmp/mac-bench.lock, CARGO_INCREMENTAL=0, the explicit agent-feature mesh library suite passed 69/69, and cargo build --locked -j3 --release -p infraternal passed. Binary SHA-256 30522c9ba591baa11836ac76db9f558c38cfd8364edf33c1f6a2104f88e061eb. The source hashes in mac-nix.log match both reviewed files exactly (macos_node.rs and mesh_mtu.rs). TARGET_CLEANED was emitted inside the lock and a subsequent check confirms no target remains. This closes the old SOCK_CLOEXEC build gap using current dev; the earlier matrix's Mac binary remains correctly identified as source 2171a4b.
  • Matrix: independently recalculated all medians from the Mac raw end.sum_received.bits_per_second values. J288's table matches, including Linux→Mac fixed mesh 78.024 / 259.003 Mbit/s, plain 105.804 / 235.303, and baseline mesh 0.435 at one flow. The two failed baseline eight-flow repetitions in each direction remain ERR, not throughput. Artifact: matrix-audit.txt; original raw data retained in mac-raw.tar, Linux report in matrix.json. This audits the recorded run, not an independent throughput rerun. Wide sample spread does not establish an eight-flow performance advantage over plain, and the normal-MTU perf comparison is still absent.
  • Boundary evidence: matrix.json confirms 1200-byte inner IPv4 probes pass 3/3 each way; Linux locally rejects 1201-byte DF (EMSGSIZE); Mac's 1201-byte DF passes. That supports “no black-holing at these tested boundaries.” It does not prove general PMTU correctness: with an IPv4 outer packet, 1201+60 still fits 1280, and neither relay framing, IPv6 outer paths, later MTU changes, nor remote ICMP/PTB behavior is established by those probes. No blanket no-black-hole conclusion is warranted.
  • #421: independently reproduced on current unmodified dev helper inputs: symlinked target fails at test-rustc-paths.py:28 (exit 1); physical target passes (exit 0). git diff --quiet confirms the helper, wrapper, remap flags and toolchain are identical between the submitted head and current dev. Artifact: path-controls.log. This is a baseline failure, not introduced by #381. The author's final fast log has exactly this one failed row; its static row stops before compiling fresh static artifacts, so it is not proof of new static artifact sizes.

The three failing regression tests are committed and pushed only to review-j350 at 38e3ddcd0e40348b505fc680ce13e572be72f22b (test source). The first two run as real root in disposable network namespaces; the third uses a real loopback relay connection with an explicit path payload limit, without a timing assertion. Both Linux and Mac job targets have been deleted; cleanup is recorded in cleanup.log and the locked Mac log.

All J350 artifacts named above are retained at /home/kayg/.cache/codex-scratch/j350/evidence/ on the control host; selected raw logs are attached. Mac original log: /Users/lume/.cache/codex-scratch/j350/evidence/mac-nix.log. Per the common brief's reviewer rule, I did not rerun quick/fast. The author's final verify fast summary at the submitted head, read from author-fast.log, is quoted below.

AssertionError
(static builds: 2s)

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| privileged integration    | PASS         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Verdict: REQUEST CHANGES. Reviewed head: 36e7a6308f5561181b56cf44b67f479a33d6d0f6.

J350 independent correctness review: **REQUEST CHANGES**. Reviewed `mesh-mtu-381` at `36e7a6308f5561181b56cf44b67f479a33d6d0f6` against `origin/dev` `6b1bdb81b3d8eb90716643e7d1bd03f61ce37827`. Native checks use scratch merge `8c646d886713c21ef4461d472d82ea6b2468d27d`; its production mesh files are byte-identical to the submitted head. No merge to dev, deployment, or production service changes. Findings (no P3 findings): - **P1: the new direct-path MTU budget breaks relay datagrams, including normal 1500-byte paths.** `mesh_mtu.rs:41-44,65-70` derives only the direct WireGuard budget. The relay handoff in `node.rs:734` / `gotatun_node.rs:661` updates the endpoint without refreshing the route, and `relay_path.rs:500` drops `SendDatagramError::TooLarge`. The live QUIC regression test negotiated an actual datagram maximum of 1434 on a 1500-byte IPv4 path: a frame for the old 1380 inner budget (1416 bytes) is accepted, while the new 1420 budget (at least 1456 bytes before any padding) returns `TooLarge`. On a 1280 path, the maximum is 1214 and the new 1200 budget needs at least 1236, also `TooLarge`. This reproduces the packet-budget failure using the real relay/TLS/QUIC code; it is not a full mesh throughput rerun. QUIC framing and the relay's 4-byte link id need a separate budget based on the actual connection's `max_datagram_size`, including path changes. Artifact: `relay.log`; test `narrow_path_mesh_budget_fits_relay_datagram` fails as expected. - **P2: successful full snapshot application leaves an unchanged peer with a stale route MTU.** `node.rs:1887-1919` derives the shared link budget from every peer but rewrites routes only for `changed` peers. An unchanged peer can retain its old budget even after a successful snapshot application observes the new path MTU. The endpoint-only path does rewrite all peer routes; the access-only fast path does not. There is no live route-change monitor, as disclosed in J288's initial plan. Root test: underlay 1500→1280, then add peer 3; existing peer 2 retains MTU 1420 while new peer 3 gets 1200. `full_peer_update_refreshes_existing_peer_route_mtu` fails, exit 101 (`mtu-repro-root.log`). Refresh every affected route when re-deriving, including unchanged peers; the interface floor alone cannot reserve encapsulation overhead. - **P2: an explicit/stale route metric can exceed the actual egress link MTU.** `wireguard.rs:1116-1119` returns a route metric without bounding it by the output interface MTU. An explicit/stale route MTU can exceed the link MTU; subtracting 80 from that value is unsafe. Root test: endpoint route metric 9000 on a 1280-byte dummy link installs mesh route MTU 8920. `endpoint_route_metric_is_bounded_by_egress_link_mtu` fails, exit 101 (`mtu-repro-root.log`). This is a local/stale route-state case, not arbitrary remote MTU injection. Query both and use the smaller effective underlay ceiling before subtracting overhead. Checks and evidence: - Subtracting 80 is conservative for IPv4 outer overhead 60 and correct for IPv6 overhead 80. IPv4 loses 20 bytes of possible budget. The shared link's 1280 floor preserves IPv6 addresses; per-peer route metrics may be smaller. Inputs are kernel/local route state, not a peer-supplied MTU field; the numeric conversion is bounded to 68..65535. That numerical bound does not establish that the value fits the actual path. No new authentication/secret-handling flaw found. - Route retirement: normalized outer `RTA_METRICS` is accepted with or without `NLA_F_NESTED`; exactly one nonnested four-byte `RTAX_MTU` is required. Extra/duplicate/malformed metrics remain foreign. Deletion also requires the recorded address inventory, matching interface index, main table 254, static protocol, host prefix, family-specific scope and no gateway/other modifiers. Three integration tests passed as uid 0 through `cua` + `sudo -n`, in separate dev-02 network namespaces: `restart_retires_orphan_peer_host_routes`, `restart_preserves_operator_gateway_host_route`, and `orphan_route_retirement_preserves_foreign_routes`. The latter preserves foreign interface/protocol/prefix/table/local routes. Artifact: `root-routes.log`. MTU unit tests 6/6 and metric-shape tests 2/2 pass (`linux-unit.log`). - macOS: under `lockf -k /tmp/mac-bench.lock`, `CARGO_INCREMENTAL=0`, the explicit agent-feature mesh library suite passed 69/69, and `cargo build --locked -j3 --release -p infraternal` passed. Binary SHA-256 `30522c9ba591baa11836ac76db9f558c38cfd8364edf33c1f6a2104f88e061eb`. The source hashes in `mac-nix.log` match both reviewed files exactly (`macos_node.rs` and `mesh_mtu.rs`). `TARGET_CLEANED` was emitted inside the lock and a subsequent check confirms no target remains. This closes the old SOCK_CLOEXEC build gap using current dev; the earlier matrix's Mac binary remains correctly identified as source `2171a4b`. - Matrix: independently recalculated all medians from the Mac raw `end.sum_received.bits_per_second` values. J288's table matches, including Linux→Mac fixed mesh 78.024 / 259.003 Mbit/s, plain 105.804 / 235.303, and baseline mesh 0.435 at one flow. The two failed baseline eight-flow repetitions in each direction remain ERR, not throughput. Artifact: `matrix-audit.txt`; original raw data retained in `mac-raw.tar`, Linux report in `matrix.json`. This audits the recorded run, not an independent throughput rerun. Wide sample spread does not establish an eight-flow performance advantage over plain, and the normal-MTU perf comparison is still absent. - Boundary evidence: `matrix.json` confirms 1200-byte inner IPv4 probes pass 3/3 each way; Linux locally rejects 1201-byte DF (`EMSGSIZE`); Mac's 1201-byte DF passes. That supports “no black-holing at these tested boundaries.” It does not prove general PMTU correctness: with an IPv4 outer packet, 1201+60 still fits 1280, and neither relay framing, IPv6 outer paths, later MTU changes, nor remote ICMP/PTB behavior is established by those probes. No blanket no-black-hole conclusion is warranted. - #421: independently reproduced on current unmodified dev helper inputs: symlinked target fails at `test-rustc-paths.py:28` (exit 1); physical target passes (exit 0). `git diff --quiet` confirms the helper, wrapper, remap flags and toolchain are identical between the submitted head and current dev. Artifact: `path-controls.log`. This is a baseline failure, not introduced by #381. The author's final fast log has exactly this one failed row; its static row stops before compiling fresh static artifacts, so it is not proof of new static artifact sizes. The three failing regression tests are committed and pushed only to `review-j350` at `38e3ddcd0e40348b505fc680ce13e572be72f22b` ([test source](https://git.kayg.org/kayg/infraternal/src/branch/review-j350/crates/plugins/mesh/tests/review_j350_mtu.rs)). The first two run as real root in disposable network namespaces; the third uses a real loopback relay connection with an explicit path payload limit, without a timing assertion. Both Linux and Mac job targets have been deleted; cleanup is recorded in `cleanup.log` and the locked Mac log. All J350 artifacts named above are retained at `/home/kayg/.cache/codex-scratch/j350/evidence/` on the control host; selected raw logs are attached. Mac original log: `/Users/lume/.cache/codex-scratch/j350/evidence/mac-nix.log`. Per the common brief's reviewer rule, I did not rerun quick/fast. The author's final `verify fast` summary at the submitted head, read from `author-fast.log`, is quoted below. ```text AssertionError (static builds: 2s) +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | Nix Cargo vendoring | PASS | | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | PASS | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` Verdict: **REQUEST CHANGES**. Reviewed head: `36e7a6308f5561181b56cf44b67f479a33d6d0f6`.
root-routes.log
722 B
mtu-repro-root.log
1.7 KiB
relay.log
2.3 KiB
mac-nix.log
25 KiB
path-controls.log
514 B
matrix-audit.txt
1.4 KiB
cleanup.log
36 B
j350-regressions.patch
7.9 KiB
author-fast.log
813 KiB
kayg commented 2026-11-07 17:18:20 +00:00
Author
Owner
Copy link

J288 design update before implementation (J350 review fixes)

The direct-path decision remains route-derived MTU. I will make the route budget depend on the active path:

  • Direct WireGuard: min(endpoint route MTU, egress link MTU) - 80. The 80-byte reserve covers the IPv6 outer header, UDP, and WireGuard; IPv4 is conservatively smaller.
  • Relay: derive from the active connection's live max_datagram_size(). Quinn 0.11.12 defines that value as the maximum application payload passed to send_datagram() and already reserves QUIC packet and DATAGRAM framing overhead, so that framing is counted exactly once. Subtract the 32-byte WireGuard packet overhead and the relay's 4-byte link ID. The observed 1434/1214-byte limits therefore allow 1398/1178-byte inner packets. The focused test will send at that derived boundary through a real QUIC relay connection.

I will recalculate the peer route when its active path changes, recheck the live QUIC limit while the relay connection remains active, and update the route as soon as the limit changes (with an immediate refresh on TooLarge). A full peer snapshot will refresh every installed peer route, even when a peer's WireGuard record is unchanged; active relay budgets remain the route source for relayed peers. The shared TUN/utun keeps its IPv6-safe 1280-byte floor while peer host routes carry smaller path budgets. macOS will use the same budget helpers and its native route updater.

Trade-offs: route metrics may be below IPv6's 1280-byte link minimum, but packets are constrained per peer while the shared interface retains the required floor. The direct 80-byte reserve gives up up to 20 bytes on IPv4 for one cross-platform rule. Polling live QUIC limits adds a small periodic check per relay connection, while avoiding a new ICMP parser or a TCP-only MSS clamp. relay_path is currently Linux-gated; I will verify the existing Mac surface as part of the native build and record any limitation on the forced-relay Mac leg rather than claiming it was measured.

**J288 design update before implementation (J350 review fixes)** The direct-path decision remains route-derived MTU. I will make the route budget depend on the active path: - Direct WireGuard: `min(endpoint route MTU, egress link MTU) - 80`. The 80-byte reserve covers the IPv6 outer header, UDP, and WireGuard; IPv4 is conservatively smaller. - Relay: derive from the active connection's live `max_datagram_size()`. Quinn 0.11.12 defines that value as the maximum application payload passed to `send_datagram()` and already reserves QUIC packet and DATAGRAM framing overhead, so that framing is counted exactly once. Subtract the 32-byte WireGuard packet overhead and the relay's 4-byte link ID. The observed 1434/1214-byte limits therefore allow 1398/1178-byte inner packets. The focused test will send at that derived boundary through a real QUIC relay connection. I will recalculate the peer route when its active path changes, recheck the live QUIC limit while the relay connection remains active, and update the route as soon as the limit changes (with an immediate refresh on `TooLarge`). A full peer snapshot will refresh every installed peer route, even when a peer's WireGuard record is unchanged; active relay budgets remain the route source for relayed peers. The shared TUN/utun keeps its IPv6-safe 1280-byte floor while peer host routes carry smaller path budgets. macOS will use the same budget helpers and its native route updater. Trade-offs: route metrics may be below IPv6's 1280-byte link minimum, but packets are constrained per peer while the shared interface retains the required floor. The direct 80-byte reserve gives up up to 20 bytes on IPv4 for one cross-platform rule. Polling live QUIC limits adds a small periodic check per relay connection, while avoiding a new ICMP parser or a TCP-only MSS clamp. `relay_path` is currently Linux-gated; I will verify the existing Mac surface as part of the native build and record any limitation on the forced-relay Mac leg rather than claiming it was measured.
kayg commented 2026-11-07 17:36:42 +00:00
Author
Owner
Copy link

J350 relay regression test is red on macOS at merged head 2a621416d (before MTU implementation changes). Under lockf -k /tmp/mac-bench.lock, CARGO_INCREMENTAL=0, the real loopback QUIC relay reported max_datagram_size=Some(1214) on a 1280 path and Some(1434) on a 1500 path. The current 1200/1420 inner budgets require 1236/1456-byte relay payloads and both send_datagram calls return TooLarge; the legacy 1380 packet still succeeds at 1500. Expected assertion failure: narrow_path_mesh_budget_fits_relay_datagram. Artifact: /Users/claude/.cache/codex-scratch/j288/evidence/mac-j350-red-r2.log. The target was deleted before the lock was released (TARGET_CLEANED). The first Mac invocation stopped at compile time because it omitted the required test-support feature; that runner error is retained as mac-j350-red.log and is superseded by the valid red run above.

J350 relay regression test is red on macOS at merged head `2a621416d` (before MTU implementation changes). Under `lockf -k /tmp/mac-bench.lock`, `CARGO_INCREMENTAL=0`, the real loopback QUIC relay reported `max_datagram_size=Some(1214)` on a 1280 path and `Some(1434)` on a 1500 path. The current 1200/1420 inner budgets require 1236/1456-byte relay payloads and both `send_datagram` calls return `TooLarge`; the legacy 1380 packet still succeeds at 1500. Expected assertion failure: `narrow_path_mesh_budget_fits_relay_datagram`. Artifact: `/Users/claude/.cache/codex-scratch/j288/evidence/mac-j350-red-r2.log`. The target was deleted before the lock was released (`TARGET_CLEANED`). The first Mac invocation stopped at compile time because it omitted the required `test-support` feature; that runner error is retained as `mac-j350-red.log` and is superseded by the valid red run above.
kayg commented 2026-11-07 17:56:36 +00:00
Author
Owner
Copy link

J288 resumed implementation on branch mesh-mtu-381 at merge base 2a621416d (36e7a6308 plus dev 6b1bdb81b). The agreed design and Mac red evidence are above. I confirmed the Linux route refresh, egress-link bound, and relay connection data flow. The focused Linux relay regression is queued on dev-02 with an explicit workspace manifest after an initial command-directory setup error; no implementation changes have been made yet.

J288 resumed implementation on branch mesh-mtu-381 at merge base 2a621416d (36e7a6308 plus dev 6b1bdb81b). The agreed design and Mac red evidence are above. I confirmed the Linux route refresh, egress-link bound, and relay connection data flow. The focused Linux relay regression is queued on dev-02 with an explicit workspace manifest after an initial command-directory setup error; no implementation changes have been made yet.
kayg commented 2026-11-07 17:59:21 +00:00
Author
Owner
Copy link

J288 red evidence: narrow_path_mesh_budget_fits_relay_datagram fails on dev-02 before implementation, in a real QUIC relay connection. At a 1280 underlay the current 1200-byte inner route requires 1236 bytes while Quinn permits 1214 (TooLarge); at 1500 the 1420-byte route requires 1456 while Quinn permits 1434 (TooLarge). The legacy 1380-byte packet fits only at 1500. Raw output: /home/kayg/.cache/codex-scratch/j288/evidence/linux-relay-red-r3.log (SHA-256 349b22e042685a47977e3358e9f421c60da0c252cd4d3262b52185a1115924dd). The Mac red run at the same merged head is recorded in the preceding comment.

J288 red evidence: `narrow_path_mesh_budget_fits_relay_datagram` fails on dev-02 before implementation, in a real QUIC relay connection. At a 1280 underlay the current 1200-byte inner route requires 1236 bytes while Quinn permits 1214 (`TooLarge`); at 1500 the 1420-byte route requires 1456 while Quinn permits 1434 (`TooLarge`). The legacy 1380-byte packet fits only at 1500. Raw output: `/home/kayg/.cache/codex-scratch/j288/evidence/linux-relay-red-r3.log` (SHA-256 `349b22e042685a47977e3358e9f421c60da0c252cd4d3262b52185a1115924dd`). The Mac red run at the same merged head is recorded in the preceding comment.
kayg commented 2026-11-07 18:03:17 +00:00
Author
Owner
Copy link

The other two J350 regressions are red on dev-02 inside separate disposable user/network namespaces. endpoint_route_metric_is_bounded_by_egress_link_mtu sees underlay link MTU 1280 and explicit route MTU 9000, then writes peer route MTU 8920. full_peer_update_refreshes_existing_peer_route_mtu sees the unchanged peer remain at 1420 after the underlay is narrowed while the new peer gets 1200. Raw output: /home/kayg/.cache/codex-scratch/j288/evidence/linux-route-red-r1.log (SHA-256 dd74606ac9ecccfa05525772e61fe937a43f3ed754dd742355c189c61eaf5bb7). No production code has changed.

The other two J350 regressions are red on dev-02 inside separate disposable user/network namespaces. `endpoint_route_metric_is_bounded_by_egress_link_mtu` sees underlay link MTU 1280 and explicit route MTU 9000, then writes peer route MTU 8920. `full_peer_update_refreshes_existing_peer_route_mtu` sees the unchanged peer remain at 1420 after the underlay is narrowed while the new peer gets 1200. Raw output: `/home/kayg/.cache/codex-scratch/j288/evidence/linux-route-red-r1.log` (SHA-256 `dd74606ac9ecccfa05525772e61fe937a43f3ed754dd742355c189c61eaf5bb7`). No production code has changed.
kayg commented 2026-11-07 18:12:06 +00:00
Author
Owner
Copy link

The new shared-helper test is also red before its implementation: it reports cannot find function for_relay_datagram for the 1214-byte, 1434-byte, and unavailable-limit cases. Output is retained at /home/kayg/.cache/codex-scratch/j288/evidence/linux-relay-helper-red.log (SHA-256 4dbb85420be08643f809699039cf868aa716089ece421af68d053cdec440a821).

The new shared-helper test is also red before its implementation: it reports `cannot find function for_relay_datagram` for the 1214-byte, 1434-byte, and unavailable-limit cases. Output is retained at `/home/kayg/.cache/codex-scratch/j288/evidence/linux-relay-helper-red.log` (SHA-256 `4dbb85420be08643f809699039cf868aa716089ece421af68d053cdec440a821`).
kayg commented 2026-11-07 18:58:39 +00:00
Author
Owner
Copy link

J288 continuation on mesh-mtu-381 at merge base 2a621416d (the review head 36e7a6308 plus dev 6b1bdb81b). The three J350 regressions are red before the pending code edits: the real QUIC relay rejects 1200/1420-byte inner budgets at max payloads 1214/1434 (linux-relay-red-r3.log, mac-j350-relay-red-current.log); Linux writes route MTU 8920 over a 1280 link and leaves an unchanged peer at 1420 after narrowing (linux-route-red-r1.log); the Mac route parser also accepts metric 9000 over link MTU 1280 (mac-j350-egress-red-current.log). Evidence is under /home/kayg/.cache/codex-scratch/j288/evidence/.

Implementation plan: keep the direct route-derived budget; clamp route metrics by egress MTU; refresh all peer routes on full snapshots; apply an active relay override based on each live Quinn connection's max_datagram_size() and update it at relay transitions and when the connection limit changes. Quinn documents this as the maximum application payload passed to send_datagram, already accounting for QUIC packet and DATAGRAM framing, so the safe route budget subtracts WireGuard's 32 bytes and the relay's 4-byte id once (1398/1178 at the observed limits). The current macOS updater already compares route MTUs for every retained peer on each config apply; I will preserve that and verify it on the Mac. Production relay_path remains Linux-gated, so its live route refresh will be exercised on compute01; the QUIC boundary test itself runs on both OSes.

J288 continuation on `mesh-mtu-381` at merge base `2a621416d` (the review head `36e7a6308` plus dev `6b1bdb81b`). The three J350 regressions are red before the pending code edits: the real QUIC relay rejects 1200/1420-byte inner budgets at max payloads 1214/1434 (`linux-relay-red-r3.log`, `mac-j350-relay-red-current.log`); Linux writes route MTU 8920 over a 1280 link and leaves an unchanged peer at 1420 after narrowing (`linux-route-red-r1.log`); the Mac route parser also accepts metric 9000 over link MTU 1280 (`mac-j350-egress-red-current.log`). Evidence is under `/home/kayg/.cache/codex-scratch/j288/evidence/`. Implementation plan: keep the direct route-derived budget; clamp route metrics by egress MTU; refresh all peer routes on full snapshots; apply an active relay override based on each live Quinn connection's `max_datagram_size()` and update it at relay transitions and when the connection limit changes. Quinn documents this as the maximum application payload passed to `send_datagram`, already accounting for QUIC packet and DATAGRAM framing, so the safe route budget subtracts WireGuard's 32 bytes and the relay's 4-byte id once (1398/1178 at the observed limits). The current macOS updater already compares route MTUs for every retained peer on each config apply; I will preserve that and verify it on the Mac. Production `relay_path` remains Linux-gated, so its live route refresh will be exercised on compute01; the QUIC boundary test itself runs on both OSes.
kayg commented 2026-11-07 19:58:47 +00:00
Author
Owner
Copy link

J288 review-fix progress: branch mesh-mtu-381 remains at review base 2a621416d (merged dev through 6b1bdb81b). The red evidence for relay sizing, route metric bounding and unchanged-peer refresh is retained under /home/kayg/.cache/codex-scratch/j288/evidence/. The first Linux green attempt compiled the test and found E0500 in endpoint-only refresh: the closure borrowed self for the engine update and then called a route-refresh method on self while the old config borrow remained live. I changed that path to capture the shared relay override state and peer snapshot before entering the closure. No green test has run yet; focused Linux verification is next.

J288 review-fix progress: branch mesh-mtu-381 remains at review base 2a621416d (merged dev through 6b1bdb81b). The red evidence for relay sizing, route metric bounding and unchanged-peer refresh is retained under /home/kayg/.cache/codex-scratch/j288/evidence/. The first Linux green attempt compiled the test and found E0500 in endpoint-only refresh: the closure borrowed self for the engine update and then called a route-refresh method on self while the old config borrow remained live. I changed that path to capture the shared relay override state and peer snapshot before entering the closure. No green test has run yet; focused Linux verification is next.
kayg commented 2026-11-07 20:12:30 +00:00
Author
Owner
Copy link

J288 J350 relay regression green on Linux (dev-02 aarch64). The real Quinn connection advertised max_datagram_size 1214 at a 1280-byte path and 1434 at 1500. Budgets 1178/1398 plus 32-byte WireGuard overhead and the 4-byte link id sent successfully; the current 1200/1420 budgets returned TooLarge. Shipped 1380 still fits the 1500 relay path and fails only on the narrow path. Test: narrow_path_mesh_budget_fits_relay_datagram, 1 passed. Full receipt: /home/kayg/.cache/codex-scratch/j288/evidence/linux-relay-green-dev02.log. The unrelated launcher attempt before this run exited before Cargo because it did not cd into the worktree; no test was executed in that attempt.

J288 J350 relay regression green on Linux (dev-02 aarch64). The real Quinn connection advertised max_datagram_size 1214 at a 1280-byte path and 1434 at 1500. Budgets 1178/1398 plus 32-byte WireGuard overhead and the 4-byte link id sent successfully; the current 1200/1420 budgets returned TooLarge. Shipped 1380 still fits the 1500 relay path and fails only on the narrow path. Test: narrow_path_mesh_budget_fits_relay_datagram, 1 passed. Full receipt: /home/kayg/.cache/codex-scratch/j288/evidence/linux-relay-green-dev02.log. The unrelated launcher attempt before this run exited before Cargo because it did not cd into the worktree; no test was executed in that attempt.
kayg commented 2026-11-07 20:15:33 +00:00
Author
Owner
Copy link

J288 Linux library regressions green on dev-02 (aarch64): 152 passed, 23 ignored, 0 failed. This includes relay_route_budget_tracks_live_datagram_limit_changes and wireguard route-metric ownership coverage. Full receipt: /home/kayg/.cache/codex-scratch/j288/evidence/linux-mesh-lib-green-dev02.log. The two new route-level J350 tests still need isolated root namespaces.

J288 Linux library regressions green on dev-02 (aarch64): 152 passed, 23 ignored, 0 failed. This includes relay_route_budget_tracks_live_datagram_limit_changes and wireguard route-metric ownership coverage. Full receipt: /home/kayg/.cache/codex-scratch/j288/evidence/linux-mesh-lib-green-dev02.log. The two new route-level J350 tests still need isolated root namespaces.
kayg commented 2026-11-07 20:22:12 +00:00
Author
Owner
Copy link

J288 Linux route regressions green in separate disposable network namespaces on compute01. endpoint_route_metric_is_bounded_by_egress_link_mtu set a 1280 egress link with an explicit 9000 route metric and observed mesh route MTU 1200. full_peer_update_refreshes_existing_peer_route_mtu observed the retained peer route change 1420 → 1200 after the underlay narrowed, while the new peer also received 1200. Receipts: /home/kayg/.cache/codex-scratch/j288/evidence/linux-egress-mtu-green-compute01.log and /home/kayg/.cache/codex-scratch/j288/evidence/linux-full-peer-refresh-green-compute01.log.

J288 Linux route regressions green in separate disposable network namespaces on compute01. endpoint_route_metric_is_bounded_by_egress_link_mtu set a 1280 egress link with an explicit 9000 route metric and observed mesh route MTU 1200. full_peer_update_refreshes_existing_peer_route_mtu observed the retained peer route change 1420 → 1200 after the underlay narrowed, while the new peer also received 1200. Receipts: /home/kayg/.cache/codex-scratch/j288/evidence/linux-egress-mtu-green-compute01.log and /home/kayg/.cache/codex-scratch/j288/evidence/linux-full-peer-refresh-green-compute01.log.
kayg commented 2026-11-07 20:26:30 +00:00
Author
Owner
Copy link

J288 committed and pushed the focused fix as 2647e8fdd51af9d35986c36ca3e57305bc9a5fa2. Green evidence now covers the real Quinn relay boundary on dev-02, the 152-test Linux mesh library suite, and both root route regressions in isolated compute01 network namespaces. I am merging the current origin/dev head next, then I will run the requested quick gate and build/test the Mac at the exact merged head under its lock.

J288 committed and pushed the focused fix as 2647e8fdd51af9d35986c36ca3e57305bc9a5fa2. Green evidence now covers the real Quinn relay boundary on dev-02, the 152-test Linux mesh library suite, and both root route regressions in isolated compute01 network namespaces. I am merging the current origin/dev head next, then I will run the requested quick gate and build/test the Mac at the exact merged head under its lock.
kayg commented 2026-11-07 20:27:42 +00:00
Author
Owner
Copy link

J288 merged current origin/dev 6ffe37e067 into mesh-mtu-381. Exact merged head is 1974aee0ff, and it is pushed. Linux red-to-green evidence is recorded above; verify quick is next on dev-02, followed by Mac build/tests at this exact head under the Mac lock.

J288 merged current origin/dev 6ffe37e067d8fa7b090ea83c21b84b77937f76d9 into mesh-mtu-381. Exact merged head is 1974aee0ffc9ee771ebeb8deb2ab42c6b57c5426, and it is pushed. Linux red-to-green evidence is recorded above; verify quick is next on dev-02, followed by Mac build/tests at this exact head under the Mac lock.
kayg commented 2026-11-07 20:29:13 +00:00
Author
Owner
Copy link

J288 verify quick on merged head 1974aee0f stopped at fmt only. Rustfmt reported layout changes in the six changed mesh files and the new regression test; scope, migrations, and Nix vendoring passed, and clippy/tests did not run. Full pre-check receipt: /home/kayg/.cache/codex-scratch/j288/evidence/verify-quick-1974aee0f.log. I am applying rustfmt in the dev-02 worktree, then will commit the formatting-only follow-up and rerun quick.

J288 verify quick on merged head 1974aee0f stopped at fmt only. Rustfmt reported layout changes in the six changed mesh files and the new regression test; scope, migrations, and Nix vendoring passed, and clippy/tests did not run. Full pre-check receipt: /home/kayg/.cache/codex-scratch/j288/evidence/verify-quick-1974aee0f.log. I am applying rustfmt in the dev-02 worktree, then will commit the formatting-only follow-up and rerun quick.
kayg commented 2026-11-07 20:31:49 +00:00
Author
Owner
Copy link

J288 formatting-only follow-up 955487b09 is pushed; it contains only rustfmt output from the first quick attempt. The Mac build and tests will use this exact pushed head. I am checking the existing Mac worktree and target state before taking the lock.

J288 formatting-only follow-up 955487b09 is pushed; it contains only rustfmt output from the first quick attempt. The Mac build and tests will use this exact pushed head. I am checking the existing Mac worktree and target state before taking the lock.
kayg commented 2026-11-07 21:15:30 +00:00
Author
Owner
Copy link

Mac final-head validation (2026-11-07): built branch head 955487b099443ae40051e351251dfbbb879ba1be under /tmp/mac-bench.lock, with CARGO_INCREMENTAL=0, and removed the 2.9G target before unlocking. The Mac mesh library suite passed 120/120, including macos_node::tests::full_peer_update_refreshes_existing_peer_route_mtu; narrow_path_mesh_budget_fits_relay_datagram passed and sent 1178 bytes at a 1214-byte QUIC application limit and 1398 bytes at a 1434-byte limit. The previous 1200/1420 budgets returned TooLarge. Release build passed. Receipts: /home/kayg/.cache/codex-scratch/j288/evidence/mac-final-build-test-955487b09.log, /home/kayg/.cache/codex-scratch/j288/evidence/mac-binary-955487b09.sha256.

Mac relay scope: relay_path is compiled only for Linux in lib.rs, so the Mac↔compute01 matrix exercises the Mac direct route path. I will run the forced relay before/after on Linux with the production GotaTun relay fixture and report that separately, rather than claim a Mac relay path that does not exist in this branch.

Mac final-head validation (2026-11-07): built branch head `955487b099443ae40051e351251dfbbb879ba1be` under `/tmp/mac-bench.lock`, with `CARGO_INCREMENTAL=0`, and removed the 2.9G target before unlocking. The Mac mesh library suite passed 120/120, including `macos_node::tests::full_peer_update_refreshes_existing_peer_route_mtu`; `narrow_path_mesh_budget_fits_relay_datagram` passed and sent 1178 bytes at a 1214-byte QUIC application limit and 1398 bytes at a 1434-byte limit. The previous 1200/1420 budgets returned `TooLarge`. Release build passed. Receipts: `/home/kayg/.cache/codex-scratch/j288/evidence/mac-final-build-test-955487b09.log`, `/home/kayg/.cache/codex-scratch/j288/evidence/mac-binary-955487b09.sha256`. Mac relay scope: `relay_path` is compiled only for Linux in `lib.rs`, so the Mac↔compute01 matrix exercises the Mac direct route path. I will run the forced relay before/after on Linux with the production GotaTun relay fixture and report that separately, rather than claim a Mac relay path that does not exist in this branch.
kayg commented 2026-11-07 21:18:31 +00:00
Author
Owner
Copy link

J288 verify quick is submitted for 955487b099443ae40051e351251dfbbb879ba1be on dev-02 after nix develop . -c cargo --version passed. At 2026-11-07T22:xxZ, build-slot showed the J288 run at queue position 4 behind three existing jobs; the active slot had been running for 46 minutes. The command is foreground and writes to /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-955487b09-dev02.log. No additional Linux slot run is started while it waits.

J288 verify quick is submitted for `955487b099443ae40051e351251dfbbb879ba1be` on dev-02 after `nix develop . -c cargo --version` passed. At 2026-11-07T22:xxZ, build-slot showed the J288 run at queue position 4 behind three existing jobs; the active slot had been running for 46 minutes. The command is foreground and writes to `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-955487b09-dev02.log`. No additional Linux slot run is started while it waits.
kayg commented 2026-11-07 21:18:52 +00:00
Author
Owner
Copy link

Correction to my previous queue note: its timestamp was a placeholder. The observed queue snapshot was at 02:48 IST on 2026-11-08. The queue position and run state are unchanged.

Correction to my previous queue note: its timestamp was a placeholder. The observed queue snapshot was at 02:48 IST on 2026-11-08. The queue position and run state are unchanged.
kayg commented 2026-11-07 21:37:10 +00:00
Author
Owner
Copy link

J288 queue update at 2026-11-08 03:07 IST: the dev-02 slot has moved from the fuzz job to J205 verify quick; my verify quick is now the only queued job behind it (PID 1505073). The run remains foreground and writes to /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-955487b09-dev02.log. I will start the compute01 build and matrix after this queued job exits, then run verify fast on compute01.

J288 queue update at 2026-11-08 03:07 IST: the dev-02 slot has moved from the fuzz job to J205 verify quick; my `verify quick` is now the only queued job behind it (PID 1505073). The run remains foreground and writes to `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-955487b09-dev02.log`. I will start the compute01 build and matrix after this queued job exits, then run verify fast on compute01.
kayg commented 2026-11-07 21:38:58 +00:00
Author
Owner
Copy link

J288 verify quick on 955487b09 acquired dev-02 at 21:36:50Z and exited with VERIFY_QUICK_EXIT=1. Scope, migration names, Nix vendoring, fmt, doc coverage, shape lints, and fixed waits passed; D345 doc comments failed for node.rs:866 retire_routes, relay_path.rs:203 Peer, relay_path.rs:882 serve, and relay_path.rs:1036 a_pair_is_relayed_only_without_a_handshake_since_its_endpoint. Full log: /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-955487b09-dev02.log. Root cause is missing documentation comments on items touched in the branch diff. I have added those four comments locally and will rerun quick before committing.

J288 verify quick on 955487b09 acquired dev-02 at 21:36:50Z and exited with `VERIFY_QUICK_EXIT=1`. Scope, migration names, Nix vendoring, fmt, doc coverage, shape lints, and fixed waits passed; D345 doc comments failed for `node.rs:866 retire_routes`, `relay_path.rs:203 Peer`, `relay_path.rs:882 serve`, and `relay_path.rs:1036 a_pair_is_relayed_only_without_a_handshake_since_its_endpoint`. Full log: `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-955487b09-dev02.log`. Root cause is missing documentation comments on items touched in the branch diff. I have added those four comments locally and will rerun quick before committing.
kayg commented 2026-11-07 21:42:50 +00:00
Author
Owner
Copy link

J288 quick retry is queued at position 3 on dev-02 as of 2026-11-08 03:12 IST (PID 1603337), behind one active quick gate and one older queued run. The retry uses the four D345 comments synced into ~/worktrees/j288-381 and writes to /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-docs-955487b09-dev02.log. I am keeping the foreground waiter attached.

J288 quick retry is queued at position 3 on dev-02 as of 2026-11-08 03:12 IST (PID 1603337), behind one active quick gate and one older queued run. The retry uses the four D345 comments synced into `~/worktrees/j288-381` and writes to `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-docs-955487b09-dev02.log`. I am keeping the foreground waiter attached.
kayg commented 2026-11-07 21:51:35 +00:00
Author
Owner
Copy link

J288 quick fix retry is queued at position 2 (one job ahead) on dev-02 as of 2026-11-08 03:21 IST, PID 1655092. The D345 comments and relay-only feature gates are synced; preflight nix develop . -c cargo --version and git diff --check passed. The foreground log is /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-fix1-955487b09-dev02.log.

J288 quick fix retry is queued at position 2 (one job ahead) on dev-02 as of 2026-11-08 03:21 IST, PID 1655092. The D345 comments and relay-only feature gates are synced; preflight `nix develop . -c cargo --version` and `git diff --check` passed. The foreground log is `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-fix1-955487b09-dev02.log`.
kayg commented 2026-11-07 21:53:12 +00:00
Author
Owner
Copy link

J288 verify quick retry 2 acquired dev-02 at 21:52:33Z and stopped at fmt with one rustfmt line-wrap difference in the new for_relay_datagram map_or closure. Scope guard, migration names, and Nix vendoring passed; clippy/tests did not run. Full log: /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-fix1-955487b09-dev02.log. I matched rustfmt output; the slot is free for the next retry.

J288 verify quick retry 2 acquired dev-02 at 21:52:33Z and stopped at fmt with one rustfmt line-wrap difference in the new `for_relay_datagram` `map_or` closure. Scope guard, migration names, and Nix vendoring passed; clippy/tests did not run. Full log: `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-fix1-955487b09-dev02.log`. I matched rustfmt output; the slot is free for the next retry.
kayg commented 2026-11-07 21:57:50 +00:00
Author
Owner
Copy link

J288 verify quick passed on dev-02 with the working tree that became pushed head 299056f17c (no code changes between the run and commit). Exact output summary from /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-fix2-955487b09-dev02.log:\n\n| Rust check | Result |\n| --- | --- |\n| scope guard | PASS |\n| migration names | PASS |\n| Nix Cargo vendoring | PASS |\n| fmt | PASS |\n| doc coverage ratchet | PASS |\n| shape lints off (D351) | PASS |\n| doc comments on diff | PASS |\n| fixed test waits | PASS |\n| clippy (workspace) | PASS |\n| unit tests (changed) | PASS, 55 passed |\n\nQUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done. The dev-02 target was removed after the run.

J288 `verify quick` passed on dev-02 with the working tree that became pushed head `299056f17c` (no code changes between the run and commit). Exact output summary from `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-fix2-955487b09-dev02.log`:\n\n| Rust check | Result |\n| --- | --- |\n| scope guard | PASS |\n| migration names | PASS |\n| Nix Cargo vendoring | PASS |\n| fmt | PASS |\n| doc coverage ratchet | PASS |\n| shape lints off (D351) | PASS |\n| doc comments on diff | PASS |\n| fixed test waits | PASS |\n| clippy (workspace) | PASS |\n| unit tests (changed) | PASS, 55 passed |\n\n`QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done.` The dev-02 target was removed after the run.
kayg commented 2026-11-07 21:58:19 +00:00
Author
Owner
Copy link

J288 verify quick passed on dev-02 with the working tree that became pushed head 299056f17c (no code changes between the run and commit).

Evidence: /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-fix2-955487b09-dev02.log

Rust check Result
scope guard PASS
migration names PASS
Nix Cargo vendoring PASS
fmt PASS
doc coverage ratchet PASS
shape lints off (D351) PASS
doc comments on diff PASS
fixed test waits PASS
clippy (workspace) PASS
unit tests (changed) PASS, 55 passed

QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done. The dev-02 target was removed after the run.

J288 `verify quick` passed on dev-02 with the working tree that became pushed head `299056f17c` (no code changes between the run and commit). Evidence: `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-fix2-955487b09-dev02.log` | Rust check | Result | | --- | --- | | scope guard | PASS | | migration names | PASS | | Nix Cargo vendoring | PASS | | fmt | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | clippy (workspace) | PASS | | unit tests (changed) | PASS, 55 passed | `QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done.` The dev-02 target was removed after the run.
kayg commented 2026-11-07 22:14:29 +00:00
Author
Owner
Copy link

Mac validation at exact pushed head 299056f17cfcae806a3e117359bbee0b5043c152 passed under /tmp/mac-bench.lock with CARGO_INCREMENTAL=0.

Check Result
Mesh library suite 120 passed, including full_peer_update_refreshes_existing_peer_route_mtu
narrow_path_mesh_budget_fits_relay_datagram 1 passed; Quinn max 1214 → inner 1178, max 1434 → inner 1398; prior 1200/1420 budgets returned TooLarge
Release build passed
Mac target cleanup before lock release MAC_TARGET_CLEAN=1; target was 3.1G before removal

Log: /Users/claude/.cache/codex-scratch/j288/evidence/mac-final-build-test-299056f17.log

Final binary SHA-256: d7058a1e0880964f5e63182dc971a63f2a769b0ed85cf0e3b3da3c53e6b4b3bc (/tmp/infraternal-mtu381-final-299056f17). Production relay_path remains Linux-only; this Mac run validates the Quinn datagram boundary and Mac direct-route behavior.

Mac validation at exact pushed head `299056f17cfcae806a3e117359bbee0b5043c152` passed under `/tmp/mac-bench.lock` with `CARGO_INCREMENTAL=0`. | Check | Result | | --- | --- | | Mesh library suite | 120 passed, including `full_peer_update_refreshes_existing_peer_route_mtu` | | `narrow_path_mesh_budget_fits_relay_datagram` | 1 passed; Quinn max 1214 → inner 1178, max 1434 → inner 1398; prior 1200/1420 budgets returned `TooLarge` | | Release build | passed | | Mac target cleanup before lock release | `MAC_TARGET_CLEAN=1`; target was 3.1G before removal | Log: `/Users/claude/.cache/codex-scratch/j288/evidence/mac-final-build-test-299056f17.log` Final binary SHA-256: `d7058a1e0880964f5e63182dc971a63f2a769b0ed85cf0e3b3da3c53e6b4b3bc` (`/tmp/infraternal-mtu381-final-299056f17`). Production `relay_path` remains Linux-only; this Mac run validates the Quinn datagram boundary and Mac direct-route behavior.
kayg commented 2026-11-07 22:54:05 +00:00
Author
Owner
Copy link

J288 progress: head 299056f17cfcae806a3e117359bbee0b5043c152. verify fast is still running in the foreground on compute01 after the workspace test and doc-test rows; observed rows have zero failures so far. Log: /home/claude/.cache/codex-scratch/j288/evidence/verify-fast-299056f17-compute01.log.

Mac validation at this exact head is already recorded above. Its target is absent, and the live-mesh claim is free. I will start the live measurement and any further compute01 build only after the active gate exits.

J288 progress: head `299056f17cfcae806a3e117359bbee0b5043c152`. `verify fast` is still running in the foreground on compute01 after the workspace test and doc-test rows; observed rows have zero failures so far. Log: `/home/claude/.cache/codex-scratch/j288/evidence/verify-fast-299056f17-compute01.log`. Mac validation at this exact head is already recorded above. Its target is absent, and the live-mesh claim is free. I will start the live measurement and any further compute01 build only after the active gate exits.
kayg commented 2026-11-07 22:58:53 +00:00
Author
Owner
Copy link

J288 gate update: verify fast has reached privileged integration on compute01. Its required row reports REAL-ROOT PROOF REQUIRED: TUN code changed; run scripts/tun-real-root-proof <commit> from code and record it on the issue. I will run scripts/tun-real-root-proof 299056f17cfcae806a3e117359bbee0b5043c152 after the current gate releases its slot, then record the output here. The active gate log remains /home/claude/.cache/codex-scratch/j288/evidence/verify-fast-299056f17-compute01.log.

J288 gate update: `verify fast` has reached privileged integration on compute01. Its required row reports `REAL-ROOT PROOF REQUIRED: TUN code changed; run scripts/tun-real-root-proof <commit> from code and record it on the issue`. I will run `scripts/tun-real-root-proof 299056f17cfcae806a3e117359bbee0b5043c152` after the current gate releases its slot, then record the output here. The active gate log remains `/home/claude/.cache/codex-scratch/j288/evidence/verify-fast-299056f17-compute01.log`.
kayg commented 2026-11-07 23:34:07 +00:00
Author
Owner
Copy link

J288 blocked on compute01 availability at 2026-11-07 23:33Z. SSH returned No route to host while verify fast was compiling the aarch64 release artifact. The foreground gate session has not returned an exit status, so this is not recorded as a test failure. No new build or live-mesh work has started. I will follow BRIEF-COMMON and retry compute01 after five minutes; the active gate log is /home/claude/.cache/codex-scratch/j288/evidence/verify-fast-299056f17-compute01.log.

J288 blocked on compute01 availability at 2026-11-07 23:33Z. SSH returned `No route to host` while `verify fast` was compiling the aarch64 release artifact. The foreground gate session has not returned an exit status, so this is not recorded as a test failure. No new build or live-mesh work has started. I will follow BRIEF-COMMON and retry compute01 after five minutes; the active gate log is `/home/claude/.cache/codex-scratch/j288/evidence/verify-fast-299056f17-compute01.log`.
kayg commented 2026-11-07 23:39:35 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-07 23:39Z): the five-minute SSH retry still returns No route to host. The verify fast session has not returned an exit status, so its result remains unknown and is not counted as a test failure. I have not started a replacement gate, build, or live-mesh step; I will retry after another five minutes.

J288 availability update (2026-11-07 23:39Z): the five-minute SSH retry still returns `No route to host`. The `verify fast` session has not returned an exit status, so its result remains unknown and is not counted as a test failure. I have not started a replacement gate, build, or live-mesh step; I will retry after another five minutes.
kayg commented 2026-11-07 23:45:01 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-07 23:44Z): the second five-minute SSH retry still returns No route to host. The original verify fast session remains without an exit status; this is still an infrastructure outage, not a reported test failure. No replacement build, gate, or live-mesh measurement has been started. I will continue five-minute retries.

J288 availability update (2026-11-07 23:44Z): the second five-minute SSH retry still returns `No route to host`. The original `verify fast` session remains without an exit status; this is still an infrastructure outage, not a reported test failure. No replacement build, gate, or live-mesh measurement has been started. I will continue five-minute retries.
kayg commented 2026-11-07 23:50:33 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-07 23:50Z): the third five-minute retry still returns No route to host; compute01 has been unreachable for about 17 minutes. The foreground gate session still has no exit status, so no test failure is inferred. I have not moved x86 work to another VM and have started no replacement job. Continuing the required five-minute availability checks.

J288 availability update (2026-11-07 23:50Z): the third five-minute retry still returns `No route to host`; compute01 has been unreachable for about 17 minutes. The foreground gate session still has no exit status, so no test failure is inferred. I have not moved x86 work to another VM and have started no replacement job. Continuing the required five-minute availability checks.
kayg commented 2026-11-07 23:56:05 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-07 23:55Z): the fourth five-minute retry still returns No route to host; compute01 has been unreachable for over 22 minutes. The gate session remains indeterminate. No replacement gate or measurement has started, and x86 work has not been moved to another VM. Continuing availability checks.

J288 availability update (2026-11-07 23:55Z): the fourth five-minute retry still returns `No route to host`; compute01 has been unreachable for over 22 minutes. The gate session remains indeterminate. No replacement gate or measurement has started, and x86 work has not been moved to another VM. Continuing availability checks.
kayg commented 2026-11-08 00:01:54 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 00:01Z): the fifth five-minute retry still returns No route to host; compute01 has been unreachable for about 28 minutes. The foreground gate session has no exit status, so its result remains unknown, not failed. No replacement gate/build/live-mesh run has started. Continuing the required checks.

J288 availability update (2026-11-08 00:01Z): the fifth five-minute retry still returns `No route to host`; compute01 has been unreachable for about 28 minutes. The foreground gate session has no exit status, so its result remains unknown, not failed. No replacement gate/build/live-mesh run has started. Continuing the required checks.
kayg commented 2026-11-08 00:15:02 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 00:14Z): the scheduled SSH retry still returns No route to host. The foreground verify fast session remains without an exit status; no new compute01 work has started.

J288 availability update (2026-11-08 00:14Z): the scheduled SSH retry still returns `No route to host`. The foreground `verify fast` session remains without an exit status; no new compute01 work has started.
kayg commented 2026-11-08 00:24:08 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 00:23Z): SSH still returns No route to host; compute01 has been unavailable for about 50 minutes. The foreground verify fast result remains indeterminate because no exit status has returned. No replacement job or alternate-host x86 work has started.

J288 availability update (2026-11-08 00:23Z): SSH still returns `No route to host`; compute01 has been unavailable for about 50 minutes. The foreground `verify fast` result remains indeterminate because no exit status has returned. No replacement job or alternate-host x86 work has started.
kayg commented 2026-11-08 00:29:16 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 00:28Z): SSH to compute01 still returns No route to host. The existing verify fast session has no exit status; no compute work has resumed and no replacement has started.

J288 availability update (2026-11-08 00:28Z): SSH to compute01 still returns `No route to host`. The existing `verify fast` session has no exit status; no compute work has resumed and no replacement has started.
kayg commented 2026-11-08 00:36:03 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 00:35Z): SSH to compute01 still returns No route to host; the host has been unavailable for about an hour. The active verify fast session has not returned an exit status, so it remains indeterminate. No replacement or alternate-host x86 work has started.

J288 availability update (2026-11-08 00:35Z): SSH to compute01 still returns `No route to host`; the host has been unavailable for about an hour. The active `verify fast` session has not returned an exit status, so it remains indeterminate. No replacement or alternate-host x86 work has started.
kayg commented 2026-11-08 00:41:44 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 00:41Z): SSH to compute01 continues to return No route to host. The active verify fast session still has no exit status. No replacement build or measurement has started.

J288 availability update (2026-11-08 00:41Z): SSH to compute01 continues to return `No route to host`. The active `verify fast` session still has no exit status. No replacement build or measurement has started.
kayg commented 2026-11-08 00:46:56 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 00:46Z): SSH still returns No route to host. The existing verify fast session has no exit status, and all compute01-dependent follow-up remains paused. Continuing five-minute retries.

J288 availability update (2026-11-08 00:46Z): SSH still returns `No route to host`. The existing `verify fast` session has no exit status, and all compute01-dependent follow-up remains paused. Continuing five-minute retries.
kayg commented 2026-11-08 00:52:23 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 00:51Z): SSH to compute01 still returns No route to host. The existing gate session has no exit status; no compute01 follow-up has started.

J288 availability update (2026-11-08 00:51Z): SSH to compute01 still returns `No route to host`. The existing gate session has no exit status; no compute01 follow-up has started.
kayg commented 2026-11-08 00:56:51 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 00:56Z): SSH to compute01 still returns No route to host. The current verify fast session has no exit status. No replacement gate or measurement has started.

J288 availability update (2026-11-08 00:56Z): SSH to compute01 still returns `No route to host`. The current `verify fast` session has no exit status. No replacement gate or measurement has started.
kayg commented 2026-11-08 01:01:44 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 01:01Z): SSH to compute01 still returns No route to host; the VM has been unreachable for about 88 minutes. The active verify fast session has no exit status, so it remains indeterminate. No replacement job or alternate-host x86 work has started.

J288 availability update (2026-11-08 01:01Z): SSH to compute01 still returns `No route to host`; the VM has been unreachable for about 88 minutes. The active `verify fast` session has no exit status, so it remains indeterminate. No replacement job or alternate-host x86 work has started.
kayg commented 2026-11-08 01:06:56 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 01:06Z): the five-minute SSH retry still returns No route to host. The existing verify fast process has no exit status; no replacement gate or measurement has started.

J288 availability update (2026-11-08 01:06Z): the five-minute SSH retry still returns `No route to host`. The existing `verify fast` process has no exit status; no replacement gate or measurement has started.
kayg commented 2026-11-08 01:11:37 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 01:11Z): SSH to compute01 still returns No route to host. The active gate session has no exit status; no replacement gate or measurement has started.

J288 availability update (2026-11-08 01:11Z): SSH to compute01 still returns `No route to host`. The active gate session has no exit status; no replacement gate or measurement has started.
kayg commented 2026-11-08 01:16:50 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 01:16Z): SSH to compute01 still returns No route to host. The existing verify fast run remains without an exit status; no replacement work has started.

J288 availability update (2026-11-08 01:16Z): SSH to compute01 still returns `No route to host`. The existing `verify fast` run remains without an exit status; no replacement work has started.
kayg commented 2026-11-08 01:22:20 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 01:21Z): SSH to compute01 still returns No route to host. The local foreground verify fast session remains pending without an exit code, and no compute01 work has restarted.

J288 availability update (2026-11-08 01:21Z): SSH to compute01 still returns `No route to host`. The local foreground `verify fast` session remains pending without an exit code, and no compute01 work has restarted.
kayg commented 2026-11-08 01:26:57 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 01:26Z): SSH to compute01 still returns No route to host; the original gate has no exit status. No build or measurement has resumed.

J288 availability update (2026-11-08 01:26Z): SSH to compute01 still returns `No route to host`; the original gate has no exit status. No build or measurement has resumed.
kayg commented 2026-11-08 01:32:00 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 01:31Z): SSH to compute01 still returns No route to host. The existing gate remains pending without an exit status; the live matrix and root proof remain paused.

J288 availability update (2026-11-08 01:31Z): SSH to compute01 still returns `No route to host`. The existing gate remains pending without an exit status; the live matrix and root proof remain paused.
kayg commented 2026-11-08 01:37:15 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 01:36Z): SSH to compute01 still returns No route to host. The active gate session has no exit status; compute01-dependent follow-up remains paused.

J288 availability update (2026-11-08 01:36Z): SSH to compute01 still returns `No route to host`. The active gate session has no exit status; compute01-dependent follow-up remains paused.
kayg commented 2026-11-08 01:41:21 +00:00
Author
Owner
Copy link

J288 gate interruption (2026-11-08 01:41Z): the foreground SSH session for verify fast ended with Read from remote host infraternal-dev-01-compute01: Connection timed out and client_loop: send disconnect: Broken pipe while the host was unreachable. This is recorded as an interrupted run, not a gate failure. I will resubmit the gate once in the foreground after compute01 returns. No replacement compute01 work has started yet.

J288 gate interruption (2026-11-08 01:41Z): the foreground SSH session for `verify fast` ended with `Read from remote host infraternal-dev-01-compute01: Connection timed out` and `client_loop: send disconnect: Broken pipe` while the host was unreachable. This is recorded as an interrupted run, not a gate failure. I will resubmit the gate once in the foreground after compute01 returns. No replacement compute01 work has started yet.
kayg commented 2026-11-08 01:46:47 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 01:46Z): SSH to compute01 still returns No route to host. The prior verify fast SSH session was interrupted by the outage and is not counted as failed. No replacement run has started.

J288 availability update (2026-11-08 01:46Z): SSH to compute01 still returns `No route to host`. The prior `verify fast` SSH session was interrupted by the outage and is not counted as failed. No replacement run has started.
kayg commented 2026-11-08 01:51:45 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 01:51Z): SSH to compute01 still returns No route to host. The interrupted gate awaits one foreground resubmission after recovery; no compute01 follow-up has started.

J288 availability update (2026-11-08 01:51Z): SSH to compute01 still returns `No route to host`. The interrupted gate awaits one foreground resubmission after recovery; no compute01 follow-up has started.
kayg commented 2026-11-08 01:57:28 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 01:56Z): SSH to compute01 still returns No route to host. No compute-dependent follow-up has resumed; the gate remains scheduled for one foreground resubmission after recovery.

J288 availability update (2026-11-08 01:56Z): SSH to compute01 still returns `No route to host`. No compute-dependent follow-up has resumed; the gate remains scheduled for one foreground resubmission after recovery.
kayg commented 2026-11-08 02:03:24 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 02:01Z): SSH to compute01 still returns No route to host. The fast gate remains interrupted with unknown result; no replacement run has started.

J288 availability update (2026-11-08 02:01Z): SSH to compute01 still returns `No route to host`. The fast gate remains interrupted with unknown result; no replacement run has started.
kayg commented 2026-11-08 02:07:32 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 02:07Z): SSH to compute01 still returns No route to host. No compute01 work has resumed; the previous gate remains interrupted with unknown result.

J288 availability update (2026-11-08 02:07Z): SSH to compute01 still returns `No route to host`. No compute01 work has resumed; the previous gate remains interrupted with unknown result.
kayg commented 2026-11-08 02:12:46 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 02:12Z): SSH to compute01 still returns No route to host. The gate remains interrupted with unknown result, and no replacement compute work has started.

J288 availability update (2026-11-08 02:12Z): SSH to compute01 still returns `No route to host`. The gate remains interrupted with unknown result, and no replacement compute work has started.
kayg commented 2026-11-08 02:20:16 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 02:18Z): SSH to compute01 still returns No route to host. The prior verify fast remains interrupted with unknown result; I have not queued replacement compute01 work. I will keep retrying every five minutes and resume the gate and live evidence after recovery.

J288 availability update (2026-11-08 02:18Z): SSH to compute01 still returns `No route to host`. The prior `verify fast` remains interrupted with unknown result; I have not queued replacement compute01 work. I will keep retrying every five minutes and resume the gate and live evidence after recovery.
kayg commented 2026-11-08 02:25:00 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 02:23Z): SSH to compute01 still returns No route to host. The previous verify fast remains interrupted with unknown result; I have not queued replacement compute01 work. The next retry is in five minutes.

J288 availability update (2026-11-08 02:23Z): SSH to compute01 still returns `No route to host`. The previous `verify fast` remains interrupted with unknown result; I have not queued replacement compute01 work. The next retry is in five minutes.
kayg commented 2026-11-08 02:29:03 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 02:28Z): SSH to compute01 still returns No route to host. The interrupted verify fast remains unknown; no replacement run or other compute01 work has started. I will retry in five minutes.

J288 availability update (2026-11-08 02:28Z): SSH to compute01 still returns `No route to host`. The interrupted `verify fast` remains unknown; no replacement run or other compute01 work has started. I will retry in five minutes.
kayg commented 2026-11-08 02:34:06 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 02:33Z): SSH to compute01 still returns No route to host. The verify fast interruption is still not classifiable as a test result, and no replacement work has been queued. Retrying in five minutes.

J288 availability update (2026-11-08 02:33Z): SSH to compute01 still returns `No route to host`. The `verify fast` interruption is still not classifiable as a test result, and no replacement work has been queued. Retrying in five minutes.
kayg commented 2026-11-08 02:39:39 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 02:38Z): SSH to compute01 still returns No route to host. The interrupted gate remains unknown and no replacement compute01 job has been queued. I will retry in five minutes.

J288 availability update (2026-11-08 02:38Z): SSH to compute01 still returns `No route to host`. The interrupted gate remains unknown and no replacement compute01 job has been queued. I will retry in five minutes.
kayg commented 2026-11-08 02:44:55 +00:00
Author
Owner
Copy link

J288 availability update (2026-11-08 02:43Z): SSH to compute01 continues to return No route to host. The old fast-gate session is still treated as interrupted/unknown; nothing has been queued against compute01. Next retry in five minutes.

J288 availability update (2026-11-08 02:43Z): SSH to compute01 continues to return `No route to host`. The old fast-gate session is still treated as interrupted/unknown; nothing has been queued against compute01. Next retry in five minutes.
kayg commented 2026-11-08 02:48:48 +00:00
Author
Owner
Copy link

J288 quick pre-check at exact final head 299056f17cfcae806a3e117359bbee0b5043c152 ran on dev-02 through build-slot with CARGO_INCREMENTAL=0. Changed-package unit tests: 55 passed, 0 failed; fmt, workspace clippy, doc coverage and the remaining quick rows passed. The target was deleted after completion (TARGET_CLEANED); slot is free. Evidence: /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-final-299056f17-dev02.log.

Verbatim quick summary:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| scope guard               | PASS         |
| migration names           | PASS         |
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| clippy (workspace)        | PASS         |
| unit tests (changed)      | PASS         |
+---------------------------+--------------+

QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done.
VERIFY_EXIT=0
J288 quick pre-check at exact final head `299056f17cfcae806a3e117359bbee0b5043c152` ran on dev-02 through build-slot with `CARGO_INCREMENTAL=0`. Changed-package unit tests: 55 passed, 0 failed; fmt, workspace clippy, doc coverage and the remaining quick rows passed. The target was deleted after completion (`TARGET_CLEANED`); slot is free. Evidence: `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-final-299056f17-dev02.log`. Verbatim quick summary: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | scope guard | PASS | | migration names | PASS | | Nix Cargo vendoring | PASS | | fmt | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | clippy (workspace) | PASS | | unit tests (changed) | PASS | +---------------------------+--------------+ QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done. VERIFY_EXIT=0 ```
kayg commented 2026-11-08 03:15:52 +00:00
Author
Owner
Copy link

J288 final-head verify fast rerun completed on compute01 at 299056f17cfcae806a3e117359bbee0b5043c152, but the gate exited 1. The failure is separate from the known source-path gate issue #421: infraternal-plugin-ssh --test previous_attempt_shell failed shared_ownership_shell_preserves_recovery_and_refusal_invariants (24/122 shell checks; exit codes 73/75, one live busy where the fixture expected live ready). Privileged integration and static builds passed. Full evidence: /home/claude/.cache/codex-scratch/j288/evidence/verify-fast-final-299056f17-compute01-rerun.log.

I am checking this focused test on origin/dev on compute01 before deciding whether it is an unrelated baseline/environment failure or a regression.

J288 final-head `verify fast` rerun completed on compute01 at `299056f17cfcae806a3e117359bbee0b5043c152`, but the gate exited 1. The failure is separate from the known source-path gate issue #421: `infraternal-plugin-ssh --test previous_attempt_shell` failed `shared_ownership_shell_preserves_recovery_and_refusal_invariants` (24/122 shell checks; exit codes 73/75, one `live busy` where the fixture expected `live ready`). Privileged integration and static builds passed. Full evidence: `/home/claude/.cache/codex-scratch/j288/evidence/verify-fast-final-299056f17-compute01-rerun.log`. I am checking this focused test on origin/dev on compute01 before deciding whether it is an unrelated baseline/environment failure or a regression.
kayg commented 2026-11-08 03:24:49 +00:00
Author
Owner
Copy link

J288 baseline control update: the focused shared_ownership_shell_preserves_recovery_and_refusal_invariants test passed on current origin/dev 7dc56b8e89ab8a9ccbfa59de3f1cfa8d776ba23f on compute01 (1 passed, 0 failed, 186.71 s; exit 0). Evidence: /home/claude/.cache/codex-scratch/j288/evidence/origin-dev-previous-attempt-shell-7dc56b8.log. The branch was based on earlier dev 6ffe37e, so I am checking that exact base before classifying the J288 gate failure.

J288 baseline control update: the focused `shared_ownership_shell_preserves_recovery_and_refusal_invariants` test passed on current origin/dev `7dc56b8e89ab8a9ccbfa59de3f1cfa8d776ba23f` on compute01 (1 passed, 0 failed, 186.71 s; exit 0). Evidence: `/home/claude/.cache/codex-scratch/j288/evidence/origin-dev-previous-attempt-shell-7dc56b8.log`. The branch was based on earlier dev `6ffe37e`, so I am checking that exact base before classifying the J288 gate failure.
kayg commented 2026-11-08 03:28:44 +00:00
Author
Owner
Copy link

J288 baseline control update: the same focused test also passed on the exact branch base 6ffe37e067d8fa7b090ea83c21b84b77937f76d9 (1 passed, 0 failed, 116.42 s; exit 0) on compute01. Evidence: /home/claude/.cache/codex-scratch/j288/evidence/origin-dev-base-6ffe-previous-attempt-shell.log. Current origin/dev 7dc56b8 also passed. The full gate failure did not reproduce on either baseline; I am rerunning the focused test at J288 head to check for a transient under concurrent VM load.

J288 baseline control update: the same focused test also passed on the exact branch base `6ffe37e067d8fa7b090ea83c21b84b77937f76d9` (1 passed, 0 failed, 116.42 s; exit 0) on compute01. Evidence: `/home/claude/.cache/codex-scratch/j288/evidence/origin-dev-base-6ffe-previous-attempt-shell.log`. Current `origin/dev` `7dc56b8` also passed. The full gate failure did not reproduce on either baseline; I am rerunning the focused test at J288 head to check for a transient under concurrent VM load.
kayg commented 2026-11-08 03:32:31 +00:00
Author
Owner
Copy link

J288 failure triage update: the focused test that failed in the single final-head verify fast run passed when rerun at J288 head 299056f17cfcae806a3e117359bbee0b5043c152 (1 passed, 0 failed, 115.01 s; exit 0). It also passed on both exact branch base 6ffe37e (116.42 s) and current origin/dev 7dc56b8 (186.71 s). Evidence: /home/claude/.cache/codex-scratch/j288/evidence/j288-head-previous-attempt-shell-rerun.log, origin-dev-base-6ffe-previous-attempt-shell.log, and origin-dev-previous-attempt-shell-7dc56b8.log under the same evidence directory. The full gate log remains exit 1; I am not relabeling it green or running a second full gate because the requested post-interruption resubmission was once. The remaining work is the root proof and live measurements.

J288 failure triage update: the focused test that failed in the single final-head `verify fast` run passed when rerun at J288 head `299056f17cfcae806a3e117359bbee0b5043c152` (1 passed, 0 failed, 115.01 s; exit 0). It also passed on both exact branch base `6ffe37e` (116.42 s) and current origin/dev `7dc56b8` (186.71 s). Evidence: `/home/claude/.cache/codex-scratch/j288/evidence/j288-head-previous-attempt-shell-rerun.log`, `origin-dev-base-6ffe-previous-attempt-shell.log`, and `origin-dev-previous-attempt-shell-7dc56b8.log` under the same evidence directory. The full gate log remains exit 1; I am not relabeling it green or running a second full gate because the requested post-interruption resubmission was once. The remaining work is the root proof and live measurements.
kayg commented 2026-11-08 03:37:26 +00:00
Author
Owner
Copy link

J288 root proof progress (2026-11-08 03:37Z): the required root TUN proof has cloned exact head 299056f17 under /root and is waiting in the shared build-slot wrapper. The four reported compute01 slots are currently occupied. Proof log: /home/kayg/.cache/codex-scratch/j288/evidence/tun-real-root-proof-299056f17.log. I am keeping the proof in the foreground and will continue polling.

J288 root proof progress (2026-11-08 03:37Z): the required root TUN proof has cloned exact head `299056f17` under `/root` and is waiting in the shared build-slot wrapper. The four reported compute01 slots are currently occupied. Proof log: `/home/kayg/.cache/codex-scratch/j288/evidence/tun-real-root-proof-299056f17.log`. I am keeping the proof in the foreground and will continue polling.
kayg commented 2026-11-08 03:38:52 +00:00
Author
Owner
Copy link

J288 root TUN proof completed for exact head 299056f17cfcae806a3e117359bbee0b5043c152 on compute01. elf-runpath-guard accepted the root-built test binary; an_unprivileged_user_cannot_attach_the_persisted_tun passed (1 passed, 0 failed), and the helper reported TUN_REAL_ROOT_PROOF_EXIT=0. Both the root clone and run directory were confirmed cleaned. Evidence: /home/kayg/.cache/codex-scratch/j288/evidence/tun-real-root-proof-299056f17.log.

Verbatim proof result:

running as root (uid 0) in a fresh network namespace
running 1 test
test an_unprivileged_user_cannot_attach_the_persisted_tun ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.24s

TUN_REAL_ROOT_PROOF_EXIT=0 commit=299056f17cfcae806a3e117359bbee0b5043c152
J288 root TUN proof completed for exact head `299056f17cfcae806a3e117359bbee0b5043c152` on compute01. `elf-runpath-guard` accepted the root-built test binary; `an_unprivileged_user_cannot_attach_the_persisted_tun` passed (1 passed, 0 failed), and the helper reported `TUN_REAL_ROOT_PROOF_EXIT=0`. Both the root clone and run directory were confirmed cleaned. Evidence: `/home/kayg/.cache/codex-scratch/j288/evidence/tun-real-root-proof-299056f17.log`. Verbatim proof result: ```text running as root (uid 0) in a fresh network namespace running 1 test test an_unprivileged_user_cannot_attach_the_persisted_tun ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.24s TUN_REAL_ROOT_PROOF_EXIT=0 commit=299056f17cfcae806a3e117359bbee0b5043c152 ```
kayg commented 2026-11-08 04:17:00 +00:00
Author
Owner
Copy link

J288 measurement binary progress: shipped-dev app and relay fixture were built at 9b0c731e89b70408ac719ef3a10cb1bf36bebf99; fixed-head app and relay fixture were built at 299056f17cfcae806a3e117359bbee0b5043c152 with a fresh isolated target, and the target was cleaned. The fixed-head app reports infraternal 0.1.0 (stable). Binary SHA-256 values and file output: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-final-299056f17/binary-verification.txt.

The first builder reused one target across the two worktrees and its fixed-head app compile failed with mismatched remote-rotation symbols; that run was not used. Its log is build-live-binaries.log. The final-only fresh-target build returned BUILD_EXIT=0 (build-fixed-head-only.log). I am proceeding with isolated relay proofs.

J288 measurement binary progress: shipped-dev app and relay fixture were built at `9b0c731e89b70408ac719ef3a10cb1bf36bebf99`; fixed-head app and relay fixture were built at `299056f17cfcae806a3e117359bbee0b5043c152` with a fresh isolated target, and the target was cleaned. The fixed-head app reports `infraternal 0.1.0 (stable)`. Binary SHA-256 values and `file` output: `/home/claude/.cache/codex-scratch/j288/evidence/live-matrix-final-299056f17/binary-verification.txt`. The first builder reused one target across the two worktrees and its fixed-head app compile failed with mismatched remote-rotation symbols; that run was not used. Its log is `build-live-binaries.log`. The final-only fresh-target build returned `BUILD_EXIT=0` (`build-fixed-head-only.log`). I am proceeding with isolated relay proofs.
kayg commented 2026-11-08 05:30:31 +00:00
Author
Owner
Copy link

J288 review-response work started on branch mesh-mtu-381 at base 299056f17cfcae806a3e117359bbee0b5043c152. The direct-path win stands. Red-first coverage is in progress for the relay datagram padding boundary and J350 route-refresh / egress-MTU cases on Linux and macOS. The fixed-head relay diagnostic on compute01 recorded both peer routes at MTU 1398 and 18 TooLarge drops before its TCP transfer timed out; I am retrieving and preserving that receipt while the Mac red run finishes.

J288 review-response work started on branch `mesh-mtu-381` at base `299056f17cfcae806a3e117359bbee0b5043c152`. The direct-path win stands. Red-first coverage is in progress for the relay datagram padding boundary and J350 route-refresh / egress-MTU cases on Linux and macOS. The fixed-head relay diagnostic on compute01 recorded both peer routes at MTU 1398 and 18 `TooLarge` drops before its TCP transfer timed out; I am retrieving and preserving that receipt while the Mac red run finishes.
kayg commented 2026-11-08 05:43:47 +00:00
Author
Owner
Copy link

Follow-up plan (estimate: 4 hours plus shared-slot/lock waits): finish the Mac red-first parity run under the Mac lock and clean its target; apply the relay budget block-alignment correction; run J350 coverage on Linux and the equivalent Mac unit tests; then run quick on dev-02 and fast on compute01. Finally rebuild the shipped-dev and exact-head binaries, repeat the requested direct mesh/plain matrix plus one forced-relay transfer, clean all interfaces/servers and targets, release the live-mesh claim, and post the logs and tables here. The Mac step is currently waiting in the foreground for /tmp/mac-bench.lock.

Follow-up plan (estimate: 4 hours plus shared-slot/lock waits): finish the Mac red-first parity run under the Mac lock and clean its target; apply the relay budget block-alignment correction; run J350 coverage on Linux and the equivalent Mac unit tests; then run quick on dev-02 and fast on compute01. Finally rebuild the shipped-dev and exact-head binaries, repeat the requested direct mesh/plain matrix plus one forced-relay transfer, clean all interfaces/servers and targets, release the live-mesh claim, and post the logs and tables here. The Mac step is currently waiting in the foreground for `/tmp/mac-bench.lock`.
kayg commented 2026-11-08 05:45:53 +00:00
Author
Owner
Copy link

J350 red-first Linux evidence from the pre-fix j288-red fixture on dev-02:

  • linux-relay-red-r3.log: the relay accepted the shipped 1380-byte inner packet on the 1500 path, but the derived 1200/1420 packets produced 1236/1456-byte relay payloads against Quinn maxima 1214/1434; both returned TooLarge.
  • linux-route-red-r1.log: with the underlay link at MTU 1280 and an explicit route metric of 9000, the mesh route was MTU 8920. After adding an unrelated peer, the new route was 1200 but the unchanged peer stayed at 1420.

The current #381 implementation already contains the egress-link bound and full-snapshot refresh paths. The new relay-padding test also reproduced 1178 != 1168 on both Linux and Mac. I’m validating the fixes at the current head now.

J350 red-first Linux evidence from the pre-fix `j288-red` fixture on dev-02: - `linux-relay-red-r3.log`: the relay accepted the shipped 1380-byte inner packet on the 1500 path, but the derived 1200/1420 packets produced 1236/1456-byte relay payloads against Quinn maxima 1214/1434; both returned `TooLarge`. - `linux-route-red-r1.log`: with the underlay link at MTU 1280 and an explicit route metric of 9000, the mesh route was MTU 8920. After adding an unrelated peer, the new route was 1200 but the unchanged peer stayed at 1420. The current #381 implementation already contains the egress-link bound and full-snapshot refresh paths. The new relay-padding test also reproduced `1178 != 1168` on both Linux and Mac. I’m validating the fixes at the current head now.
kayg commented 2026-11-08 06:13:20 +00:00
Author
Owner
Copy link

J288 review response progress: commit 4ca886f65efc33a007bd604d125c1fc22903d1ee is pushed to mesh-mtu-381. The relay budget now rounds down to the WireGuard 16-byte padding boundary after using Quinn’s live application-payload limit and reserving WireGuard plus link-ID overhead.

Linux red evidence on dev-02 remains in linux-relay-red-r3.log and linux-route-red-r1.log; the fixed-source green checks are narrow-path-green-299056f17-dev02.log (1 passed) and j350-green-unit-299056f17-dev02.log (152 passed, including the live relay MTU tracker).

Mac red evidence reproduced 1178 vs 1168. The fixed-source Mac checks passed 120 unit tests plus the narrow relay fixture, and the target was removed before releasing /tmp/mac-bench.lock; log: j350-green-fix-299056f17-mac.log.

verify quick passed on dev-02 after formatting and doc-comment fixes; complete log: /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-j350-relay-fix-dev02-r3.log. Next: build/test Mac at exact head 4ca886f65, run verify fast on compute01, then the real path matrix and forced-relay transfer.

J288 review response progress: commit `4ca886f65efc33a007bd604d125c1fc22903d1ee` is pushed to `mesh-mtu-381`. The relay budget now rounds down to the WireGuard 16-byte padding boundary after using Quinn’s live application-payload limit and reserving WireGuard plus link-ID overhead. Linux red evidence on dev-02 remains in `linux-relay-red-r3.log` and `linux-route-red-r1.log`; the fixed-source green checks are `narrow-path-green-299056f17-dev02.log` (1 passed) and `j350-green-unit-299056f17-dev02.log` (152 passed, including the live relay MTU tracker). Mac red evidence reproduced 1178 vs 1168. The fixed-source Mac checks passed 120 unit tests plus the narrow relay fixture, and the target was removed before releasing `/tmp/mac-bench.lock`; log: `j350-green-fix-299056f17-mac.log`. `verify quick` passed on dev-02 after formatting and doc-comment fixes; complete log: `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-j350-relay-fix-dev02-r3.log`. Next: build/test Mac at exact head `4ca886f65`, run `verify fast` on compute01, then the real path matrix and forced-relay transfer.
kayg commented 2026-11-08 06:18:11 +00:00
Author
Owner
Copy link

J288 resumed review response on mesh-mtu-381, current head 4ca886f65efc33a007bd604d125c1fc22903d1ee. I fetched origin/dev at 5d34c133a9008ee69939e6c0c0a735ada410a223; per the request I will merge that dev head, resolve any mesh conflicts against the J350 MTU regressions, then rerun the named red/green cases and the quick/fast gates. Plan: Mac exact-head tests under /tmp/mac-bench.lock with CARGO_INCREMENTAL=0 and target cleanup; quick on dev-02; fast on compute01; rebuild shipped-dev and merged-head binaries; run the direct Mac↔compute01 matrix plus one forced-relay path under ~/mesh-vm.claim; clean all interfaces, servers, targets and release locks/claim; attach the measurement table and gate evidence. Estimate: 4 hours plus shared-slot and Mac-lock waits.

Prior red/green evidence and the passing pre-merge quick log remain in the paths recorded in the preceding J288 update; these are not evidence for the merged head. Mac reports 20 GiB free and has no target/ under the J288 checkout, so I am checking its worktree location before the locked build.

J288 resumed review response on `mesh-mtu-381`, current head `4ca886f65efc33a007bd604d125c1fc22903d1ee`. I fetched `origin/dev` at `5d34c133a9008ee69939e6c0c0a735ada410a223`; per the request I will merge that dev head, resolve any mesh conflicts against the J350 MTU regressions, then rerun the named red/green cases and the quick/fast gates. Plan: Mac exact-head tests under `/tmp/mac-bench.lock` with `CARGO_INCREMENTAL=0` and target cleanup; quick on dev-02; fast on compute01; rebuild shipped-dev and merged-head binaries; run the direct Mac↔compute01 matrix plus one forced-relay path under `~/mesh-vm.claim`; clean all interfaces, servers, targets and release locks/claim; attach the measurement table and gate evidence. Estimate: 4 hours plus shared-slot and Mac-lock waits. Prior red/green evidence and the passing pre-merge quick log remain in the paths recorded in the preceding J288 update; these are not evidence for the merged head. Mac reports 20 GiB free and has no `target/` under the J288 checkout, so I am checking its worktree location before the locked build.
kayg commented 2026-11-08 06:26:08 +00:00
Author
Owner
Copy link

J288 merge update: merged current origin/dev (5d34c133a9008ee69939e6c0c0a735ada410a223) into mesh-mtu-381 at 7b8406ad5192ae18248c898823b9ded03d0b9088, pushed. The merge overlapped the new GotaTun key-rotation worker with MTU updates and route ownership. Resolution keeps both serialized worker commands, the live TUN MTU watcher, dev's rotation link proof, Mac key updates, and the unchanged-peer route refresh. The earlier Linux and Mac red evidence remains in the preceding comment; all exact-head green/gate evidence is still pending.

J288 merge update: merged current `origin/dev` (`5d34c133a9008ee69939e6c0c0a735ada410a223`) into `mesh-mtu-381` at `7b8406ad5192ae18248c898823b9ded03d0b9088`, pushed. The merge overlapped the new GotaTun key-rotation worker with MTU updates and route ownership. Resolution keeps both serialized worker commands, the live TUN MTU watcher, dev's rotation link proof, Mac key updates, and the unchanged-peer route refresh. The earlier Linux and Mac red evidence remains in the preceding comment; all exact-head green/gate evidence is still pending.
kayg commented 2026-11-08 06:34:13 +00:00
Author
Owner
Copy link

J288 exact-head quick progress (7b8406ad5192ae18248c898823b9ded03d0b9088): narrow_path_mesh_budget_fits_relay_datagram passed on Linux with --features relay,test-support. At 1280, the derived 1168-byte inner budget produced a 1204-byte relay frame under Quinn's 1214-byte limit; at 1500, 1392 produced 1428 under 1434. The old direct-path budgets returned TooLarge in both cases. Receipt: /home/claude/.cache/codex-scratch/j288/evidence/narrow-path-7b8406-dev02-green.log.

verify quick passed on dev-02 at this head. Full log: /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-7b8406-dev02-rerun.log. Verbatim summary:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| scope guard               | PASS         |
| migration names           | PASS         |
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| clippy (workspace)        | PASS         |
| unit tests (changed)      | PASS         |
+---------------------------+--------------+

QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done.
VERIFY_QUICK_EXIT=0
J288 exact-head quick progress (`7b8406ad5192ae18248c898823b9ded03d0b9088`): `narrow_path_mesh_budget_fits_relay_datagram` passed on Linux with `--features relay,test-support`. At 1280, the derived 1168-byte inner budget produced a 1204-byte relay frame under Quinn's 1214-byte limit; at 1500, 1392 produced 1428 under 1434. The old direct-path budgets returned `TooLarge` in both cases. Receipt: `/home/claude/.cache/codex-scratch/j288/evidence/narrow-path-7b8406-dev02-green.log`. `verify quick` passed on dev-02 at this head. Full log: `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-7b8406-dev02-rerun.log`. Verbatim summary: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | scope guard | PASS | | migration names | PASS | | Nix Cargo vendoring | PASS | | fmt | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | clippy (workspace) | PASS | | unit tests (changed) | PASS | +---------------------------+--------------+ QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done. VERIFY_QUICK_EXIT=0 ```
kayg commented 2026-11-08 06:43:17 +00:00
Author
Owner
Copy link

J288 exact-head macOS validation passed at 7b8406ad5192ae18248c898823b9ded03d0b9088 under /tmp/mac-bench.lock, with CARGO_INCREMENTAL=0. The Mac mesh lib suite ran 125 passed, 0 failed, 4 ignored; this includes macos_node::tests::full_peer_update_refreshes_existing_peer_route_mtu, mesh_mtu::tests::endpoint_route_metric_is_bounded_by_egress_link_mtu, and mesh_mtu::tests::relay_route_budget_fits_padded_wireguard_packets. The relay fixture also passed 1/1 at 1280 and 1500, with 1204/1214 and 1428/1434 bytes respectively. Full evidence: /Users/claude/.cache/codex-scratch/j288/evidence/mac-7b8406-build-test.log. The target is absent after cleanup, and the Mac lock has been released.

J288 exact-head macOS validation passed at `7b8406ad5192ae18248c898823b9ded03d0b9088` under `/tmp/mac-bench.lock`, with `CARGO_INCREMENTAL=0`. The Mac mesh lib suite ran 125 passed, 0 failed, 4 ignored; this includes `macos_node::tests::full_peer_update_refreshes_existing_peer_route_mtu`, `mesh_mtu::tests::endpoint_route_metric_is_bounded_by_egress_link_mtu`, and `mesh_mtu::tests::relay_route_budget_fits_padded_wireguard_packets`. The relay fixture also passed 1/1 at 1280 and 1500, with 1204/1214 and 1428/1434 bytes respectively. Full evidence: `/Users/claude/.cache/codex-scratch/j288/evidence/mac-7b8406-build-test.log`. The target is absent after cleanup, and the Mac lock has been released.
kayg commented 2026-11-08 06:58:55 +00:00
Author
Owner
Copy link

J288 merged-head Mac release binary is built at 7b8406ad5192ae18248c898823b9ded03d0b9088 under /tmp/mac-bench.lock with CARGO_INCREMENTAL=0: Mach-O 64-bit executable arm64, SHA-256 9a166863f2c0d64aa7c98edd8aad5970e29a21023afeaed2bb34e04594076a56. Evidence: /Users/claude/.cache/codex-scratch/j288/evidence/mac-release-7b8406.log. target/ was removed before lock release; Mac free space is back to 20 GiB. Next I am building the shipped-dev control binary under a separate locked step.

J288 merged-head Mac release binary is built at `7b8406ad5192ae18248c898823b9ded03d0b9088` under `/tmp/mac-bench.lock` with `CARGO_INCREMENTAL=0`: `Mach-O 64-bit executable arm64`, SHA-256 `9a166863f2c0d64aa7c98edd8aad5970e29a21023afeaed2bb34e04594076a56`. Evidence: `/Users/claude/.cache/codex-scratch/j288/evidence/mac-release-7b8406.log`. `target/` was removed before lock release; Mac free space is back to 20 GiB. Next I am building the shipped-dev control binary under a separate locked step.
kayg commented 2026-11-08 07:36:45 +00:00
Author
Owner
Copy link

J288 progress at 7b8406ad5192ae18248c898823b9ded03d0b9088: verify fast is still running on compute01 through build-slot; log: /home/claude/.cache/codex-scratch/j288/evidence/verify-fast-7b8406-compute01.log. The partial run has a node_actions::tests::failed_cleanup_reports_stopping_and_reconnect_keeps_explicit_off failure (configure: Rejected) plus several review_* nftables namespace failures. I am not classifying these as branch or host failures until the gate exits and I compare controls. The symlinked-target source-path failure is already tracked separately in #421 with origin/dev and physical-target controls. Quick and exact-head Mac tests remain recorded as passing. After the gate I will finish the direct Mac↔compute01 matrix and forced-relay comparison, then post the results and cleanup evidence.

J288 progress at `7b8406ad5192ae18248c898823b9ded03d0b9088`: `verify fast` is still running on compute01 through build-slot; log: `/home/claude/.cache/codex-scratch/j288/evidence/verify-fast-7b8406-compute01.log`. The partial run has a `node_actions::tests::failed_cleanup_reports_stopping_and_reconnect_keeps_explicit_off` failure (`configure: Rejected`) plus several `review_*` nftables namespace failures. I am not classifying these as branch or host failures until the gate exits and I compare controls. The symlinked-target source-path failure is already tracked separately in #421 with origin/dev and physical-target controls. Quick and exact-head Mac tests remain recorded as passing. After the gate I will finish the direct Mac↔compute01 matrix and forced-relay comparison, then post the results and cleanup evidence.
kayg commented 2026-11-08 08:07:40 +00:00
Author
Owner
Copy link

J288 follow-up start — continuing mesh-mtu-381 at 7b8406ad5192ae18248c898823b9ded03d0b9088; base is current origin/dev 5d34c133a9008ee69939e6c0c0a735ada410a223.

The J350 test-only review commit 38e3ddc records all three failures against submitted head 36e7a6308 (relay.log and mtu-repro-root.log in the review evidence). The current branch contains 2647e8f for endpoint-route clamping and full peer-route refresh, and 4ca886f for relay sizing after WireGuard 16-byte padding. I am checking those cases at the merged head on Linux and macOS, then I will run the requested gates and live matrix. #421 is already open with the origin/dev symlink-target control; I will append a fresh control receipt.

**J288 follow-up start** — continuing `mesh-mtu-381` at `7b8406ad5192ae18248c898823b9ded03d0b9088`; base is current `origin/dev` `5d34c133a9008ee69939e6c0c0a735ada410a223`. The J350 test-only review commit `38e3ddc` records all three failures against submitted head `36e7a6308` (`relay.log` and `mtu-repro-root.log` in the review evidence). The current branch contains `2647e8f` for endpoint-route clamping and full peer-route refresh, and `4ca886f` for relay sizing after WireGuard 16-byte padding. I am checking those cases at the merged head on Linux and macOS, then I will run the requested gates and live matrix. #421 is already open with the origin/dev symlink-target control; I will append a fresh control receipt.
kayg commented 2026-11-08 08:26:47 +00:00
Author
Owner
Copy link

J288 progress — Linux regressions and quick pre-check

The J350 test-only change 38e3ddc fails against submitted 36e7a6308 with the expected stale route MTUs and relay TooLarge results (review artifacts mtu-repro-root.log and relay.log). At the merged head 7b8406ad5, all three focused regressions pass on Linux dev-02 in isolated namespaces:

  • narrow_path_mesh_budget_fits_relay_datagram: pass; safe inner MTUs 1168 at max_datagram_size=1214 and 1392 at 1434; frames 1204 and 1428 accepted. Log: claude@infraternal-dev-02-std:~/.cache/codex-scratch/j288/evidence/linux-relay-budget-green.log.
  • endpoint_route_metric_is_bounded_by_egress_link_mtu: pass; route metric 9000 is capped by the egress link at 1280, producing mesh route MTU 1200. Log: .../linux-endpoint-mtu-green.log.
  • full_peer_update_refreshes_existing_peer_route_mtu: pass; retained peer moves from 1420 to 1200 after the 1280 path change, matching the new peer. Log: .../linux-peer-route-refresh-green.log.

verify quick passed on dev-02 against origin/dev 5d34c133a9. Verbatim summary:

+---------------------------+--------------+
| Rust gate | Result |
+---------------------------+--------------+
| scope guard | PASS |
| migration names | PASS |
| Nix Cargo vendoring | PASS |
| fmt | PASS |
| doc coverage ratchet | PASS |
| shape lints off (D351) | PASS |
| doc comments on diff | PASS |
| fixed test waits | PASS |
| clippy (workspace) | PASS |
| unit tests (changed) | PASS |
+---------------------------+--------------+

QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done.

Mac build and parity tests are running at exact head 7b8406ad51 under /tmp/mac-bench.lock; target cleanup is part of the locked step. Log: /home/kayg/.cache/codex-scratch/j288/evidence/mac-head-7b840-build-test.log.

**J288 progress — Linux regressions and quick pre-check** The J350 test-only change 38e3ddc fails against submitted 36e7a6308 with the expected stale route MTUs and relay TooLarge results (review artifacts mtu-repro-root.log and relay.log). At the merged head 7b8406ad5, all three focused regressions pass on Linux dev-02 in isolated namespaces: - narrow_path_mesh_budget_fits_relay_datagram: pass; safe inner MTUs 1168 at max_datagram_size=1214 and 1392 at 1434; frames 1204 and 1428 accepted. Log: claude@infraternal-dev-02-std:~/.cache/codex-scratch/j288/evidence/linux-relay-budget-green.log. - endpoint_route_metric_is_bounded_by_egress_link_mtu: pass; route metric 9000 is capped by the egress link at 1280, producing mesh route MTU 1200. Log: .../linux-endpoint-mtu-green.log. - full_peer_update_refreshes_existing_peer_route_mtu: pass; retained peer moves from 1420 to 1200 after the 1280 path change, matching the new peer. Log: .../linux-peer-route-refresh-green.log. verify quick passed on dev-02 against origin/dev 5d34c133a9008ee69939e6c0c0a735ada410a223. Verbatim summary: +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | scope guard | PASS | | migration names | PASS | | Nix Cargo vendoring | PASS | | fmt | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | clippy (workspace) | PASS | | unit tests (changed) | PASS | +---------------------------+--------------+ QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done. Mac build and parity tests are running at exact head 7b8406ad5192ae18248c898823b9ded03d0b9088 under /tmp/mac-bench.lock; target cleanup is part of the locked step. Log: /home/kayg/.cache/codex-scratch/j288/evidence/mac-head-7b840-build-test.log.
kayg commented 2026-11-08 08:50:33 +00:00
Author
Owner
Copy link

J288 finding at head 231143ade3c8f7a62dcd3b197053235dcdccdc24:

The first compute01 verify fast run failed the two route-MTU regressions because ip route get omitted the configured per-route MTU in its lookup-cache output. The assertions now inspect the exact peer-prefix FIB entry with ip route show exact, which reflects the installed route metric.

Both focused compute01 namespace tests pass at the pushed head:

  • endpoint_route_metric_is_bounded_by_egress_link_mtu: underlay link MTU 1280, installed mesh route MTU 1200.
  • full_peer_update_refreshes_existing_peer_route_mtu: unchanged peer route 1420 before the path narrows, then 1200 after; the newly added peer is also 1200.

Evidence: /home/claude/.cache/codex-scratch/j288/evidence/route-table-observation-tests.log. The earlier gate log is /home/claude/.cache/codex-scratch/j288/evidence/verify-fast.log; its remaining unit-test failure was logins::durable_tests::login_durable_writes_leave_the_reactor_responsive, which I will recheck in the final gate run.

J288 finding at head `231143ade3c8f7a62dcd3b197053235dcdccdc24`: The first compute01 `verify fast` run failed the two route-MTU regressions because `ip route get` omitted the configured per-route MTU in its lookup-cache output. The assertions now inspect the exact peer-prefix FIB entry with `ip route show exact`, which reflects the installed route metric. Both focused compute01 namespace tests pass at the pushed head: - `endpoint_route_metric_is_bounded_by_egress_link_mtu`: underlay link MTU 1280, installed mesh route MTU 1200. - `full_peer_update_refreshes_existing_peer_route_mtu`: unchanged peer route 1420 before the path narrows, then 1200 after; the newly added peer is also 1200. Evidence: `/home/claude/.cache/codex-scratch/j288/evidence/route-table-observation-tests.log`. The earlier gate log is `/home/claude/.cache/codex-scratch/j288/evidence/verify-fast.log`; its remaining unit-test failure was `logins::durable_tests::login_durable_writes_leave_the_reactor_responsive`, which I will recheck in the final gate run.
kayg commented 2026-11-08 09:35:22 +00:00
Author
Owner
Copy link

J288 review-response continuation at mesh-mtu-381 head 231143ade3f260e081fb23ee0513bd3a0872c707. The branch already merges current origin/dev 5d34c133a9008ee69939e6c0c0a735ada410a223 at 7b8406ad5; that dev history includes the macOS build fix from #379.

Plan (estimate: 4 hours plus shared-slot/lock waits): isolate the privileged failures from the previous fast run against the passing origin/dev control; use the J350 regression tests red-first and green on Linux and macOS; fix any confirmed regression and commit/push atomically; rerun quick on dev-02 and fast on compute01; build and test the Mac at the final head under /tmp/mac-bench.lock; then measure shipped dev versus the final head on the live Mac↔compute01 path (plain/mesh, both directions, 1/8 flows, 3 repetitions, plus a forced relay), clean all resources and post evidence.

The three J350 assertions have prior red receipts on Linux and Mac and green receipts on Linux; the Mac exact-head tests previously passed at 7b8406ad5, before the route-observation test-only commit. The current compute01 fast log and the origin/dev control are being inspected before the next code change. No new VM run is queued.

J288 review-response continuation at `mesh-mtu-381` head `231143ade3f260e081fb23ee0513bd3a0872c707`. The branch already merges current `origin/dev` `5d34c133a9008ee69939e6c0c0a735ada410a223` at `7b8406ad5`; that dev history includes the macOS build fix from #379. Plan (estimate: 4 hours plus shared-slot/lock waits): isolate the privileged failures from the previous fast run against the passing origin/dev control; use the J350 regression tests red-first and green on Linux and macOS; fix any confirmed regression and commit/push atomically; rerun quick on dev-02 and fast on compute01; build and test the Mac at the final head under `/tmp/mac-bench.lock`; then measure shipped dev versus the final head on the live Mac↔compute01 path (plain/mesh, both directions, 1/8 flows, 3 repetitions, plus a forced relay), clean all resources and post evidence. The three J350 assertions have prior red receipts on Linux and Mac and green receipts on Linux; the Mac exact-head tests previously passed at `7b8406ad5`, before the route-observation test-only commit. The current compute01 fast log and the origin/dev control are being inspected before the next code change. No new VM run is queued.
kayg commented 2026-11-08 09:56:25 +00:00
Author
Owner
Copy link

J288 finding: the previous fast failure was introduced at the live-link proof boundary. Node::apply now needs an interface index before it can update the MTU; owned_gotatun_link delegated to rotation_link, which requires a durable node_state::Store. Namespace integration fixtures construct a live in-memory Node::with_firewall without that store, so the first full snapshot returned NotConnected before MTU configuration. The unmodified origin/dev privileged control passed 105/105 (/home/claude/.cache/codex-scratch/j288/evidence/verify-privileged-origin-dev-5d34.log).

The focused red receipt is /home/claude/.cache/codex-scratch/j288/evidence/focused-authorized-reenrollment-diag.log; the marker confirms failure at the ownership proof. The fix keeps durable proof for restored nodes and proves a fresh in-memory node against its live GotaTun engine handle and generated identity key. The clean privileged reenrollment test now passes 1/1 at current source: /home/claude/.cache/codex-scratch/j288/evidence/focused-authorized-reenrollment-clean.log. Next I’m running quick, then the full fast gate on compute01 to confirm the other first-snapshot failures clear.

J288 finding: the previous fast failure was introduced at the live-link proof boundary. `Node::apply` now needs an interface index before it can update the MTU; `owned_gotatun_link` delegated to `rotation_link`, which requires a durable `node_state::Store`. Namespace integration fixtures construct a live in-memory `Node::with_firewall` without that store, so the first full snapshot returned `NotConnected` before MTU configuration. The unmodified `origin/dev` privileged control passed 105/105 (`/home/claude/.cache/codex-scratch/j288/evidence/verify-privileged-origin-dev-5d34.log`). The focused red receipt is `/home/claude/.cache/codex-scratch/j288/evidence/focused-authorized-reenrollment-diag.log`; the marker confirms failure at the ownership proof. The fix keeps durable proof for restored nodes and proves a fresh in-memory node against its live GotaTun engine handle and generated identity key. The clean privileged reenrollment test now passes 1/1 at current source: `/home/claude/.cache/codex-scratch/j288/evidence/focused-authorized-reenrollment-clean.log`. Next I’m running quick, then the full fast gate on compute01 to confirm the other first-snapshot failures clear.
kayg commented 2026-11-08 11:19:33 +00:00
Author
Owner
Copy link

J288 follow-up progress at mesh-mtu-381 head 0f85e3a2e9c486a90b91bb2bdc3f9c84357ad2aa, which includes current origin/dev 5d34c133a9008ee69939e6c0c0a735ada410a223.

  • Exact-head Mac validation passed under /tmp/mac-bench.lock with CARGO_INCREMENTAL=0: 125 passed, 0 failed, 4 ignored; the relay datagram test accepted 1204/1214 bytes at 1280 and 1428/1434 at 1500. Log: /home/kayg/.cache/codex-scratch/j288/evidence/mac-head-0f85e3a2e-build-test.log. Mac release binary SHA-256: 9a166863f2c0d64aa7c98edd8aad5970e29a21023afeaed2bb34e04594076a56; target was removed before releasing the lock.

  • verify quick passed on dev-02 at this exact head. Log: /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-final-0f85e3a2e-dev02.log.

  • verify fast ran on compute01. It fails the tracer docs test because doc.js is 13,408 bytes against the 12,000-byte limit. The same test exits 1 on clean origin/dev 5d34c133a9008ee69939e6c0c0a735ada410a223; control log: /home/kayg/.cache/codex-scratch/j288/evidence/docjs-origin-dev-5d34-control.log. Final-head log: /home/claude/.cache/codex-scratch/j288/evidence/verify-fast-final-0f85e3a2-compute01.log.

  • Forced-relay proof passed for shipped dev 9b0c731e8 and the fixed-head fixture using the same baseline-compatible harness. Each delivered a SHA-256-verified 32 MiB relayed stream with too_large=0; the build-host functional rates were 26.3 and 28.8 Mbit/s respectively (not D348 benchmark evidence). The rest of the proof, including relay failover, passed and host state was unchanged. Logs: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-final-0f85e3a2e/relay-shipped-dev-retry.log and .../relay-fixed-head-compatible.log.

Next I’m running the direct Mac↔compute01 matrix (plain and mesh, both directions, 1 and 8 flows, 3 repetitions) under the mesh claim and Mac lock.

J288 follow-up progress at `mesh-mtu-381` head `0f85e3a2e9c486a90b91bb2bdc3f9c84357ad2aa`, which includes current `origin/dev` `5d34c133a9008ee69939e6c0c0a735ada410a223`. - Exact-head Mac validation passed under `/tmp/mac-bench.lock` with `CARGO_INCREMENTAL=0`: 125 passed, 0 failed, 4 ignored; the relay datagram test accepted 1204/1214 bytes at 1280 and 1428/1434 at 1500. Log: `/home/kayg/.cache/codex-scratch/j288/evidence/mac-head-0f85e3a2e-build-test.log`. Mac release binary SHA-256: `9a166863f2c0d64aa7c98edd8aad5970e29a21023afeaed2bb34e04594076a56`; target was removed before releasing the lock. - `verify quick` passed on dev-02 at this exact head. Log: `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-final-0f85e3a2e-dev02.log`. - `verify fast` ran on compute01. It fails the tracer docs test because `doc.js` is 13,408 bytes against the 12,000-byte limit. The same test exits 1 on clean `origin/dev` `5d34c133a9008ee69939e6c0c0a735ada410a223`; control log: `/home/kayg/.cache/codex-scratch/j288/evidence/docjs-origin-dev-5d34-control.log`. Final-head log: `/home/claude/.cache/codex-scratch/j288/evidence/verify-fast-final-0f85e3a2-compute01.log`. - Forced-relay proof passed for shipped dev `9b0c731e8` and the fixed-head fixture using the same baseline-compatible harness. Each delivered a SHA-256-verified 32 MiB relayed stream with `too_large=0`; the build-host functional rates were 26.3 and 28.8 Mbit/s respectively (not D348 benchmark evidence). The rest of the proof, including relay failover, passed and host state was unchanged. Logs: `/home/claude/.cache/codex-scratch/j288/evidence/live-matrix-final-0f85e3a2e/relay-shipped-dev-retry.log` and `.../relay-fixed-head-compatible.log`. Next I’m running the direct Mac↔compute01 matrix (plain and mesh, both directions, 1 and 8 flows, 3 repetitions) under the mesh claim and Mac lock.
kayg commented 2026-11-08 11:41:45 +00:00
Author
Owner
Copy link

J288 direct-matrix attempt 1 stopped before mesh setup: all three shipped-dev plain repetitions completed on the Mac utun100 path (MTU 1280, RTT about 149 ms), then the scratch Linux controller raised AttributeError for args.mac_before_sha; its parser defines mac_before_sha256. The run report is marked FAIL. Cleanup evidence confirms no Linux interface, Mac utun, listener, scratch tree, or mesh claim remains. I am preserving the partial report and rerunning the complete matrix with the corrected harness. Evidence: compute01 /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-final-0f85e3a2e/measurement/direct-matrix-final-0f85e3a2e.log and /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-final-0f85e3a2e/direct-path-final/mesh-mtu-live-matrix-report.json; Mac /Users/lume/.cache/codex-scratch/j288/live-matrix-final-0f85e3a2e/evidence/mac-controller-report.json.

J288 direct-matrix attempt 1 stopped before mesh setup: all three shipped-dev plain repetitions completed on the Mac utun100 path (MTU 1280, RTT about 149 ms), then the scratch Linux controller raised AttributeError for args.mac_before_sha; its parser defines mac_before_sha256. The run report is marked FAIL. Cleanup evidence confirms no Linux interface, Mac utun, listener, scratch tree, or mesh claim remains. I am preserving the partial report and rerunning the complete matrix with the corrected harness. Evidence: compute01 /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-final-0f85e3a2e/measurement/direct-matrix-final-0f85e3a2e.log and /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-final-0f85e3a2e/direct-path-final/mesh-mtu-live-matrix-report.json; Mac /Users/lume/.cache/codex-scratch/j288/live-matrix-final-0f85e3a2e/evidence/mac-controller-report.json.
kayg commented 2026-11-08 12:40:30 +00:00
Author
Owner
Copy link

J288 live-matrix harness correction: the prior Mac controller ran as lume, but macos_key::load_or_create requires effective root. That attempt stopped before mesh throughput was measured; its partial plain-only samples are excluded. The root-mode diagnostic passes mesh.up/mesh.down for both shipped-dev and head, with no utun left behind (/Users/lume/.cache/codex-scratch/j288/live-matrix-final-corrected-0f85e3a2e/evidence/root-node-diagnostic.log). I corrected the controller to run as root under /tmp/mac-bench.lock, preserve HOME=/Users/lume, and return report ownership to lume. The full direct matrix is being rerun now.

J288 live-matrix harness correction: the prior Mac controller ran as `lume`, but `macos_key::load_or_create` requires effective root. That attempt stopped before mesh throughput was measured; its partial plain-only samples are excluded. The root-mode diagnostic passes `mesh.up`/`mesh.down` for both shipped-dev and head, with no utun left behind (`/Users/lume/.cache/codex-scratch/j288/live-matrix-final-corrected-0f85e3a2e/evidence/root-node-diagnostic.log`). I corrected the controller to run as root under `/tmp/mac-bench.lock`, preserve `HOME=/Users/lume`, and return report ownership to `lume`. The full direct matrix is being rerun now.
kayg commented 2026-11-08 13:06:48 +00:00
Author
Owner
Copy link

Finding: the current live run stops at Mac mesh.configure before any mesh throughput sample. The Mac controller reply is just Rejected, but tracing the data path shows the scratch chunk omits local_key; macos_node::chunk requires local_key == the enabled Mac identity key. The capture shows the actual underlay through utun100 at MTU 1280 and no mesh utun after cleanup. Failed controller receipt: /home/kayg/.cache/codex-scratch/j288/evidence/live-matrix-rooted-0f85e3a2e/diagnostics/mac-controller-report.json; failure trace: .../diagnostics/mac-ssh.stderr. I will add the public local_key to the measurement fixture, merge current origin/dev (abad8c6c0), rebuild/test Mac at the resulting exact head, then rerun the matrix. Earlier plain-only samples remain excluded.

Finding: the current live run stops at Mac mesh.configure before any mesh throughput sample. The Mac controller reply is just Rejected, but tracing the data path shows the scratch chunk omits local_key; macos_node::chunk requires local_key == the enabled Mac identity key. The capture shows the actual underlay through utun100 at MTU 1280 and no mesh utun after cleanup. Failed controller receipt: /home/kayg/.cache/codex-scratch/j288/evidence/live-matrix-rooted-0f85e3a2e/diagnostics/mac-controller-report.json; failure trace: .../diagnostics/mac-ssh.stderr. I will add the public local_key to the measurement fixture, merge current origin/dev (abad8c6c0), rebuild/test Mac at the resulting exact head, then rerun the matrix. Earlier plain-only samples remain excluded.
kayg commented 2026-11-08 13:42:23 +00:00
Author
Owner
Copy link

J288 progress at mesh-mtu-381 head 35e3f08bac0813807741ada4d46789415c1b9a2c, merging current origin/dev abad8c6c0 for exact-head Mac validation.

  • verify quick passed on dev-02. Log: /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-35e3f08-dev02.log. Summary: scope guard, migration names, Nix Cargo vendoring, fmt, doc coverage ratchet, shape lints, D345 comments, fixed waits, workspace clippy, and changed unit tests all PASS. The mesh package ran 56 passed, 0 failed.
  • Exact-head Mac release build passed under /tmp/mac-bench.lock with CARGO_INCREMENTAL=0. Binary SHA-256: 8b595c9ae1b2afa129be55bc61de991ef9b02fa4fc9038a2374cf3e50495ae09. With relay,test-support enabled, Mac mesh tests ran 130 passed, 0 failed, 4 ignored; narrow_path_mesh_budget_fits_relay_datagram passed 1/1. Logs: /Users/claude/.cache/codex-scratch/j288/evidence/mac-head-35e3f08-build-test.log and /Users/claude/.cache/codex-scratch/j288/evidence/mac-j350-featured-35e3f08.log. Both locked steps removed target/ before releasing the lock.
  • Live mesh throughput is still pending. I traced the previous refusal to the scratch fixture omitting local_key, which Mac requires to match its active public key. The fixture now includes that public key; the next live run will validate setup before measuring.
  • verify fast is running on compute01 at this head; final output and any control comparison will follow.
J288 progress at `mesh-mtu-381` head `35e3f08bac0813807741ada4d46789415c1b9a2c`, merging current `origin/dev` `abad8c6c0` for exact-head Mac validation. - `verify quick` passed on dev-02. Log: `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-35e3f08-dev02.log`. Summary: scope guard, migration names, Nix Cargo vendoring, fmt, doc coverage ratchet, shape lints, D345 comments, fixed waits, workspace clippy, and changed unit tests all PASS. The mesh package ran 56 passed, 0 failed. - Exact-head Mac release build passed under `/tmp/mac-bench.lock` with `CARGO_INCREMENTAL=0`. Binary SHA-256: `8b595c9ae1b2afa129be55bc61de991ef9b02fa4fc9038a2374cf3e50495ae09`. With `relay,test-support` enabled, Mac mesh tests ran 130 passed, 0 failed, 4 ignored; `narrow_path_mesh_budget_fits_relay_datagram` passed 1/1. Logs: `/Users/claude/.cache/codex-scratch/j288/evidence/mac-head-35e3f08-build-test.log` and `/Users/claude/.cache/codex-scratch/j288/evidence/mac-j350-featured-35e3f08.log`. Both locked steps removed `target/` before releasing the lock. - Live mesh throughput is still pending. I traced the previous refusal to the scratch fixture omitting `local_key`, which Mac requires to match its active public key. The fixture now includes that public key; the next live run will validate setup before measuring. - `verify fast` is running on compute01 at this head; final output and any control comparison will follow.
kayg commented 2026-11-08 14:58:10 +00:00
Author
Owner
Copy link

J288 gate finding at exact head 35e3f08bac0813807741ada4d46789415c1b9a2c:

verify fast exited 1. Every Rust gate row passed, including the workspace tests, 102 privileged integration checks, and static builds. The remaining Tracer suite reports 654 passed and 1 failed: docs-build.test.ts says doc.js stays small, expected < 12000 bytes, received 13408.

This reproduces on current origin/dev 7c8bd2221a2cfabd1cbf2f839edc3a1aeb3bfb7a in a clean detached worktree on dev-02: bun test docs-build.test.ts reports 16 passed and the same one failure with the same 13,408-byte result. The mesh branch changes no Tracer files. Evidence:

  • Submitted head gate: /home/claude/.cache/codex-scratch/j288/evidence/verify-fast-35e3f08-compute01.log
  • Current origin/dev control: /home/claude/.cache/codex-scratch/j288/evidence/docs-origin-dev-control.log

This is an existing origin/dev gate failure, so I’m continuing with the requested live measurement. The separate symlinked-target static-path gate issue is already tracked with control evidence in #421.

J288 gate finding at exact head `35e3f08bac0813807741ada4d46789415c1b9a2c`: `verify fast` exited 1. Every Rust gate row passed, including the workspace tests, 102 privileged integration checks, and static builds. The remaining Tracer suite reports 654 passed and 1 failed: `docs-build.test.ts` says `doc.js stays small`, expected `< 12000` bytes, received `13408`. This reproduces on current `origin/dev` `7c8bd2221a2cfabd1cbf2f839edc3a1aeb3bfb7a` in a clean detached worktree on dev-02: `bun test docs-build.test.ts` reports 16 passed and the same one failure with the same 13,408-byte result. The mesh branch changes no Tracer files. Evidence: - Submitted head gate: `/home/claude/.cache/codex-scratch/j288/evidence/verify-fast-35e3f08-compute01.log` - Current origin/dev control: `/home/claude/.cache/codex-scratch/j288/evidence/docs-origin-dev-control.log` This is an existing `origin/dev` gate failure, so I’m continuing with the requested live measurement. The separate symlinked-target static-path gate issue is already tracked with control evidence in #421.
kayg commented 2026-11-08 15:55:35 +00:00
Author
Owner
Copy link

J288 review-response: the direct Linux↔Mac matrix completed at head 35e3f08bac on the real compute01↔Mac path. It ran 2026-11-08 15:27:21–15:47:40 UTC with three repetitions per path/flow count. The underlay was MTU 1280 on both sides, with about 149 ms idle median RTT. “Down” is Linux→Mac; “up” is Mac→Linux. Rates are Mbit/s; each cell lists the three samples and their median where all three completed.

Build/path Flows Linux→Mac Mac→Linux
shipped dev / plain 1 127.4, 140.2, 136.4 (136.4) 105.6, 102.1, 106.8 (105.6)
shipped dev / plain 8 382.7, 497.4, 441.4 (441.4) 208.1, 296.6, 311.8 (296.6)
shipped dev / mesh 1 0.4, 0.4, 0.4 (0.4) 110.8, 128.6, 89.9 (110.8)
shipped dev / mesh 8 error, error, 50.4 (r3 only) error, error, 141.6 (r3 only)
35e3f08 / plain 1 141.7, 136.2, 144.5 (141.7) 106.4, 103.4, 105.8 (105.8)
35e3f08 / plain 8 631.9, 504.2, 488.5 (504.2) 317.9, 202.0, 326.1 (317.9)
35e3f08 / mesh 1 129.0, 130.1, 129.4 (129.4) 164.4, 166.3, 165.1 (165.1)
35e3f08 / mesh 8 408.6, 393.2, 418.2 (408.6) 215.0, 201.1, 184.3 (201.1)

The shipped-dev Linux mesh link stayed at MTU 1380 over the 1280 underlay. At 35e3f08 the Linux mesh link was 1280 with peer route MTU 1200; the Mac route to its peer also reported MTU 1200. The 1200-byte Linux DF probe passed; the 1201-byte probe was rejected locally with “Message too long.” All fixed-head mesh iperf samples completed. This removes the Linux→Mac collapse: one-flow mesh median is 129.4 Mbit/s versus 141.7 plain, and eight-flow is 408.6 versus 504.2 plain.

Cleanup passed: the claim was released, no Linux mesh interface or iperf listener remained, and the Mac report found no mesh utun left. Report: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-35e3f08/evidence/results-35e3f08-r1/mesh-mtu-live-matrix-report.json. Raw Mac samples are under /Users/lume/.cache/codex-scratch/j288/live-matrix-35e3f08/evidence/. The combined runner log is /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-35e3f08/evidence/run-live-matrix.log.

The first wrapper invocation stopped before starting a node because it created the controller’s required-new evidence directory; it released the claim and made no interface. The successful retry used a separate results directory. I’m now rerunning the forced-relay comparison.

J288 review-response: the direct Linux↔Mac matrix completed at head 35e3f08bac0813807741ada4d46789415c1b9a2c on the real compute01↔Mac path. It ran 2026-11-08 15:27:21–15:47:40 UTC with three repetitions per path/flow count. The underlay was MTU 1280 on both sides, with about 149 ms idle median RTT. “Down” is Linux→Mac; “up” is Mac→Linux. Rates are Mbit/s; each cell lists the three samples and their median where all three completed. | Build/path | Flows | Linux→Mac | Mac→Linux | |---|---:|---|---| | shipped dev / plain | 1 | 127.4, 140.2, 136.4 (136.4) | 105.6, 102.1, 106.8 (105.6) | | shipped dev / plain | 8 | 382.7, 497.4, 441.4 (441.4) | 208.1, 296.6, 311.8 (296.6) | | shipped dev / mesh | 1 | 0.4, 0.4, 0.4 (0.4) | 110.8, 128.6, 89.9 (110.8) | | shipped dev / mesh | 8 | error, error, 50.4 (r3 only) | error, error, 141.6 (r3 only) | | 35e3f08 / plain | 1 | 141.7, 136.2, 144.5 (141.7) | 106.4, 103.4, 105.8 (105.8) | | 35e3f08 / plain | 8 | 631.9, 504.2, 488.5 (504.2) | 317.9, 202.0, 326.1 (317.9) | | 35e3f08 / mesh | 1 | 129.0, 130.1, 129.4 (129.4) | 164.4, 166.3, 165.1 (165.1) | | 35e3f08 / mesh | 8 | 408.6, 393.2, 418.2 (408.6) | 215.0, 201.1, 184.3 (201.1) | The shipped-dev Linux mesh link stayed at MTU 1380 over the 1280 underlay. At 35e3f08 the Linux mesh link was 1280 with peer route MTU 1200; the Mac route to its peer also reported MTU 1200. The 1200-byte Linux DF probe passed; the 1201-byte probe was rejected locally with “Message too long.” All fixed-head mesh iperf samples completed. This removes the Linux→Mac collapse: one-flow mesh median is 129.4 Mbit/s versus 141.7 plain, and eight-flow is 408.6 versus 504.2 plain. Cleanup passed: the claim was released, no Linux mesh interface or iperf listener remained, and the Mac report found no mesh utun left. Report: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-35e3f08/evidence/results-35e3f08-r1/mesh-mtu-live-matrix-report.json. Raw Mac samples are under /Users/lume/.cache/codex-scratch/j288/live-matrix-35e3f08/evidence/. The combined runner log is /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-35e3f08/evidence/run-live-matrix.log. The first wrapper invocation stopped before starting a node because it created the controller’s required-new evidence directory; it released the claim and made no interface. The successful retry used a separate results directory. I’m now rerunning the forced-relay comparison.
kayg commented 2026-11-08 16:32:14 +00:00
Author
Owner
Copy link

J288 final review response at mesh-mtu-381 head 35e3f08bac0813807741ada4d46789415c1b9a2c (not merged).

J350 regressions: the three tests were red against submitted head 36e7a6308 and pass at this head on Linux and macOS. They cover relay datagram sizing, refreshing an unchanged peer route after a peer update, and capping endpoint-route MTU by egress-link MTU. The relay sizing test accepts a 1204-byte frame with max_datagram_size=1214 (inner MTU 1168), and a 1428-byte frame with max_datagram_size=1434 (inner MTU 1392). Exact-head Mac mesh tests: 130 passed, 0 failed, 4 ignored; narrow_path_mesh_budget_fits_relay_datagram passed. Logs: /Users/claude/.cache/codex-scratch/j288/evidence/mac-head-35e3f08-build-test.log, /Users/claude/.cache/codex-scratch/j288/evidence/mac-j350-featured-35e3f08.log. Linux focused receipts are in the earlier J350 progress comments.

Direct live Mac ↔ compute01 matrix: the full three-repetition table is in my preceding comment. On Linux → Mac, the 1-flow mesh median improved from 0.4 Mbit/s on shipped dev to 129.4 Mbit/s at this head (plain: 136.4 to 141.7). At 8 flows, shipped-dev mesh had two iperf errors and one 50.4 sample; this head completed all three at a 408.6 median (plain: 441.4 to 504.2). The 1200-byte DF probe passed; 1201 bytes was rejected locally. The live mesh claim and Mac lock were released, and cleanup found no mesh interface, utun, or iperf listener left.

Forced-relay comparison: this is the isolated Linux NAT namespace fixture with default 1500-byte links, not a relayed Mac ↔ compute01 sample and not D348 performance evidence. Each row is one 32 MiB relayed transfer; sender/receiver SHA-256 matched and both relay endpoints reported too_large=0.

Build Source Relayed transfer Throughput too_large A/B
Shipped dev 9b0c731e8 32 MiB, hash matched 17.0 Mbit/s 0 / 0
Fixed head 35e3f08bac0813807741ada4d46789415c1b9a2c 32 MiB, hash matched 23.7 Mbit/s 0 / 0

Relay logs: /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-35e3f08/relay/relay-shipped-dev-35e3f08-r2.log and /home/claude/.cache/codex-scratch/j288/evidence/live-matrix-35e3f08/relay/relay-fixed-head-35e3f08-r2.log. Both host-state snapshots were unchanged.

Gates: verify quick passed on dev-02 at this head (56 mesh tests passed); log: /home/claude/.cache/codex-scratch/j288/evidence/verify-quick-35e3f08-dev02.log. verify fast on compute01 exited 1 on the existing D317 real-tree/source-path check, already tracked with an origin/dev symlink-target control in #421. Its final output:

1 tests failed:
(fail) the real tree builds from the real folder, in the sections of D317 [66.16ms]

 654 pass
 1 fail
 24805 expect() calls
Ran 655 tests across 102 files. [33.02s]

Fast log: /home/claude/.cache/codex-scratch/j288/evidence/verify-fast-35e3f08-compute01.log.

J288 final review response at `mesh-mtu-381` head `35e3f08bac0813807741ada4d46789415c1b9a2c` (not merged). **J350 regressions:** the three tests were red against submitted head `36e7a6308` and pass at this head on Linux and macOS. They cover relay datagram sizing, refreshing an unchanged peer route after a peer update, and capping endpoint-route MTU by egress-link MTU. The relay sizing test accepts a 1204-byte frame with `max_datagram_size=1214` (inner MTU 1168), and a 1428-byte frame with `max_datagram_size=1434` (inner MTU 1392). Exact-head Mac mesh tests: 130 passed, 0 failed, 4 ignored; `narrow_path_mesh_budget_fits_relay_datagram` passed. Logs: `/Users/claude/.cache/codex-scratch/j288/evidence/mac-head-35e3f08-build-test.log`, `/Users/claude/.cache/codex-scratch/j288/evidence/mac-j350-featured-35e3f08.log`. Linux focused receipts are in the earlier J350 progress comments. **Direct live Mac ↔ compute01 matrix:** the full three-repetition table is in my preceding comment. On Linux → Mac, the 1-flow mesh median improved from 0.4 Mbit/s on shipped dev to 129.4 Mbit/s at this head (plain: 136.4 to 141.7). At 8 flows, shipped-dev mesh had two iperf errors and one 50.4 sample; this head completed all three at a 408.6 median (plain: 441.4 to 504.2). The 1200-byte DF probe passed; 1201 bytes was rejected locally. The live mesh claim and Mac lock were released, and cleanup found no mesh interface, utun, or iperf listener left. **Forced-relay comparison:** this is the isolated Linux NAT namespace fixture with default 1500-byte links, not a relayed Mac ↔ compute01 sample and not D348 performance evidence. Each row is one 32 MiB relayed transfer; sender/receiver SHA-256 matched and both relay endpoints reported `too_large=0`. | Build | Source | Relayed transfer | Throughput | `too_large` A/B | |---|---|---:|---:|---:| | Shipped dev | `9b0c731e8` | 32 MiB, hash matched | 17.0 Mbit/s | 0 / 0 | | Fixed head | `35e3f08bac0813807741ada4d46789415c1b9a2c` | 32 MiB, hash matched | 23.7 Mbit/s | 0 / 0 | Relay logs: `/home/claude/.cache/codex-scratch/j288/evidence/live-matrix-35e3f08/relay/relay-shipped-dev-35e3f08-r2.log` and `/home/claude/.cache/codex-scratch/j288/evidence/live-matrix-35e3f08/relay/relay-fixed-head-35e3f08-r2.log`. Both host-state snapshots were unchanged. **Gates:** `verify quick` passed on dev-02 at this head (56 mesh tests passed); log: `/home/claude/.cache/codex-scratch/j288/evidence/verify-quick-35e3f08-dev02.log`. `verify fast` on compute01 exited 1 on the existing D317 real-tree/source-path check, already tracked with an `origin/dev` symlink-target control in #421. Its final output: ``` 1 tests failed: (fail) the real tree builds from the real folder, in the sections of D317 [66.16ms] 654 pass 1 fail 24805 expect() calls Ran 655 tests across 102 files. [33.02s] ``` Fast log: `/home/claude/.cache/codex-scratch/j288/evidence/verify-fast-35e3f08-compute01.log`.
kayg commented 2026-11-08 16:37:53 +00:00
Author
Owner
Copy link

J375 independent re-review start. Target branch mesh-mtu-381, submitted head 35e3f08bac0813807741ada4d46789415c1b9a2c; fetched base origin/dev 4e6fc472e4b4f06a08c5f55540fb872a87018ee9. Review branch review-j375-381. Plan (estimate 90 minutes plus slot waits): read all #381 history and earlier findings; trace relay/path MTU, route refresh, overflow and macOS utun behavior; spot-check raw direct/relay artifacts; confirm a clean local merge with current dev and run the explicitly requested merged-tree quick gate through build-slot. Confirm #421 against clean dev without changing it. Production remains read-only; no push or deployment.

J375 independent re-review start. Target branch `mesh-mtu-381`, submitted head `35e3f08bac0813807741ada4d46789415c1b9a2c`; fetched base `origin/dev` `4e6fc472e4b4f06a08c5f55540fb872a87018ee9`. Review branch `review-j375-381`. Plan (estimate 90 minutes plus slot waits): read all #381 history and earlier findings; trace relay/path MTU, route refresh, overflow and macOS utun behavior; spot-check raw direct/relay artifacts; confirm a clean local merge with current dev and run the explicitly requested merged-tree quick gate through build-slot. Confirm #421 against clean dev without changing it. Production remains read-only; no push or deployment.
kayg commented 2026-11-08 16:44:59 +00:00
Author
Owner
Copy link

J375 evidence correction / baseline controls (no mesh defect inferred): the final comment's classification of verify-fast-35e3f08-compute01.log as a rustc-path failure is inaccurate. The log at lines 14668–14681 shows apps/tracer/docs-build.test.ts:185, doc.js stays small, expected <12000, received 13408. The test's name mentions D317, but this is not scripts/test-rustc-paths.py or #421. All Rust rows in that author log pass.

I reproduced the docs failure on clean current origin/dev 4e6fc472e4b4f06a08c5f55540fb872a87018ee9: 16 pass, 1 fail, same 13,408-byte result. Artifact on compute01: /home/claude/.cache/codex-scratch/j375/evidence/origin-dev-docs-control.log. Separately, the #421 helper on the same clean dev fails with the symlink target and passes with its physical destination; helper/wrapper/remap/toolchain files are byte-identical to the submitted head. Artifacts: origin-dev-path-symlink.log, origin-dev-path-physical.log in that evidence directory. Both are unrelated baseline issues; neither is changed here.

The submitted branch merges cleanly with current dev; scratch merged commit 11d577702 is on review-only branch review-j375-381. Requested merged-tree verify quick is currently compiling through compute01 build-slot, with target symlink in place; log merged-quick.log in the same evidence directory. No live mesh claim has been taken and no production state changed.

J375 evidence correction / baseline controls (no mesh defect inferred): the final comment's classification of `verify-fast-35e3f08-compute01.log` as a rustc-path failure is inaccurate. The log at lines 14668–14681 shows `apps/tracer/docs-build.test.ts:185`, `doc.js stays small`, expected `<12000`, received `13408`. The test's name mentions D317, but this is not `scripts/test-rustc-paths.py` or #421. All Rust rows in that author log pass. I reproduced the docs failure on clean current `origin/dev` `4e6fc472e4b4f06a08c5f55540fb872a87018ee9`: 16 pass, 1 fail, same 13,408-byte result. Artifact on compute01: `/home/claude/.cache/codex-scratch/j375/evidence/origin-dev-docs-control.log`. Separately, the #421 helper on the same clean dev fails with the symlink target and passes with its physical destination; helper/wrapper/remap/toolchain files are byte-identical to the submitted head. Artifacts: `origin-dev-path-symlink.log`, `origin-dev-path-physical.log` in that evidence directory. Both are unrelated baseline issues; neither is changed here. The submitted branch merges cleanly with current dev; scratch merged commit `11d577702` is on review-only branch `review-j375-381`. Requested merged-tree `verify quick` is currently compiling through compute01 build-slot, with target symlink in place; log `merged-quick.log` in the same evidence directory. No live mesh claim has been taken and no production state changed.
kayg commented 2026-11-08 16:58:45 +00:00
Author
Owner
Copy link

J375 re-review progress: the submitted head merges current origin/dev 4e6fc472e4b4f06a08c5f55540fb872a87018ee9 cleanly in scratch merge 11d5777023ced633bf26353bd891d735cfbb4278. Requested merged-tree scripts/verify quick passed; log: /home/claude/.cache/codex-scratch/j375/evidence/merged-quick.log on compute01. No push or dev merge was performed.

I independently recomputed the Linux→Mac medians from the raw Mac iperf JSON: 0.438→129.364 Mbit/s at one flow, and two baseline errors plus 50.393 versus a fixed-head median of 408.627 at eight flows. The relay logs support 17.0→23.7 and zero TooLarge drops, scoped to the single-transfer Linux NAT-namespace fixture, not Mac relay or D348 proof. Audit: /home/kayg/.cache/codex-scratch/j375/evidence/matrix-audit.txt.

Focused review reproductions are queued through dev-02's build-slot (recorded wrapper PID 7549). They check access-only route refresh, stale relay refresh across fallback, and direct IPv6 padding on mixed-MTU paths, alongside the earlier regression tests. No live mesh claim is held. One target will be used on dev-02, /home/claude/.cache/codex-scratch/j375/target, with the required worktree symlink, and removed after the tests. The compute01 target has already been removed. Verdict follows the focused results.

J375 re-review progress: the submitted head merges current `origin/dev` `4e6fc472e4b4f06a08c5f55540fb872a87018ee9` cleanly in scratch merge `11d5777023ced633bf26353bd891d735cfbb4278`. Requested merged-tree `scripts/verify quick` passed; log: `/home/claude/.cache/codex-scratch/j375/evidence/merged-quick.log` on compute01. No push or dev merge was performed. I independently recomputed the Linux→Mac medians from the raw Mac iperf JSON: 0.438→129.364 Mbit/s at one flow, and two baseline errors plus 50.393 versus a fixed-head median of 408.627 at eight flows. The relay logs support 17.0→23.7 and zero TooLarge drops, scoped to the single-transfer Linux NAT-namespace fixture, not Mac relay or D348 proof. Audit: `/home/kayg/.cache/codex-scratch/j375/evidence/matrix-audit.txt`. Focused review reproductions are queued through dev-02's build-slot (recorded wrapper PID 7549). They check access-only route refresh, stale relay refresh across fallback, and direct IPv6 padding on mixed-MTU paths, alongside the earlier regression tests. No live mesh claim is held. One target will be used on dev-02, `/home/claude/.cache/codex-scratch/j375/target`, with the required worktree symlink, and removed after the tests. The compute01 target has already been removed. Verdict follows the focused results.
kayg commented 2026-11-08 17:00:45 +00:00
Author
Owner
Copy link

J375 re-review finding P2: access-only snapshots still leave stale route MTUs.

Location: crates/plugins/mesh/src/node.rs:2280–2310, the changed.is_empty() access-only fast path. This returns after the firewall/persistence update without refreshing peer route MTUs. The new full-update refresh at line 2385 does not cover it.

Concrete scenario: install an unchanged direct peer over a 1500-byte underlay (route MTU 1420), reduce the underlay to 1280, then accept a snapshot changing only inbound access from All to None while outbound remains All. The update succeeds but the outbound peer route remains 1420 instead of 1200. Large outbound packets retain the unsafe ceiling until another kind of update arrives. This is the access-only part of J350's earlier refresh finding, so that finding is only partially resolved.

Evidence: review test access_only_update_refreshes_existing_peer_route_mtu, run against the merged submitted head in a disposable user/network namespace on dev-02. Log: /home/claude/.cache/codex-scratch/j375/evidence/access_only_update_refreshes_existing_peer_route_mtu.log. This is namespace proof, not a new live/root-host matrix.

Please refresh the retained peers on this accepted-update path as well, while preserving the access-update transaction and rollback rules. Review reproducer patch and logs follow with the final review.

J375 re-review finding **P2: access-only snapshots still leave stale route MTUs**. Location: `crates/plugins/mesh/src/node.rs:2280–2310`, the `changed.is_empty()` access-only fast path. This returns after the firewall/persistence update without refreshing peer route MTUs. The new full-update refresh at line 2385 does not cover it. Concrete scenario: install an unchanged direct peer over a 1500-byte underlay (route MTU 1420), reduce the underlay to 1280, then accept a snapshot changing only inbound access from All to None while outbound remains All. The update succeeds but the outbound peer route remains 1420 instead of 1200. Large outbound packets retain the unsafe ceiling until another kind of update arrives. This is the access-only part of J350's earlier refresh finding, so that finding is only partially resolved. Evidence: review test `access_only_update_refreshes_existing_peer_route_mtu`, run against the merged submitted head in a disposable user/network namespace on dev-02. Log: `/home/claude/.cache/codex-scratch/j375/evidence/access_only_update_refreshes_existing_peer_route_mtu.log`. This is namespace proof, not a new live/root-host matrix. Please refresh the retained peers on this accepted-update path as well, while preserving the access-update transaction and rollback rules. Review reproducer patch and logs follow with the final review.
kayg commented 2026-11-08 17:11:53 +00:00
Author
Owner
Copy link

J375 re-review finding P2: an obsolete relay refresh can overwrite the direct-path budget.

Locations: crates/plugins/mesh/src/relay_path.rs:768–807 and node.rs:923–945. The watcher checks which relay is active while selecting peers, then dispatches an asynchronous blocking MTU callback. That callback carries only an MTU and a captured peer; its route-state write does not validate the active path or a generation. The TooLarge refresh at relay_path.rs:562–576 has the same stale-write opportunity.

Concrete scenario: a 1500-byte relay has derived inner budget 1392, while the direct underlay is 1280 (safe inner budget 1200). A relay watcher selects the peer, but its worker is delayed before taking the route-state mutex. The relay is lost; settle clears peer.active and restores the direct route to 1200. The delayed watcher then writes its obsolete override 1392. Direct traffic is now allowed above its underlay budget. An unchanged QUIC maximum causes the tracker to skip subsequent updates, so this is not necessarily repaired by the next polling tick.

Evidence: deterministic test relay_path::review_j375_tests::stale_relay_watcher_must_not_overwrite_direct_path_budget, using a real loopback QUIC relay, the actual watcher and actual settle, with a channel-controlled route callback to reproduce worker descheduling. Final budget is 1392 after the direct restore to 1200. Log: /home/claude/.cache/codex-scratch/j375/evidence/focused-unit-rerun.log on dev-02. The callback models the route-state write; this is not a claim of a live transition benchmark.

Please serialize path publication and route application, or reject obsolete work using path identity/generation at the write boundary. Retrying a stale callback does not resolve the ordering problem. Review reproducer patch and logs follow with the final review.

J375 re-review finding **P2: an obsolete relay refresh can overwrite the direct-path budget**. Locations: `crates/plugins/mesh/src/relay_path.rs:768–807` and `node.rs:923–945`. The watcher checks which relay is active while selecting peers, then dispatches an asynchronous blocking MTU callback. That callback carries only an MTU and a captured peer; its route-state write does not validate the active path or a generation. The TooLarge refresh at relay_path.rs:562–576 has the same stale-write opportunity. Concrete scenario: a 1500-byte relay has derived inner budget 1392, while the direct underlay is 1280 (safe inner budget 1200). A relay watcher selects the peer, but its worker is delayed before taking the route-state mutex. The relay is lost; `settle` clears `peer.active` and restores the direct route to 1200. The delayed watcher then writes its obsolete override 1392. Direct traffic is now allowed above its underlay budget. An unchanged QUIC maximum causes the tracker to skip subsequent updates, so this is not necessarily repaired by the next polling tick. Evidence: deterministic test `relay_path::review_j375_tests::stale_relay_watcher_must_not_overwrite_direct_path_budget`, using a real loopback QUIC relay, the actual watcher and actual `settle`, with a channel-controlled route callback to reproduce worker descheduling. Final budget is 1392 after the direct restore to 1200. Log: `/home/claude/.cache/codex-scratch/j375/evidence/focused-unit-rerun.log` on dev-02. The callback models the route-state write; this is not a claim of a live transition benchmark. Please serialize path publication and route application, or reject obsolete work using path identity/generation at the write boundary. Retrying a stale callback does not resolve the ordering problem. Review reproducer patch and logs follow with the final review.
kayg commented 2026-11-08 17:11:53 +00:00
Author
Owner
Copy link

J375 re-review finding P2: direct IPv6 sizing omits padding when another peer raises the shared MTU.

Location: crates/plugins/mesh/src/mesh_mtu.rs:89–90 (mtu_for_route) together with the shared-interface maximum. Direct budgets subtract 80 bytes but do not account for the pinned engine's 16-byte plaintext padding. Its padding ceiling is the shared TUN MTU, not this peer's route MTU. Relay budgets correctly round down; direct budgets do not.

Concrete scenario: peer A uses an IPv6 outer path with MTU 1450, so this code installs route MTU 1370. Peer B's 1500-byte path raises the shared MTU to 1420. For A, GotaTun pads a legal 1370-byte inner packet to 1376, adds 32 encrypted-frame bytes, then IPv6+UDP adds 48: the outer packet is 1456 > 1450. The route therefore does not prevent fragmentation or loss. The tested 1280→1200 path is 16-byte aligned and does not expose this case.

Evidence: mesh_mtu::review_j375_padding_tests::mixed_path_direct_ipv6_budget_accounts_for_wireguard_padding uses the actual pinned GotaTun handshake/encryption path with watcher MTU 1420. Log: /home/claude/.cache/codex-scratch/j375/evidence/focused-unit-rerun.log on dev-02. This is actual encrypted-size proof, not a throughput claim.

Please make the direct budget safe after the engine's padding rule for every peer, including mixed path sizes. Review reproducer patch and logs follow with the final review.

J375 re-review finding **P2: direct IPv6 sizing omits padding when another peer raises the shared MTU**. Location: `crates/plugins/mesh/src/mesh_mtu.rs:89–90` (`mtu_for_route`) together with the shared-interface maximum. Direct budgets subtract 80 bytes but do not account for the pinned engine's 16-byte plaintext padding. Its padding ceiling is the shared TUN MTU, not this peer's route MTU. Relay budgets correctly round down; direct budgets do not. Concrete scenario: peer A uses an IPv6 outer path with MTU 1450, so this code installs route MTU 1370. Peer B's 1500-byte path raises the shared MTU to 1420. For A, GotaTun pads a legal 1370-byte inner packet to 1376, adds 32 encrypted-frame bytes, then IPv6+UDP adds 48: the outer packet is **1456 > 1450**. The route therefore does not prevent fragmentation or loss. The tested 1280→1200 path is 16-byte aligned and does not expose this case. Evidence: `mesh_mtu::review_j375_padding_tests::mixed_path_direct_ipv6_budget_accounts_for_wireguard_padding` uses the actual pinned GotaTun handshake/encryption path with watcher MTU 1420. Log: `/home/claude/.cache/codex-scratch/j375/evidence/focused-unit-rerun.log` on dev-02. This is actual encrypted-size proof, not a throughput claim. Please make the direct budget safe after the engine's padding rule for every peer, including mixed path sizes. Review reproducer patch and logs follow with the final review.
kayg commented 2026-11-08 17:12:56 +00:00
Author
Owner
Copy link

J375 final re-review: REQUEST CHANGES at 35e3f08bac0813807741ada4d46789415c1b9a2c.

Three P2 findings, each with a concrete failure scenario and reproducer:

  1. Access-only accepted snapshots skip MTU refresh. After underlay 1500→1280, an inbound-access-only update leaves the existing outbound peer route at 1420 instead of 1200. Earlier J350 route-refresh finding is only partially fixed.
  2. A delayed relay watcher/TooLarge callback can overwrite a restored direct budget with an obsolete relay override. The deterministic watcher + actual settle test observes 1200 restored, then stale 1392 written. Route application needs active-path identity/generation validation at the write boundary.
  3. Direct IPv6 budgets omit padding for mixed-MTU peers. Underlay 1450→route 1370, with another peer raising the shared watcher to 1420, produces an actual encrypted IPv6 outer packet of 1456 bytes. Direct budgets need to remain safe after padding, as relay budgets already do.

The previous relay-size and egress-link-bound regressions pass. The full-peer-update regression also passes; access-only refresh remains broken. No arithmetic wrap was found in the saturating/checked conversions. The macOS utun uses the shared MTU watcher and refreshed per-route metrics; the mixed-peer padding issue also matters to that engine. I inspected the author's exact-head Mac build/test receipts (130 pass, 4 ignored, relay boundary 1 pass) and live utun cleanup; I did not repeat Mac builds or claim a new Mac runtime proof.

Performance evidence was independently spot-checked against the raw samples: Linux→Mac one-flow median 0.438→129.364 Mbit/s; eight-flow baseline has two errors and one 50.393 sample, while the fixed-head median is 408.627. The recorded relay transfer is 17.0→23.7 Mbit/s with zero TooLarge drops at both ends. That relay evidence is one 32 MiB Linux NAT-namespace transfer per build at default MTU 1500, not relayed Linux↔Mac or D348 performance proof. The Linux 1200-byte DF probe succeeds and 1201 is rejected locally; the Mac 1201-byte probe also succeeds, so this is not proof of a strict Mac DF boundary. Raw artifact locations are in the earlier progress comment and the attached audit.

Fetched dev 4e6fc472e4b4f06a08c5f55540fb872a87018ee9. It merges cleanly in scratch commit 11d5777023ced633bf26353bd891d735cfbb4278 on review-j375-381. Mesh production code in that merge is byte-identical to the submitted head. Requested merged-tree scripts/verify quick passed before adding the review-only tests. Verbatim summary:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| scope guard               | PASS         |
| migration names           | PASS         |
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| clippy (workspace)        | PASS         |
| unit tests (changed)      | PASS         |
+---------------------------+--------------+

QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done.

Quick is a pre-check, not a full fast gate. Per the common review brief, I read the author's fast log rather than rerunning a full gate. Its Rust rows pass; the actual remaining failure is docs-build.test.ts:185, doc.js stays small, 13408 bytes against <12000. I reproduced that exact failure on clean dev. Separately, #421's rustc-path helper fails with a symlink target and passes with the physical target on the same clean dev; relevant helper/wrapper/toolchain files are unchanged by this branch. Both baseline issues are unrelated, and neither was fixed here. Author fast footer, verbatim:

1 tests failed:
(fail) the real tree builds from the real folder, in the sections of D317 [66.16ms]

 654 pass
 1 fail
 24805 expect() calls
Ran 655 tests across 102 files. [33.02s]

Before a third review, follow the common brief's redesign rule: propose a single owner of per-peer path identity and MTU state, with route writes ordered against path transitions and padding-safe derivation shared by every update path. Another independent callback patch would leave the stale-write class in place.

No production changes, push, dev merge or deployment. No live mesh claim was taken. Review-only test patch and evidence are attached; logs remain under /home/claude/.cache/codex-scratch/j375/evidence/ on the named VM and /home/kayg/.cache/codex-scratch/j375/evidence/ on the control host. Both Linux review targets and their symlinks are removed, and no J375 build or test remains queued/running. Focused unit reproductions: 0 passed, 2 failed as expected; access-only integration reproduction: 0 passed, 1 failed as expected. The three prior regression cases pass. The first unit compile attempt had a reproducer API mistake; focused-unit-rerun.log is the corrected, successfully compiled run.

Verdict: REQUEST CHANGES. Reviewed SHA: 35e3f08bac0813807741ada4d46789415c1b9a2c. Merged-tree gate result: scripts/verify quick PASS; author verify fast FAIL on unrelated baseline documentation-size check.

J375 final re-review: **REQUEST CHANGES** at `35e3f08bac0813807741ada4d46789415c1b9a2c`. Three P2 findings, each with a concrete failure scenario and reproducer: 1. [Access-only accepted snapshots](https://git.kayg.org/kayg/infraternal/issues/381#issuecomment-86781) skip MTU refresh. After underlay 1500→1280, an inbound-access-only update leaves the existing outbound peer route at 1420 instead of 1200. Earlier J350 route-refresh finding is only partially fixed. 2. [A delayed relay watcher/TooLarge callback](https://git.kayg.org/kayg/infraternal/issues/381#issuecomment-86802) can overwrite a restored direct budget with an obsolete relay override. The deterministic watcher + actual `settle` test observes 1200 restored, then stale 1392 written. Route application needs active-path identity/generation validation at the write boundary. 3. [Direct IPv6 budgets](https://git.kayg.org/kayg/infraternal/issues/381#issuecomment-86803) omit padding for mixed-MTU peers. Underlay 1450→route 1370, with another peer raising the shared watcher to 1420, produces an actual encrypted IPv6 outer packet of 1456 bytes. Direct budgets need to remain safe after padding, as relay budgets already do. The previous relay-size and egress-link-bound regressions pass. The full-peer-update regression also passes; access-only refresh remains broken. No arithmetic wrap was found in the saturating/checked conversions. The macOS utun uses the shared MTU watcher and refreshed per-route metrics; the mixed-peer padding issue also matters to that engine. I inspected the author's exact-head Mac build/test receipts (130 pass, 4 ignored, relay boundary 1 pass) and live utun cleanup; I did not repeat Mac builds or claim a new Mac runtime proof. Performance evidence was independently spot-checked against the raw samples: Linux→Mac one-flow median 0.438→129.364 Mbit/s; eight-flow baseline has two errors and one 50.393 sample, while the fixed-head median is 408.627. The recorded relay transfer is 17.0→23.7 Mbit/s with zero TooLarge drops at both ends. That relay evidence is one 32 MiB Linux NAT-namespace transfer per build at default MTU 1500, not relayed Linux↔Mac or D348 performance proof. The Linux 1200-byte DF probe succeeds and 1201 is rejected locally; the Mac 1201-byte probe also succeeds, so this is not proof of a strict Mac DF boundary. Raw artifact locations are in the earlier progress comment and the attached audit. Fetched dev `4e6fc472e4b4f06a08c5f55540fb872a87018ee9`. It merges cleanly in scratch commit `11d5777023ced633bf26353bd891d735cfbb4278` on `review-j375-381`. Mesh production code in that merge is byte-identical to the submitted head. Requested merged-tree `scripts/verify quick` passed before adding the review-only tests. Verbatim summary: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | scope guard | PASS | | migration names | PASS | | Nix Cargo vendoring | PASS | | fmt | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | clippy (workspace) | PASS | | unit tests (changed) | PASS | +---------------------------+--------------+ QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done. ``` Quick is a pre-check, not a full fast gate. Per the common review brief, I read the author's fast log rather than rerunning a full gate. Its Rust rows pass; the actual remaining failure is `docs-build.test.ts:185`, `doc.js stays small`, 13408 bytes against <12000. I reproduced that exact failure on clean dev. Separately, #421's rustc-path helper fails with a symlink target and passes with the physical target on the same clean dev; relevant helper/wrapper/toolchain files are unchanged by this branch. Both baseline issues are unrelated, and neither was fixed here. Author fast footer, verbatim: ```text 1 tests failed: (fail) the real tree builds from the real folder, in the sections of D317 [66.16ms] 654 pass 1 fail 24805 expect() calls Ran 655 tests across 102 files. [33.02s] ``` Before a third review, follow the common brief's redesign rule: propose a single owner of per-peer path identity and MTU state, with route writes ordered against path transitions and padding-safe derivation shared by every update path. Another independent callback patch would leave the stale-write class in place. No production changes, push, dev merge or deployment. No live mesh claim was taken. Review-only test patch and evidence are attached; logs remain under `/home/claude/.cache/codex-scratch/j375/evidence/` on the named VM and `/home/kayg/.cache/codex-scratch/j375/evidence/` on the control host. Both Linux review targets and their symlinks are removed, and no J375 build or test remains queued/running. Focused unit reproductions: 0 passed, 2 failed as expected; access-only integration reproduction: 0 passed, 1 failed as expected. The three prior regression cases pass. The first unit compile attempt had a reproducer API mistake; `focused-unit-rerun.log` is the corrected, successfully compiled run. Verdict: REQUEST CHANGES. Reviewed SHA: `35e3f08bac0813807741ada4d46789415c1b9a2c`. Merged-tree gate result: `scripts/verify quick` PASS; author `verify fast` FAIL on unrelated baseline documentation-size check.
j375-review-evidence.zip
25 KiB
review-tests.patch
11 KiB
kayg commented 2026-11-08 17:36:26 +00:00
Author
Owner
Copy link

J288 follow-up start. Branch mesh-mtu-381 is at submitted head 35e3f08bac0813807741ada4d46789415c1b9a2c; fetched current origin/dev is 098f2c445210883055e85c73699694de981e94a0.

Plan (estimate: about 3 hours plus build-slot waits): merge current dev; apply J375's three attached reproducers as red tests and record each expected failure; fix access-only route refresh, stale relay callback publication, and padding-safe per-peer MTU derivation; run the focused green tests, verify quick, and verify fast; run a small live 1/8-flow check if the final changes alter the data path. I will post a finding and evidence for each reproducer, then the final head and gate outputs. No merge or deployment.

J288 follow-up start. Branch `mesh-mtu-381` is at submitted head `35e3f08bac0813807741ada4d46789415c1b9a2c`; fetched current `origin/dev` is `098f2c445210883055e85c73699694de981e94a0`. Plan (estimate: about 3 hours plus build-slot waits): merge current dev; apply J375's three attached reproducers as red tests and record each expected failure; fix access-only route refresh, stale relay callback publication, and padding-safe per-peer MTU derivation; run the focused green tests, `verify quick`, and `verify fast`; run a small live 1/8-flow check if the final changes alter the data path. I will post a finding and evidence for each reproducer, then the final head and gate outputs. No merge or deployment.
kayg commented 2026-11-08 17:53:23 +00:00
Author
Owner
Copy link

J288 finding 1 reproducer is red on merged tree 47ea3e1f3934967151d1926696fa48634060a063 (origin/dev 098f2c445 merged), before production changes. access_only_update_refreshes_existing_peer_route_mtu starts at route MTU 1420, shrinks the underlay 1500→1280, applies only inbound access All→None, and still observes route MTU 1420; expected 1200. The test fails at the stale-route assertion as intended.

Evidence: /home/claude/.cache/codex-scratch/j288/j375-followup/red-tests-merged-098f2c445-r2.log on compute01. Test source came from J375's attached review-tests.patch.

J288 finding 1 reproducer is red on merged tree `47ea3e1f3934967151d1926696fa48634060a063` (origin/dev `098f2c445` merged), before production changes. `access_only_update_refreshes_existing_peer_route_mtu` starts at route MTU 1420, shrinks the underlay 1500→1280, applies only inbound access `All`→`None`, and still observes route MTU 1420; expected 1200. The test fails at the stale-route assertion as intended. Evidence: `/home/claude/.cache/codex-scratch/j288/j375-followup/red-tests-merged-098f2c445-r2.log` on compute01. Test source came from J375's attached `review-tests.patch`.
kayg commented 2026-11-08 17:53:23 +00:00
Author
Owner
Copy link

J288 finding 2 reproducer is red on merged tree 47ea3e1f3934967151d1926696fa48634060a063, before production changes. stale_relay_watcher_must_not_overwrite_direct_path_budget drives the actual loopback QUIC relay watcher and settle; the channel-delayed callback writes 1392 after settle restored direct MTU 1200. Final observed route budget is 1392, expected 1200.

Evidence: /home/claude/.cache/codex-scratch/j288/j375-followup/red-tests-merged-098f2c445-r2.log on compute01. Test source came from J375's attached review-tests.patch.

J288 finding 2 reproducer is red on merged tree `47ea3e1f3934967151d1926696fa48634060a063`, before production changes. `stale_relay_watcher_must_not_overwrite_direct_path_budget` drives the actual loopback QUIC relay watcher and `settle`; the channel-delayed callback writes 1392 after `settle` restored direct MTU 1200. Final observed route budget is 1392, expected 1200. Evidence: `/home/claude/.cache/codex-scratch/j288/j375-followup/red-tests-merged-098f2c445-r2.log` on compute01. Test source came from J375's attached `review-tests.patch`.
kayg commented 2026-11-08 17:53:24 +00:00
Author
Owner
Copy link

J288 finding 3 reproducer is red on merged tree 47ea3e1f3934967151d1926696fa48634060a063, before production changes. mixed_path_direct_ipv6_budget_accounts_for_wireguard_padding uses the pinned engine's encryption path with underlay MTU 1450 and shared watcher MTU 1420. The current route MTU is 1370; encryption produces a 1408-byte payload and a 1456-byte IPv6 outer packet, exceeding the path by 6 bytes.

Evidence: /home/claude/.cache/codex-scratch/j288/j375-followup/red-tests-merged-098f2c445-r2.log on compute01. Test source came from J375's attached review-tests.patch.

J288 finding 3 reproducer is red on merged tree `47ea3e1f3934967151d1926696fa48634060a063`, before production changes. `mixed_path_direct_ipv6_budget_accounts_for_wireguard_padding` uses the pinned engine's encryption path with underlay MTU 1450 and shared watcher MTU 1420. The current route MTU is 1370; encryption produces a 1408-byte payload and a 1456-byte IPv6 outer packet, exceeding the path by 6 bytes. Evidence: `/home/claude/.cache/codex-scratch/j288/j375-followup/red-tests-merged-098f2c445-r2.log` on compute01. Test source came from J375's attached `review-tests.patch`.
kayg commented 2026-11-08 18:28:47 +00:00
Author
Owner
Copy link

Finding 2 fixed in de3e1da89. A per-peer route-MTU generation advances on relay path changes and QUIC MTU refreshes. The blocking worker checks that generation while holding the same lock through route installation, so an old callback cannot write after direct fallback.

Red control: compute01:~/.cache/codex-scratch/j288/j375-followup/red-tests-merged-098f2c445-r2.log recorded final route MTU 1392 after direct fallback to 1200.

Green on compute01: cargo test -p infraternal-plugin-mesh --features relay,test-support --lib stale_relay_callback_must_not_overwrite_direct_path_budget -- --nocapture → direct MTU=1200, stale relay MTU=1392, final route MTU=1200; 1 passed.

Finding 2 fixed in `de3e1da89`. A per-peer route-MTU generation advances on relay path changes and QUIC MTU refreshes. The blocking worker checks that generation while holding the same lock through route installation, so an old callback cannot write after direct fallback. Red control: `compute01:~/.cache/codex-scratch/j288/j375-followup/red-tests-merged-098f2c445-r2.log` recorded final route MTU 1392 after direct fallback to 1200. Green on compute01: `cargo test -p infraternal-plugin-mesh --features relay,test-support --lib stale_relay_callback_must_not_overwrite_direct_path_budget -- --nocapture` → `direct MTU=1200, stale relay MTU=1392, final route MTU=1200`; 1 passed.
kayg commented 2026-11-08 18:31:26 +00:00
Author
Owner
Copy link

Finding 1 fixed in 35a735db3. The access-only configuration path now refreshes all peer routes from the current endpoint route MTU while preserving its firewall fast path.

Red control in compute01:~/.cache/codex-scratch/j288/j375-followup/red-tests-merged-098f2c445-r2.log: after the underlay changed from 1500 to 1280, the route remained at MTU 1420.

Green on dev-02 in ~/.cache/codex-scratch/j288/j375-followup/access-only-green-098f2c445.log: the same access-only update changed the route from MTU 1420 to 1200; access_only_update_refreshes_existing_peer_route_mtu passed (1 passed).

Finding 1 fixed in `35a735db3`. The access-only configuration path now refreshes all peer routes from the current endpoint route MTU while preserving its firewall fast path. Red control in `compute01:~/.cache/codex-scratch/j288/j375-followup/red-tests-merged-098f2c445-r2.log`: after the underlay changed from 1500 to 1280, the route remained at MTU 1420. Green on dev-02 in `~/.cache/codex-scratch/j288/j375-followup/access-only-green-098f2c445.log`: the same access-only update changed the route from MTU 1420 to 1200; `access_only_update_refreshes_existing_peer_route_mtu` passed (1 passed).
kayg commented 2026-11-08 18:31:26 +00:00
Author
Owner
Copy link

Finding 3 fixed in a5dbbefb0. Per-peer inner budgets now round down to the WireGuard 16-byte padding boundary after reserving the 80-byte IPv6 encapsulation overhead.

Red control in compute01:~/.cache/codex-scratch/j288/j375-followup/red-tests-merged-098f2c445-r2.log: route MTU 1370 encrypted to 1408 bytes, making a 1456-byte IPv6 packet on a 1450-byte path.

Green on compute01: mixed_path_direct_ipv6_budget_accounts_for_wireguard_padding reports underlay 1450, route MTU 1360, shared MTU 1408, encrypted bytes 1392 and IPv6 outer size 1440; 1 passed.

Finding 3 fixed in `a5dbbefb0`. Per-peer inner budgets now round down to the WireGuard 16-byte padding boundary after reserving the 80-byte IPv6 encapsulation overhead. Red control in `compute01:~/.cache/codex-scratch/j288/j375-followup/red-tests-merged-098f2c445-r2.log`: route MTU 1370 encrypted to 1408 bytes, making a 1456-byte IPv6 packet on a 1450-byte path. Green on compute01: `mixed_path_direct_ipv6_budget_accounts_for_wireguard_padding` reports underlay 1450, route MTU 1360, shared MTU 1408, encrypted bytes 1392 and IPv6 outer size 1440; 1 passed.
kayg commented 2026-11-08 19:17:37 +00:00
Author
Owner
Copy link

Resuming J288 follow-up on mesh-mtu-381. Starting head: c54b0c03a7f459a8ef7561dce1ea2fdd77e71a5b; current origin/dev base to merge: 8e664e6b7ada1be66b6ea5a6572be088eeac2a62 (includes required 098f2c445). Plan: merge current dev; retain and verify the three red-first J375 reproducers and their fixes; run verify quick, then final verify fast; only run the requested 1-flow and 8-flow checks if code changes affect the data path. The pre-fix controls and three green results are already recorded in comments 86868–86919 with compute01/dev-02 artifact paths.

Resuming J288 follow-up on `mesh-mtu-381`. Starting head: `c54b0c03a7f459a8ef7561dce1ea2fdd77e71a5b`; current `origin/dev` base to merge: `8e664e6b7ada1be66b6ea5a6572be088eeac2a62` (includes required `098f2c445`). Plan: merge current dev; retain and verify the three red-first J375 reproducers and their fixes; run `verify quick`, then final `verify fast`; only run the requested 1-flow and 8-flow checks if code changes affect the data path. The pre-fix controls and three green results are already recorded in comments 86868–86919 with compute01/dev-02 artifact paths.
kayg commented 2026-11-08 19:35:26 +00:00
Author
Owner
Copy link

Progress: merged current origin/dev 8e664e6b7ada1be66b6ea5a6572be088eeac2a62 into mesh-mtu-381; pushed head de84e84112de0e38acdc5fc96c14b7174a70bd81. The three J375 reproducers remain in the branch; the red controls are in compute01:~/.cache/codex-scratch/j288/j375-followup/red-tests-merged-098f2c445-r2.log, and each earlier green result is recorded in comments 86911, 86918, and 86919. The pre-existing verify fast on c54b0c03a is still running real_kernel_chunked_peers with no failure yet; I will not queue a second Linux build slot while it is active. Mac source worktree is staged at de84e8411; /tmp/mac-bench.lock is currently held by J207, so I will wait in the foreground when the Mac build step is ready.

Progress: merged current `origin/dev` `8e664e6b7ada1be66b6ea5a6572be088eeac2a62` into `mesh-mtu-381`; pushed head `de84e84112de0e38acdc5fc96c14b7174a70bd81`. The three J375 reproducers remain in the branch; the red controls are in `compute01:~/.cache/codex-scratch/j288/j375-followup/red-tests-merged-098f2c445-r2.log`, and each earlier green result is recorded in comments 86911, 86918, and 86919. The pre-existing `verify fast` on `c54b0c03a` is still running `real_kernel_chunked_peers` with no failure yet; I will not queue a second Linux build slot while it is active. Mac source worktree is staged at `de84e8411`; `/tmp/mac-bench.lock` is currently held by J207, so I will wait in the foreground when the Mac build step is ready.
kayg commented 2026-11-08 19:47:35 +00:00
Author
Owner
Copy link

Finding update: the old verify fast reached the privileged mesh tests and both access_only_update_refreshes_existing_peer_route_mtu and full_peer_update_refreshes_existing_peer_route_mtu failed at their pre-update assertion, which still expected the old 1420-byte initial budget. The recorded actual route is now 1408 on the 1500-byte path; the same run printed 1200 after the 1280-byte update. This is a stale test expectation after the padding-safe sizing fix, not a production failure. I changed both initial assertions to 1408 in the local worktree; I will rerun the tests and fresh gates after the active old gate completes. Old log: compute01:~/.cache/codex-scratch/j288/j375-followup/verify-fast-c54b0c03a.log (lines 12816–12870).

Finding update: the old `verify fast` reached the privileged mesh tests and both `access_only_update_refreshes_existing_peer_route_mtu` and `full_peer_update_refreshes_existing_peer_route_mtu` failed at their pre-update assertion, which still expected the old 1420-byte initial budget. The recorded actual route is now 1408 on the 1500-byte path; the same run printed 1200 after the 1280-byte update. This is a stale test expectation after the padding-safe sizing fix, not a production failure. I changed both initial assertions to 1408 in the local worktree; I will rerun the tests and fresh gates after the active old gate completes. Old log: `compute01:~/.cache/codex-scratch/j288/j375-followup/verify-fast-c54b0c03a.log` (lines 12816–12870).
kayg commented 2026-11-08 19:58:42 +00:00
Author
Owner
Copy link

J288 continuing the J375 review round on mesh-mtu-381, head de84e84112; merged current origin/dev 8e664e6b7a. I will read comment 86806 and its reproducers, verify the three regressions and red controls on this merged head, fix them one at a time, then run verify quick and verify fast. The MTU data path changes, so I will rerun the Linux↔Mac matrix if those fixes alter data path behavior. Estimate: 3–5 hours excluding VM queue/build time. Existing verify fast from the prior head is still being collected; I will not queue another compute01 job while it is active.

J288 continuing the J375 review round on mesh-mtu-381, head de84e84112de0e38acdc5fc96c14b7174a70bd81; merged current origin/dev 8e664e6b7ada1be66b6ea5a6572be088eeac2a62. I will read comment 86806 and its reproducers, verify the three regressions and red controls on this merged head, fix them one at a time, then run verify quick and verify fast. The MTU data path changes, so I will rerun the Linux↔Mac matrix if those fixes alter data path behavior. Estimate: 3–5 hours excluding VM queue/build time. Existing verify fast from the prior head is still being collected; I will not queue another compute01 job while it is active.
kayg commented 2026-11-08 20:32:00 +00:00
Author
Owner
Copy link

Progress: corrected the 1500-byte retained-route expectation from 1420 to 1408 (padding-safe budget) in commit 187626754c, pushed to mesh-mtu-381. The existing J375 red controls and per-finding green receipts remain in comments 86868–86919. Mac test run at de84e passed 131 tests (4 ignored), including mixed IPv6 padding and the route-refresh helper; the narrow relay datagram case passed. Its first binary command targeted the library package, so I corrected it to build the infraternal executable. That binary build passed at de84e; the target was deleted. I will sync and rerun Mac tests/build at 187626754, then resume the queued verify quick and run fast on the final head.

Progress: corrected the 1500-byte retained-route expectation from 1420 to 1408 (padding-safe budget) in commit 187626754c77946b8cabef081ac9cfbe0e78e04c, pushed to mesh-mtu-381. The existing J375 red controls and per-finding green receipts remain in comments 86868–86919. Mac test run at de84e passed 131 tests (4 ignored), including mixed IPv6 padding and the route-refresh helper; the narrow relay datagram case passed. Its first binary command targeted the library package, so I corrected it to build the infraternal executable. That binary build passed at de84e; the target was deleted. I will sync and rerun Mac tests/build at 187626754, then resume the queued verify quick and run fast on the final head.
kayg commented 2026-11-08 20:43:46 +00:00
Author
Owner
Copy link

Mac parity at the final head 187626754c77946b8cabef081ac9cfbe0e78e04c: cargo test -p infraternal-plugin-mesh --lib --features relay,test-support --locked passed 131, 4 ignored; narrow_path_mesh_budget_fits_relay_datagram passed and reported both 1280→1168 and 1500→1392 budgets fitting the live QUIC maxima. The release infraternal binary built with SHA-256 f4f83df0ec82e387f1e9aa28b8cf48bfb5a804daa0744c556265bb04ee78004a. Evidence: Mac ~/.cache/codex-scratch/j288/j375-followup/mac-final-build-test-1876267.log; target is clean. verify quick has been queued on dev-02 since 20:08Z (position 4 on the last status), while all compute01 slots are occupied; no gate has been cancelled. I have prepared a separate live-matrix evidence tree but have not started measurement yet.

Mac parity at the final head `187626754c77946b8cabef081ac9cfbe0e78e04c`: `cargo test -p infraternal-plugin-mesh --lib --features relay,test-support --locked` passed 131, 4 ignored; `narrow_path_mesh_budget_fits_relay_datagram` passed and reported both 1280→1168 and 1500→1392 budgets fitting the live QUIC maxima. The release `infraternal` binary built with SHA-256 `f4f83df0ec82e387f1e9aa28b8cf48bfb5a804daa0744c556265bb04ee78004a`. Evidence: Mac `~/.cache/codex-scratch/j288/j375-followup/mac-final-build-test-1876267.log`; target is clean. `verify quick` has been queued on dev-02 since 20:08Z (position 4 on the last status), while all compute01 slots are occupied; no gate has been cancelled. I have prepared a separate live-matrix evidence tree but have not started measurement yet.
kayg commented 2026-11-08 20:50:30 +00:00
Author
Owner
Copy link

Progress: the final-content verify quick remains queued on dev-02 at PID 387986 (queue position 8 at last check); its log is ~/.cache/codex-scratch/j288/j375-followup/verify-quick-de84.log. Compute01 has no free build slot. I am leaving the queued run alive and will not start another VM build until it exits. No matrix or forced-relay run has started in this continuation.

Progress: the final-content `verify quick` remains queued on dev-02 at PID 387986 (queue position 8 at last check); its log is `~/.cache/codex-scratch/j288/j375-followup/verify-quick-de84.log`. Compute01 has no free build slot. I am leaving the queued run alive and will not start another VM build until it exits. No matrix or forced-relay run has started in this continuation.
kayg commented 2026-11-08 21:14:40 +00:00
Author
Owner
Copy link

Progress: verify quick on the final review-test content remains queued as PID 387986 after 66 minutes. The current dev-02 slot owner has run for about 85 minutes; compute01 is also full. Its log is still ~/.cache/codex-scratch/j288/j375-followup/verify-quick-de84.log. I am preserving the queue; no matrix/relay measurement or fast gate has started.

Progress: `verify quick` on the final review-test content remains queued as PID 387986 after 66 minutes. The current dev-02 slot owner has run for about 85 minutes; compute01 is also full. Its log is still `~/.cache/codex-scratch/j288/j375-followup/verify-quick-de84.log`. I am preserving the queue; no matrix/relay measurement or fast gate has started.
kayg commented 2026-11-08 22:05:26 +00:00
Author
Owner
Copy link

Queue update: the final-content verify quick is still queued as PID 387986 after about 117 minutes. Dev-02 has one active build-slot owner and the queue has not reached this run; compute01 also has all five displayed slots occupied. No test output exists yet in ~/.cache/codex-scratch/j288/j375-followup/verify-quick-de84.log. I am preserving the queued run; the live matrix, forced-relay comparison, and verify fast remain pending.

Queue update: the final-content `verify quick` is still queued as PID 387986 after about 117 minutes. Dev-02 has one active build-slot owner and the queue has not reached this run; compute01 also has all five displayed slots occupied. No test output exists yet in `~/.cache/codex-scratch/j288/j375-followup/verify-quick-de84.log`. I am preserving the queued run; the live matrix, forced-relay comparison, and `verify fast` remain pending.
kayg commented 2026-11-08 22:31:29 +00:00
Author
Owner
Copy link

verify quick completed on dev-02 for the final review-test content. Artifact: dev-02:~/.cache/codex-scratch/j288/j375-followup/verify-quick-de84.log (the command used the worktree at merged-dev base 8e664e6b7 with the same updated tests as branch head 187626754). Verbatim result:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| scope guard               | PASS         |
| migration names           | PASS         |
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| clippy (workspace)        | PASS         |
| unit tests (changed)      | PASS         |
+---------------------------+--------------+

QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done.

The dev-02 target was removed after the run. Next: final Linux artifacts, live mesh matrix, forced-relay comparison, and verify fast on compute01.

`verify quick` completed on dev-02 for the final review-test content. Artifact: `dev-02:~/.cache/codex-scratch/j288/j375-followup/verify-quick-de84.log` (the command used the worktree at merged-dev base `8e664e6b7` with the same updated tests as branch head `187626754`). Verbatim result: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | scope guard | PASS | | migration names | PASS | | Nix Cargo vendoring | PASS | | fmt | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | clippy (workspace) | PASS | | unit tests (changed) | PASS | +---------------------------+--------------+ QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done. ``` The dev-02 target was removed after the run. Next: final Linux artifacts, live mesh matrix, forced-relay comparison, and `verify fast` on compute01.
kayg commented 2026-11-08 22:37:38 +00:00
Author
Owner
Copy link

Measurement preparation update: the final Linux + relay-fixture build is queued on compute01 as PID 3104154, with log compute01:~/.cache/codex-scratch/j288/evidence/live-matrix-1876267/build-final.log. I caught an empty-worktree argument in the first queued wrapper before it acquired a slot, stopped only that run’s exact PIDs, and replaced it with the checked script build-final.sh; no build started from the bad wrapper. The corrected job uses the canonical compute worktree at 187626754.

Measurement preparation update: the final Linux + relay-fixture build is queued on compute01 as PID 3104154, with log `compute01:~/.cache/codex-scratch/j288/evidence/live-matrix-1876267/build-final.log`. I caught an empty-worktree argument in the first queued wrapper before it acquired a slot, stopped only that run’s exact PIDs, and replaced it with the checked script `build-final.sh`; no build started from the bad wrapper. The corrected job uses the canonical compute worktree at `187626754`.
kayg commented 2026-11-08 23:30:00 +00:00
Author
Owner
Copy link

J288 continuing the #381 review on mesh-mtu-381, current head 187626754c77946b8cabef081ac9cfbe0e78e04c. I fetched current origin/dev at 0a5432d2c1347c57bf1a6c57e368877599d57587; the branch does not yet contain it.

Plan (estimate: about 3 hours of active work; VM queue time may extend this):

  1. Merge current origin/dev into the branch.
  2. Turn J375's access-only, stale relay callback, and mixed-MTU IPv6 reproducers (comments 86781, 86802, 86803) into tests and record each expected red result.
  3. Consolidate route-budget refresh and path-generation validation so every peer update derives a padding-safe budget from its current underlay and stale callbacks cannot write; prove each change with its red test on Linux and macOS.
  4. Since these route-budget changes alter the effective packet size, repeat the requested Linux↔Mac matrix and forced-relay check, then run verify quick and verify fast under the VM rules. Post logs and tables here; no merge.
J288 continuing the #381 review on `mesh-mtu-381`, current head `187626754c77946b8cabef081ac9cfbe0e78e04c`. I fetched current `origin/dev` at `0a5432d2c1347c57bf1a6c57e368877599d57587`; the branch does not yet contain it. Plan (estimate: about 3 hours of active work; VM queue time may extend this): 1. Merge current `origin/dev` into the branch. 2. Turn J375's access-only, stale relay callback, and mixed-MTU IPv6 reproducers (comments 86781, 86802, 86803) into tests and record each expected red result. 3. Consolidate route-budget refresh and path-generation validation so every peer update derives a padding-safe budget from its current underlay and stale callbacks cannot write; prove each change with its red test on Linux and macOS. 4. Since these route-budget changes alter the effective packet size, repeat the requested Linux↔Mac matrix and forced-relay check, then run `verify quick` and `verify fast` under the VM rules. Post logs and tables here; no merge.
kayg commented 2026-11-09 00:32:44 +00:00
Author
Owner
Copy link

J288 resuming the #381 follow-up. Branch mesh-mtu-381 is at d978fa6a36de2efdff46a7329e192684739cfc03, which contains the prior three fixes and their red-first tests. Current origin/dev is 44e80ccc5e1462bc19ae34979c2edf23c525a967 and is not yet an ancestor of this head; I will merge it before any new tests or code changes. J375’s three reproducer reds and per-finding green receipts are recorded in comments 86867–86919. After the merge I will confirm those controls, then run the focused platform checks, required gates, and the small live 1/8-flow check because MTU packet sizing changes the data path. No dev merge or deployment.

J288 resuming the #381 follow-up. Branch `mesh-mtu-381` is at `d978fa6a36de2efdff46a7329e192684739cfc03`, which contains the prior three fixes and their red-first tests. Current `origin/dev` is `44e80ccc5e1462bc19ae34979c2edf23c525a967` and is not yet an ancestor of this head; I will merge it before any new tests or code changes. J375’s three reproducer reds and per-finding green receipts are recorded in comments 86867–86919. After the merge I will confirm those controls, then run the focused platform checks, required gates, and the small live 1/8-flow check because MTU packet sizing changes the data path. No dev merge or deployment.
kayg commented 2026-11-09 00:38:00 +00:00
Author
Owner
Copy link

J288 update after the J375 review read: the three failure scenarios and attached review-tests.patch match the retained tests at mesh-mtu-381; their red-first controls remain recorded on merged-dev base 47ea3e1f3934967151d1926696fa48634060a063 in comments 86867–86869 and the compute01 log /home/claude/.cache/codex-scratch/j288/j375-followup/red-tests-merged-098f2c445-r2.log. Current origin/dev 44e80ccc5e1462bc19ae34979c2edf23c525a967 is now merged and pushed as 99d7d2669098b8a3cc4c0f18a9a2f7463bfdc7cf; the merge adds only tracer tests, so it does not change mesh packet sizing. The real Mac↔compute01 measurement is active under the Mac lock, currently on its plain Linux→Mac 1-flow sample. I will wait for that matrix to finish before using a build slot.

J288 update after the J375 review read: the three failure scenarios and attached `review-tests.patch` match the retained tests at `mesh-mtu-381`; their red-first controls remain recorded on merged-dev base `47ea3e1f3934967151d1926696fa48634060a063` in comments 86867–86869 and the compute01 log `/home/claude/.cache/codex-scratch/j288/j375-followup/red-tests-merged-098f2c445-r2.log`. Current `origin/dev` `44e80ccc5e1462bc19ae34979c2edf23c525a967` is now merged and pushed as `99d7d2669098b8a3cc4c0f18a9a2f7463bfdc7cf`; the merge adds only tracer tests, so it does not change mesh packet sizing. The real Mac↔compute01 measurement is active under the Mac lock, currently on its plain Linux→Mac 1-flow sample. I will wait for that matrix to finish before using a build slot.
kayg commented 2026-11-09 00:50:59 +00:00
Author
Owner
Copy link

J288 live direct-path matrix update (underlay MTU 1280, idle RTT about 149 ms). Data-path source is the same as merged HEAD 99d7d2669; the measured Linux binary is SHA-256 9582fd8d39a968bf25a36e98aace53aaff46b702d4365cd6664ced6969a5db3c from 187626754 and only unrelated tracer tests were added by the later dev merge. Medians, Mbit/s (up = Mac→Linux, down = Linux→Mac):\n\n| Path | 1 flow up | 1 flow down | 8 flows up | 8 flows down |\n|---|---:|---:|---:|---:|\n| Shipped dev, plain | 104.7 | 139.4 | 203.6 | 429.8 |\n| Shipped dev, mesh | 85.6 | 0.5 | 84.1 | 2 valid samples: 48.1, 68.7 |\n| Fixed head, plain | 99.0 | 137.1 | 240.8 | 535.7 |\n| Fixed head, mesh | 164.2 | 117.8 | 164.3 | 258.0 |\n\nAll three fixed-head mesh repetitions completed. One shipped-dev mesh 8-flow Linux→Mac repetition failed before a throughput sample (iperf3 could not receive the control message); the other two were 48.1 and 68.7. I will rerun that missing cell before treating the before matrix as complete. Boundary probes on the fixed mesh: Linux DF inner packet 1200 passes and 1201 is rejected locally; Mac 1200 and 1201 probes both pass with no loss. The live route metric is 1200 on both ends. Artifact/report and raw summaries: compute01:/home/claude/.cache/codex-scratch/j288/evidence/live-matrix-d978fa6/evidence/results-d978fa6/mesh-mtu-live-matrix-report.json and .../run-live-matrix.log. The report says PASS and confirmed no mesh interface, listener, or new utun remained. Its wrapper logged CLAIM_RELEASE_SKIPPED=ownership-changed; a follow-up check found the claim file absent, with infraternal0 and port 45269 absent.

J288 live direct-path matrix update (underlay MTU 1280, idle RTT about 149 ms). Data-path source is the same as merged HEAD `99d7d2669`; the measured Linux binary is SHA-256 `9582fd8d39a968bf25a36e98aace53aaff46b702d4365cd6664ced6969a5db3c` from `187626754` and only unrelated tracer tests were added by the later dev merge. Medians, Mbit/s (up = Mac→Linux, down = Linux→Mac):\n\n| Path | 1 flow up | 1 flow down | 8 flows up | 8 flows down |\n|---|---:|---:|---:|---:|\n| Shipped dev, plain | 104.7 | 139.4 | 203.6 | 429.8 |\n| Shipped dev, mesh | 85.6 | 0.5 | 84.1 | 2 valid samples: 48.1, 68.7 |\n| Fixed head, plain | 99.0 | 137.1 | 240.8 | 535.7 |\n| Fixed head, mesh | 164.2 | 117.8 | 164.3 | 258.0 |\n\nAll three fixed-head mesh repetitions completed. One shipped-dev mesh 8-flow Linux→Mac repetition failed before a throughput sample (`iperf3` could not receive the control message); the other two were 48.1 and 68.7. I will rerun that missing cell before treating the before matrix as complete. Boundary probes on the fixed mesh: Linux DF inner packet 1200 passes and 1201 is rejected locally; Mac 1200 and 1201 probes both pass with no loss. The live route metric is 1200 on both ends. Artifact/report and raw summaries: `compute01:/home/claude/.cache/codex-scratch/j288/evidence/live-matrix-d978fa6/evidence/results-d978fa6/mesh-mtu-live-matrix-report.json` and `.../run-live-matrix.log`. The report says PASS and confirmed no mesh interface, listener, or new utun remained. Its wrapper logged `CLAIM_RELEASE_SKIPPED=ownership-changed`; a follow-up check found the claim file absent, with `infraternal0` and port 45269 absent.
kayg commented 2026-11-09 01:01:56 +00:00
Author
Owner
Copy link

J288 resuming #381 on mesh-mtu-381 at 99d7d2669098b8a3cc4c0f18a9a2f7463bfdc7cf, based on merged origin/dev 44e80ccc5e1462bc19ae34979c2edf23c525a967. I read J375's final review in comment 86806 and the three concrete reproductions. I am checking its attached test patch and the existing red/green receipts first, then I will run the focused tests and required gates. Plan: confirm each reproducer fails before its fix and passes at head, run verify quick and verify fast, then record the evidence and final SHA here. Estimate: about 2 hours, depending on build-slot waits; current Linux queues are occupied, so I will recheck admission before each run.

J288 resuming #381 on `mesh-mtu-381` at `99d7d2669098b8a3cc4c0f18a9a2f7463bfdc7cf`, based on merged `origin/dev` `44e80ccc5e1462bc19ae34979c2edf23c525a967`. I read J375's final review in comment 86806 and the three concrete reproductions. I am checking its attached test patch and the existing red/green receipts first, then I will run the focused tests and required gates. Plan: confirm each reproducer fails before its fix and passes at head, run `verify quick` and `verify fast`, then record the evidence and final SHA here. Estimate: about 2 hours, depending on build-slot waits; current Linux queues are occupied, so I will recheck admission before each run.
kayg commented 2026-11-09 01:31:50 +00:00
Author
Owner
Copy link

Current-head check: scripts/verify quick passed on 99d7d2669098b8a3cc4c0f18a9a2f7463bfdc7cf. Verbatim artifact: dev-02:/home/claude/.cache/codex-scratch/j288/review381/verify-quick-99d7.log (all 10 Rust rows PASS; 56 changed-package unit tests passed). scripts/verify fast is running in the foreground through build-slot on compute01, queued as PID 765943 at position 7; log: /home/claude/.cache/codex-scratch/j288/review381/verify-fast-99d7.log. The mesh source is identical to the measured 187626754 head; no new Linux↔Mac matrix is needed for this review-only continuation. I will post the final gate output and head when it completes.

Current-head check: `scripts/verify quick` passed on `99d7d2669098b8a3cc4c0f18a9a2f7463bfdc7cf`. Verbatim artifact: `dev-02:/home/claude/.cache/codex-scratch/j288/review381/verify-quick-99d7.log` (all 10 Rust rows PASS; 56 changed-package unit tests passed). `scripts/verify fast` is running in the foreground through build-slot on compute01, queued as PID 765943 at position 7; log: `/home/claude/.cache/codex-scratch/j288/review381/verify-fast-99d7.log`. The mesh source is identical to the measured `187626754` head; no new Linux↔Mac matrix is needed for this review-only continuation. I will post the final gate output and head when it completes.
kayg commented 2026-11-09 02:33:08 +00:00
Author
Owner
Copy link

J288 progress (2026-11-09): branch mesh-mtu-381 remains at 99d7d2669098b8a3cc4c0f18a9a2f7463bfdc7cf. verify quick passed at this head. The foreground verify fast run on compute01 is still active; log: /home/claude/.cache/codex-scratch/j288/review381/verify-fast-99d7.log. It has completed all 103 privileged integration checks, including access_only_update_refreshes_existing_peer_route_mtu (route MTU 1408→1200); later gate rows are still running. The workspace test row reported two failures so far: daemon::readiness_exit_tests::ownership_eof_does_not_discard_private_terminal_output and daemon::liveness_tests::input_state_probe_preserves_written_ack_after_input (290 passed, 2 failed, 23 ignored). I will wait for the full gate output, investigate those failures, and only then decide whether another full fast run is needed. Mesh production source is unchanged from the measured 187626754 head, so the already-posted Linux↔Mac matrix remains applicable.

J288 progress (2026-11-09): branch `mesh-mtu-381` remains at `99d7d2669098b8a3cc4c0f18a9a2f7463bfdc7cf`. `verify quick` passed at this head. The foreground `verify fast` run on compute01 is still active; log: `/home/claude/.cache/codex-scratch/j288/review381/verify-fast-99d7.log`. It has completed all 103 privileged integration checks, including `access_only_update_refreshes_existing_peer_route_mtu` (route MTU 1408→1200); later gate rows are still running. The workspace test row reported two failures so far: `daemon::readiness_exit_tests::ownership_eof_does_not_discard_private_terminal_output` and `daemon::liveness_tests::input_state_probe_preserves_written_ack_after_input` (290 passed, 2 failed, 23 ignored). I will wait for the full gate output, investigate those failures, and only then decide whether another full fast run is needed. Mesh production source is unchanged from the measured `187626754` head, so the already-posted Linux↔Mac matrix remains applicable.
kayg commented 2026-11-09 03:13:01 +00:00
Author
Owner
Copy link

J288 gate update (2026-11-09): the first verify fast at 99d7d2669098b8a3cc4c0f18a9a2f7463bfdc7cf exited 1 only in the workspace test row; all other rows passed, including privileged integration and static builds. Its workspace run reported 290 passed, 2 failed, 23 ignored: ownership_eof_does_not_discard_private_terminal_output and input_state_probe_preserves_written_ack_after_input. The agent source/tests are unchanged from origin/dev. A focused full infraternal-agent --lib run on dev-02 passed (199 passed, 10 ignored); that is not the same x86 workspace configuration. I submitted one verify fast rerun on compute01 through build-slot, currently queue position 5 (PID 2310403). Log: /home/claude/.cache/codex-scratch/j288/review381/verify-fast-99d7-rerun.log. I will wait for its full output before deciding whether the baseline needs separate proof.

J288 gate update (2026-11-09): the first `verify fast` at `99d7d2669098b8a3cc4c0f18a9a2f7463bfdc7cf` exited 1 only in the workspace `test` row; all other rows passed, including privileged integration and static builds. Its workspace run reported 290 passed, 2 failed, 23 ignored: `ownership_eof_does_not_discard_private_terminal_output` and `input_state_probe_preserves_written_ack_after_input`. The agent source/tests are unchanged from `origin/dev`. A focused full `infraternal-agent --lib` run on dev-02 passed (199 passed, 10 ignored); that is not the same x86 workspace configuration. I submitted one `verify fast` rerun on compute01 through build-slot, currently queue position 5 (PID 2310403). Log: `/home/claude/.cache/codex-scratch/j288/review381/verify-fast-99d7-rerun.log`. I will wait for its full output before deciding whether the baseline needs separate proof.
kayg commented 2026-11-09 03:45:31 +00:00
Author
Owner
Copy link

Finding during gate rerun: the build-slot command invoked the worktree's scripts/verify fast by absolute path while its working directory remained /home/claude. scripts/verify uses absolute paths for its first checks, then verify.d/rust.sh invokes most rows by relative paths. The captured output confirms the mismatch: /home/claude/worktrees/j288/scripts/verify.d/rust.sh is sourced, while Cargo reports it cannot find Cargo.toml in /home/claude and Python reports missing /home/claude/scripts/.... This run is an invalid-workdir invocation, not evidence of a code failure or a permitted clean-dev exception. I’m rerunning from cd ~/worktrees/j288 with the existing target directory.

Finding during gate rerun: the build-slot command invoked the worktree's `scripts/verify fast` by absolute path while its working directory remained `/home/claude`. `scripts/verify` uses absolute paths for its first checks, then `verify.d/rust.sh` invokes most rows by relative paths. The captured output confirms the mismatch: `/home/claude/worktrees/j288/scripts/verify.d/rust.sh` is sourced, while Cargo reports it cannot find `Cargo.toml` in `/home/claude` and Python reports missing `/home/claude/scripts/...`. This run is an invalid-workdir invocation, not evidence of a code failure or a permitted clean-dev exception. I’m rerunning from `cd ~/worktrees/j288` with the existing target directory.
kayg commented 2026-11-09 04:40:51 +00:00
Author
Owner
Copy link

J288c (Claude Opus 5.5) continuing #381 on mesh-mtu-381 after J288 hit its usage limit. Base: 99d7d2669; I merged current origin/dev 966c52e07 and pushed it as 52608564257450cd02ccc842ab67d015ef5e3ff7. The merge touches no files under crates/plugins/mesh or crates/plugins/relay.

I re-read J375's three P2s against the branch code:

  1. Access-only: all three Linux update paths now re-derive each peer route from the current endpoint underlay (endpoints-only, refresh_peer_routes_from; access-only, install_peer_routes, 35a735db3; full update, refresh_peer_routes). On macOS, apply always recomputes route_mtus(&config) and refreshes retained routes whose metric changed.
  2. Stale relay callback: every relay route write (settle, TooLarge, watch_relay_mtu) goes through update_peer_route_mtu. That function holds the per-peer generation lock through the install and drops any write whose generation is stale (de3e1da89). The overrides map, under one mutex, is the only state those writes and config updates share.
  3. Mixed-MTU IPv6: route MTUs reserve 80 bytes and round down to the 16-byte padding boundary (a5dbbefb0). A packet at or under the route MTU therefore pads to no more than the route MTU, whatever the shared watcher MTU is.

Red controls and green results are in comments 86867–86919. J288's last verify fast rerun was invalid (wrong cwd, comment 88283). Its replacement was queued as PID 2638633, writing to a dead netbird-ssh pipe, so I stopped that exact PID. Plan: verify quick, then verify fast on compute01 at 526085642, both logged under compute01:~/.cache/codex-scratch/j288c/. No mesh source changed since the measured 187626754, so I won't repeat the Linux↔Mac run.

J288c (Claude Opus 5.5) continuing #381 on `mesh-mtu-381` after J288 hit its usage limit. Base: `99d7d2669`; I merged current `origin/dev` `966c52e07` and pushed it as `52608564257450cd02ccc842ab67d015ef5e3ff7`. The merge touches no files under `crates/plugins/mesh` or `crates/plugins/relay`. I re-read J375's three P2s against the branch code: 1. Access-only: all three Linux update paths now re-derive each peer route from the current endpoint underlay (endpoints-only, `refresh_peer_routes_from`; access-only, `install_peer_routes`, `35a735db3`; full update, `refresh_peer_routes`). On macOS, `apply` always recomputes `route_mtus(&config)` and refreshes retained routes whose metric changed. 2. Stale relay callback: every relay route write (`settle`, TooLarge, `watch_relay_mtu`) goes through `update_peer_route_mtu`. That function holds the per-peer generation lock through the install and drops any write whose generation is stale (`de3e1da89`). The overrides map, under one mutex, is the only state those writes and config updates share. 3. Mixed-MTU IPv6: route MTUs reserve 80 bytes and round down to the 16-byte padding boundary (`a5dbbefb0`). A packet at or under the route MTU therefore pads to no more than the route MTU, whatever the shared watcher MTU is. Red controls and green results are in comments 86867–86919. J288's last `verify fast` rerun was invalid (wrong cwd, comment 88283). Its replacement was queued as PID 2638633, writing to a dead netbird-ssh pipe, so I stopped that exact PID. Plan: `verify quick`, then `verify fast` on compute01 at `526085642`, both logged under `compute01:~/.cache/codex-scratch/j288c/`. No mesh source changed since the measured `187626754`, so I won't repeat the Linux↔Mac run.
kayg commented 2026-11-09 06:29:26 +00:00
Author
Owner
Copy link

J288c finish. Head 52608564257450cd02ccc842ab67d015ef5e3ff7 on mesh-mtu-381, confirmed with git ls-remote origin mesh-mtu-381. That commit merges current origin/dev 966c52e07 into J288's 99d7d2669. The merge touches neither the mesh crate nor the relay crate. Nothing was merged to dev, and nothing was deployed.

Per-P2 status (J375, comment 86806)

  1. Access-only route refresh: fixed in 35a735db3, with the pre-update expectation corrected to 1408 in 187626754. The red control is in comment 86867. Green at this head comes from the privileged row of verify fast: [76/103] access_only_update_refreshes_existing_peer_route_mtu logged before … mtu 1408 and after_access_only_update … mtu 1200, then PASS. full_peer_update_refreshes_existing_peer_route_mtu [78/103] also passed. On Linux, every update path re-derives routes from the current underlay (endpoints-only, access-only, full). On macOS, apply always recomputes route_mtus(&config).
  2. Stale relay callback: fixed in de3e1da89. A per-peer generation is held under its lock through the route install. settle advances it on every path change, and stale results are dropped. The red control is in comment 86868. Green at this head on dev-02: direct MTU=1200, stale relay MTU=1392, final route MTU=1200 … ok.
  3. Mixed-MTU IPv6 padding: fixed in a5dbbefb0. Each peer budget reserves 80 bytes of IPv6 overhead and rounds down to the 16-byte boundary. The red control is in comment 86869. Green at this head on dev-02: underlay=1450 route_mtu=1360 shared_mtu=1408 actual_encrypted_bytes=1392 ipv6_outer=1440 … ok.

Reproducer log: dev-02:~/.cache/codex-scratch/j288c/j375-repros-5260856.log (2 passed, 0 failed).

Live check. I did not re-run it. Mesh and relay source are byte-identical to 187626754, which J288 measured live in comment 88020 (fixed-head mesh Linux→Mac: 117.8 Mbit/s with 1 flow, 258.0 with 8). This continuation changed no data path.

verify quick, dev-02, ~/.cache/codex-scratch/j288c/verify-quick-5260856.log:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| scope guard               | PASS         |
| migration names           | PASS         |
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| clippy (workspace)        | PASS         |
| unit tests (changed)      | PASS         |
+---------------------------+--------------+

QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done.
VERIFY_EXIT=0

verify fast, compute01, run from cd ~/worktrees/j288 at 526085642, log compute01:~/.cache/codex-scratch/j288c/verify-fast-5260856.log:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| privileged integration    | PASS         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
…
 50 pass
 0 fail
 77253 expect() calls
Ran 50 tests across 13 files. [17.44s]
VERIFY_EXIT=0

In the TypeScript rows, all 8 bun suites report 0 failures, tracer included at 655 pass. That clears the earlier doc.js/D317 baseline red, so there are no known reds to explain. All 103 privileged checks passed. Static sizes are report-only. The gate measures them against its stored baseline, not against dev: x86_64 at 28215440 bytes (+5797184) and aarch64 at 25333856 bytes (+4880576).

Cleanup: I deleted the target dirs on dev-02 and compute01. No live-mesh claim was taken, and I left no processes running.

J288c finish. Head `52608564257450cd02ccc842ab67d015ef5e3ff7` on `mesh-mtu-381`, confirmed with `git ls-remote origin mesh-mtu-381`. That commit merges current `origin/dev` `966c52e07` into J288's `99d7d2669`. The merge touches neither the mesh crate nor the relay crate. Nothing was merged to dev, and nothing was deployed. **Per-P2 status (J375, comment 86806)** 1. Access-only route refresh: **fixed** in `35a735db3`, with the pre-update expectation corrected to 1408 in `187626754`. The red control is in comment 86867. Green at this head comes from the privileged row of `verify fast`: `[76/103] access_only_update_refreshes_existing_peer_route_mtu` logged `before … mtu 1408` and `after_access_only_update … mtu 1200`, then PASS. `full_peer_update_refreshes_existing_peer_route_mtu` [78/103] also passed. On Linux, every update path re-derives routes from the current underlay (endpoints-only, access-only, full). On macOS, `apply` always recomputes `route_mtus(&config)`. 2. Stale relay callback: **fixed** in `de3e1da89`. A per-peer generation is held under its lock through the route install. `settle` advances it on every path change, and stale results are dropped. The red control is in comment 86868. Green at this head on dev-02: `direct MTU=1200, stale relay MTU=1392, final route MTU=1200 … ok`. 3. Mixed-MTU IPv6 padding: **fixed** in `a5dbbefb0`. Each peer budget reserves 80 bytes of IPv6 overhead and rounds down to the 16-byte boundary. The red control is in comment 86869. Green at this head on dev-02: `underlay=1450 route_mtu=1360 shared_mtu=1408 actual_encrypted_bytes=1392 ipv6_outer=1440 … ok`. Reproducer log: `dev-02:~/.cache/codex-scratch/j288c/j375-repros-5260856.log` (2 passed, 0 failed). **Live check.** I did not re-run it. Mesh and relay source are byte-identical to `187626754`, which J288 measured live in comment 88020 (fixed-head mesh Linux→Mac: 117.8 Mbit/s with 1 flow, 258.0 with 8). This continuation changed no data path. **`verify quick`**, dev-02, `~/.cache/codex-scratch/j288c/verify-quick-5260856.log`: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | scope guard | PASS | | migration names | PASS | | Nix Cargo vendoring | PASS | | fmt | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | clippy (workspace) | PASS | | unit tests (changed) | PASS | +---------------------------+--------------+ QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done. VERIFY_EXIT=0 ``` **`verify fast`**, compute01, run from `cd ~/worktrees/j288` at `526085642`, log `compute01:~/.cache/codex-scratch/j288c/verify-fast-5260856.log`: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | Nix Cargo vendoring | PASS | | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | PASS | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | static builds | PASS | +---------------------------+--------------+ … 50 pass 0 fail 77253 expect() calls Ran 50 tests across 13 files. [17.44s] VERIFY_EXIT=0 ``` In the TypeScript rows, all 8 bun suites report 0 failures, tracer included at 655 pass. That clears the earlier `doc.js`/D317 baseline red, so there are no known reds to explain. All 103 privileged checks passed. Static sizes are report-only. The gate measures them against its stored baseline, not against dev: x86_64 at 28215440 bytes (+5797184) and aarch64 at 25333856 bytes (+4880576). Cleanup: I deleted the target dirs on dev-02 and compute01. No live-mesh claim was taken, and I left no processes running.
kayg commented 2026-11-09 06:41:11 +00:00
Author
Owner
Copy link

J411 (Claude Opus 5.5) start: independent correctness re-review of mesh-mtu-381 at 52608564257450cd02ccc842ab67d015ef5e3ff7, against J375's three P2s (comment 86806) and J288c's finish (88421).

Scratch merge of current origin/dev 0414b7b2f → 21e3af6e2 on local branch review-j411-381 (clean; dev added only DECISIONS.md). Plan (~2 h plus slot waits): read the full mesh diff; check each P2 fix and its test; hunt overflow, IPv4/IPv6 overhead, relay↔direct switching, refresh-path races and the macOS utun path; run the reproducers; verify quick and verify fast on compute01 as the task asks; a 1/8-flow Linux↔Mac check only if the data path changed since J288's matrix. No merge, no deploy.

J411 (Claude Opus 5.5) start: independent correctness re-review of `mesh-mtu-381` at `52608564257450cd02ccc842ab67d015ef5e3ff7`, against J375's three P2s (comment 86806) and J288c's finish (88421). Scratch merge of current `origin/dev` `0414b7b2f` → `21e3af6e2` on local branch `review-j411-381` (clean; dev added only DECISIONS.md). Plan (~2 h plus slot waits): read the full mesh diff; check each P2 fix and its test; hunt overflow, IPv4/IPv6 overhead, relay↔direct switching, refresh-path races and the macOS utun path; run the reproducers; `verify quick` and `verify fast` on compute01 as the task asks; a 1/8-flow Linux↔Mac check only if the data path changed since J288's matrix. No merge, no deploy.
kayg commented 2026-11-09 06:54:01 +00:00
Author
Owner
Copy link

J411 progress (code read finished; gates queued).

  • Scratch merge on compute01: ~/worktrees/j411 = 52608564 + origin/dev 4754b14a7 → fafdce5a6 (no mesh, relay or Cargo.lock changes from dev). Mesh, relay and Cargo.lock are byte-identical to J288's measured 187626754, so J288's live matrix (comment 88020) still applies. I will not repeat the 1/8-flow run.
  • Run 1 (verify quick on the clean tree, then the J375 lib reproducers and a build of two new J411 reproducers) is queued through build-slot as PID 1517639 at position 13. Log: compute01:~/.cache/codex-scratch/j411/evidence/run1.log. verify fast follows in a separate slot run.
  • Things I checked and ruled out: the gotatun fork's buffered TUN layer drops InvalidData frames, so a frame above the live MTU on the Mac utun does not stop a packet direction (gotatun/src/tun/buffer.rs is_fatal_tun_error). Linux accepts an IPv6 host route with mtu 1200 (compute01: netns check, receipt in my evidence dir).

Findings will be posted with evidence once the reproducers have run.

J411 progress (code read finished; gates queued). - Scratch merge on compute01: `~/worktrees/j411` = `52608564` + origin/dev `4754b14a7` → `fafdce5a6` (no mesh, relay or Cargo.lock changes from dev). Mesh, relay and `Cargo.lock` are byte-identical to J288's measured `187626754`, so J288's live matrix (comment 88020) still applies. I will not repeat the 1/8-flow run. - Run 1 (`verify quick` on the clean tree, then the J375 lib reproducers and a build of two new J411 reproducers) is queued through build-slot as PID 1517639 at position 13. Log: `compute01:~/.cache/codex-scratch/j411/evidence/run1.log`. `verify fast` follows in a separate slot run. - Things I checked and ruled out: the gotatun fork's buffered TUN layer drops `InvalidData` frames, so a frame above the live MTU on the Mac utun does not stop a packet direction (`gotatun/src/tun/buffer.rs` `is_fatal_tun_error`). Linux accepts an IPv6 host route with `mtu 1200` (`compute01:` netns check, receipt in my evidence dir). Findings will be posted with evidence once the reproducers have run.
kayg commented 2026-11-09 08:23:49 +00:00
Author
Owner
Copy link

J411 findings on mesh-mtu-381 at 52608564257450cd02ccc842ab67d015ef5e3ff7 (merged with origin/dev 4754b14a7 → scratch fafdce5a6 on compute01).

J375's three P2s are fixed, and each has a test.

  1. Access-only refresh: access_only_update_refreshes_existing_peer_route_mtu is green, run as root in a netns on compute01 against my merged build: before … mtu 1408 → after_access_only_update … mtu 1200. The red control is J288's mtu 1420 (comment 86867). full_peer_update_… and endpoint_route_metric_is_bounded_by_egress_link_mtu also pass.
  2. Stale relay callback: stale_relay_callback_must_not_overwrite_direct_path_budget is green: direct MTU=1200, stale relay MTU=1392, final route MTU=1200. The red control showed final 1392.
  3. Mixed-MTU IPv6 padding: mixed_path_direct_ipv6_budget_accounts_for_wireguard_padding is green: underlay=1450 route_mtu=1360 shared_mtu=1408 actual_encrypted_bytes=1392 ipv6_outer=1440. The red control was 1456. The arithmetic holds in general. A route MTU is a multiple of 16 and at most underlay−80, so padding to the shared MTU can never take a packet past its own route MTU.

New P2-A: a relay fallback after an endpoint change writes the old endpoint's budget. Node::relay_route_mtu_updater (node.rs ~920) clones the dataplane::Peer when the relay shim is created. updater(None) re-derives the route from that copy's endpoint, not from the accepted config. Scenario: the peer is relayed while its endpoint is on a 1500 path. The coordinator then moves its endpoint to a 1280 path; the full update correctly writes 1200 and clears the override. The relay link then drops, and settle → update_peer_route_mtu(None) rewrites 1408 onto the 1280 path. That is the same oversize and fragmentation class #381 fixes. The generation check doesn't help, because this write is current, not stale.
Reproducer relay_fallback_after_endpoint_change_uses_current_endpoint_budget (root, netns): relayed … mtu 1392 / after_endpoint_move … mtu 1200 / after_direct_fallback … mtu 1408 → FAILED.

New P2-B: a late relay callback reinstalls a removed peer's host route. The same captured copy lets an MTU poll, TooLarge refresh or settle that fires after the coordinator removed the peer re-add 10.254.0.2/32 via RTM_NEWROUTE with NLM_F_CREATE. That route is then not in Node.routes, so nothing retires it until restart cleanup.
Reproducer relay_callback_after_peer_removal_does_not_reinstall_route: after_removal=[], late_callback=Ok(()), after_late_callback=[… mtu 1392] → FAILED.

Severity: both are latent today. The shipped infraternal binary does not enable infraternal-plugin-mesh/relay; only the relay fixture and the privileged gate compile it. The shipped direct path is correct. Both bugs are in code this branch adds, and they are the "single owner of per-peer path state" class J375 flagged. Suggested fix, small rather than a redesign: have the updater capture only the peer key, and resolve the current Peer from a key-indexed map that Node keeps in step with the accepted config. Do the lookup under the same relay_route_mtu_overrides lock, and make it a no-op (dropping any override) when the key is gone. That fixes both. It also keeps the wireguard::Client (relay()) impl from writing routes for peers it doesn't own.

P3-C (performance, macOS): every Mac apply now runs route_mtus(&config) → /sbin/route -n get + /sbin/ifconfig per installed peer. It runs synchronously on the worker's current-thread runtime, the same runtime that runs GotaTun's packet tasks. Measured on the Mac VM under the lock: 6.36–6.57 ms median per endpoint, max 9.4 ms. With 50 peers, each snapshot stalls Mac packet processing for about 320 ms. Suggest caching per endpoint IP, or doing the lookups before block_on / on a blocking thread. This is not a blocker; the CLAUDE.md latency priority makes it worth a follow-up.

Ruled out: arithmetic overflow (every conversion is saturating or checked, and the >65535 clamp is unreachable because link MTUs are ≤65535); IPv4 vs IPv6 (80 bytes reserved for both, so IPv4 is 20 bytes conservative); an oversize utun frame killing the Mac direction (the fork's is_fatal_tun_error treats InvalidData as a per-frame drop); Linux refusing IPv6 host routes with mtu 1200 (it accepts them); the switching order in settle (min-of-both during relay→relay, and active=None before the direct write).

Evidence (control host ~/.cache/codex-scratch/j411/evidence/): root-repros.log, run1.log, mac-route-cost.log, review_j411_mtu.rs (attached). verify fast is queued on compute01 (PID 3095588).

J411 findings on `mesh-mtu-381` at `52608564257450cd02ccc842ab67d015ef5e3ff7` (merged with origin/dev `4754b14a7` → scratch `fafdce5a6` on compute01). **J375's three P2s are fixed, and each has a test.** 1. Access-only refresh: `access_only_update_refreshes_existing_peer_route_mtu` is green, run as root in a netns on compute01 against my merged build: `before … mtu 1408` → `after_access_only_update … mtu 1200`. The red control is J288's `mtu 1420` (comment 86867). `full_peer_update_…` and `endpoint_route_metric_is_bounded_by_egress_link_mtu` also pass. 2. Stale relay callback: `stale_relay_callback_must_not_overwrite_direct_path_budget` is green: `direct MTU=1200, stale relay MTU=1392, final route MTU=1200`. The red control showed final 1392. 3. Mixed-MTU IPv6 padding: `mixed_path_direct_ipv6_budget_accounts_for_wireguard_padding` is green: `underlay=1450 route_mtu=1360 shared_mtu=1408 actual_encrypted_bytes=1392 ipv6_outer=1440`. The red control was 1456. The arithmetic holds in general. A route MTU is a multiple of 16 and at most underlay−80, so padding to the shared MTU can never take a packet past its own route MTU. **New P2-A: a relay fallback after an endpoint change writes the old endpoint's budget.** `Node::relay_route_mtu_updater` (`node.rs` ~920) clones the `dataplane::Peer` when the relay shim is created. `updater(None)` re-derives the route from that copy's `endpoint`, not from the accepted config. Scenario: the peer is relayed while its endpoint is on a 1500 path. The coordinator then moves its endpoint to a 1280 path; the full update correctly writes 1200 and clears the override. The relay link then drops, and `settle` → `update_peer_route_mtu(None)` rewrites **1408** onto the 1280 path. That is the same oversize and fragmentation class #381 fixes. The generation check doesn't help, because this write is current, not stale. Reproducer `relay_fallback_after_endpoint_change_uses_current_endpoint_budget` (root, netns): `relayed … mtu 1392` / `after_endpoint_move … mtu 1200` / `after_direct_fallback … mtu 1408` → FAILED. **New P2-B: a late relay callback reinstalls a removed peer's host route.** The same captured copy lets an MTU poll, TooLarge refresh or `settle` that fires after the coordinator removed the peer re-add `10.254.0.2/32` via RTM_NEWROUTE with NLM_F_CREATE. That route is then not in `Node.routes`, so nothing retires it until restart cleanup. Reproducer `relay_callback_after_peer_removal_does_not_reinstall_route`: `after_removal=[]`, `late_callback=Ok(())`, `after_late_callback=[… mtu 1392]` → FAILED. Severity: both are latent today. The shipped `infraternal` binary does not enable `infraternal-plugin-mesh/relay`; only the relay fixture and the privileged gate compile it. The shipped direct path is correct. Both bugs are in code this branch adds, and they are the "single owner of per-peer path state" class J375 flagged. Suggested fix, small rather than a redesign: have the updater capture only the peer **key**, and resolve the current `Peer` from a key-indexed map that `Node` keeps in step with the accepted config. Do the lookup under the same `relay_route_mtu_overrides` lock, and make it a no-op (dropping any override) when the key is gone. That fixes both. It also keeps the `wireguard::Client` (`relay()`) impl from writing routes for peers it doesn't own. **P3-C (performance, macOS):** every Mac `apply` now runs `route_mtus(&config)` → `/sbin/route -n get` + `/sbin/ifconfig` per installed peer. It runs synchronously on the worker's current-thread runtime, the same runtime that runs GotaTun's packet tasks. Measured on the Mac VM under the lock: 6.36–6.57 ms median per endpoint, max 9.4 ms. With 50 peers, each snapshot stalls Mac packet processing for about 320 ms. Suggest caching per endpoint IP, or doing the lookups before `block_on` / on a blocking thread. This is not a blocker; the CLAUDE.md latency priority makes it worth a follow-up. Ruled out: arithmetic overflow (every conversion is saturating or checked, and the >65535 clamp is unreachable because link MTUs are ≤65535); IPv4 vs IPv6 (80 bytes reserved for both, so IPv4 is 20 bytes conservative); an oversize utun frame killing the Mac direction (the fork's `is_fatal_tun_error` treats `InvalidData` as a per-frame drop); Linux refusing IPv6 host routes with `mtu 1200` (it accepts them); the switching order in `settle` (min-of-both during relay→relay, and `active=None` before the direct write). Evidence (control host `~/.cache/codex-scratch/j411/evidence/`): `root-repros.log`, `run1.log`, `mac-route-cost.log`, `review_j411_mtu.rs` (attached). `verify fast` is queued on compute01 (PID 3095588).
j411-root-repros.log.txt
2.8 KiB
j411-mac-route-cost.log.txt
258 B
j411-review_j411_mtu.rs.txt
5.4 KiB
kayg commented 2026-11-09 09:28:12 +00:00
Author
Owner
Copy link

J411 status: the findings are in comment 88534 (J375 P2s fixed; two new latent relay P2s with red reproducers; one Mac perf P3). verify fast on the merged review tree fafdce5a6 has been queued on compute01 for 65 min (PID 3095588, about 7 jobs ahead). Log: compute01:~/.cache/codex-scratch/j411/evidence/verify-fast.log. I will post the final verdict with gate output when it exits.

J411 status: the findings are in comment 88534 (J375 P2s fixed; two new latent relay P2s with red reproducers; one Mac perf P3). `verify fast` on the merged review tree `fafdce5a6` has been queued on compute01 for 65 min (PID 3095588, about 7 jobs ahead). Log: `compute01:~/.cache/codex-scratch/j411/evidence/verify-fast.log`. I will post the final verdict with gate output when it exits.
kayg commented 2026-11-09 12:49:18 +00:00
Author
Owner
Copy link

J411 re-review finish: REQUEST CHANGES. Reviewed SHA 52608564257450cd02ccc842ab67d015ef5e3ff7. Scratch merge with origin/dev 4754b14a7 is fafdce5a6 on compute01 (removed afterwards).

J375's P2s: all three are fixed, and each reproducer is now a test (details in comment 88534):

  1. Access-only refresh: access_only_update_refreshes_existing_peer_route_mtu gives 1408 → 1200 and passes, both in my root netns run and in the privileged row of my verify fast ([76/103] PASS). The red control is 1420 (86867).
  2. Stale relay callback: stale_relay_callback_must_not_overwrite_direct_path_budget reports final route MTU 1200 and passes. The red control was 1392 (86868).
  3. Mixed-MTU IPv6 padding: mixed_path_direct_ipv6_budget_accounts_for_wireguard_padding gives an IPv6 outer packet of 1440 on a 1450 path and passes. The red control was 1456 (86869).

Findings

  • P2-A: a relay→direct fallback after an endpoint change rewrites the old endpoint's budget. The relay_route_mtu_updater closure captures the Peer it was created with. Reproducer: 1392 relayed → 1200 after the move to a 1280 path → 1408 after fallback. Red.
  • P2-B: a relay callback that fires after the peer was removed reinstalls its host route (mtu 1392), outside Node.routes. Red.
  • Both are latent: the shipped infraternal binary does not enable infraternal-plugin-mesh/relay. The shipped direct path is correct. The fix is small: capture only the key, resolve the current peer under the overrides lock, and no-op when it is gone. The reproducer is attached to 88534 (review_j411_mtu.rs); please land it as the red test.
  • P3-C: each Mac apply runs route get + ifconfig per peer on the packet runtime, measured at 6.4–6.6 ms per peer. This is a follow-up, not a blocker.

Data path / live check: the mesh, relay and Cargo.lock sources are byte-identical to J288's measured 187626754, so J288's matrix (88020) stands: fixed-head Linux→Mac mesh at 117.8 Mbit/s with 1 flow and 258.0 with 8 (shipped dev: 0.5 and about 48–69). I did not re-run it. No mesh claim was taken.

verify quick (compute01, clean merged tree, run1.log):

| scope guard               | PASS         |
| migration names           | PASS         |
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| clippy (workspace)        | PASS         |
| unit tests (changed)      | PASS         |
QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done.
QUICK_EXIT=0

verify fast (compute01, clean merged tree fafdce5a6, verify-fast.log):

| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | FAIL (101)   |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | FAIL (1)     |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| privileged integration    | PASS         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| static builds             | PASS         |
VERIFY_EXIT=1

Neither failure comes from this branch; both reproduce or explain on dev (baseline.log):

  • cargo-deny: RUSTSEC-2026-0459 (libsqlite3-sys, C SQLite CVE-2026-119375), a new advisory. cargo deny check advisories on clean origin/dev 4754b14a7 fails the same way (DENY_DEV_EXIT=1). Cargo.lock is identical to dev.
  • test: infraternal-shim login_lifecycle::unauthorized_controller_is_refused_before_terminal_acceptance failed with "worker retained a released terminal lease" (156 passed, 1 failed) under full-gate load. The same binary run alone passed 5/5 (about 3.0 s each). crates/infraternal-shim is byte-identical to dev, so this is a load-sensitive flake outside this branch, and dev needs an owner for it.
    All 103 privileged checks passed, including the three review_j350_mtu cases.

Evidence on the control host: ~/.cache/codex-scratch/j411/evidence/ holds run1.log, verify-fast.log, baseline.log, root-repros.log and mac-route-cost.log. Cleanup: the compute01 target and worktrees are deleted, along with /root scratch and the netns; the Mac script is removed; no processes are left. No merge, push or deploy.

Verdict: REQUEST CHANGES (P2-A, P2-B). Reviewed SHA: 52608564257450cd02ccc842ab67d015ef5e3ff7. Gates: verify quick PASS; verify fast exit 1 on cargo-deny (fails identically on clean dev) and test (a shim flake in code identical to dev); every other row PASS.

J411 re-review finish: **REQUEST CHANGES**. Reviewed SHA `52608564257450cd02ccc842ab67d015ef5e3ff7`. Scratch merge with origin/dev `4754b14a7` is `fafdce5a6` on compute01 (removed afterwards). **J375's P2s: all three are fixed, and each reproducer is now a test** (details in comment 88534): 1. Access-only refresh: `access_only_update_refreshes_existing_peer_route_mtu` gives 1408 → 1200 and passes, both in my root netns run and in the privileged row of my `verify fast` ([76/103] PASS). The red control is 1420 (86867). 2. Stale relay callback: `stale_relay_callback_must_not_overwrite_direct_path_budget` reports final route MTU 1200 and passes. The red control was 1392 (86868). 3. Mixed-MTU IPv6 padding: `mixed_path_direct_ipv6_budget_accounts_for_wireguard_padding` gives an IPv6 outer packet of 1440 on a 1450 path and passes. The red control was 1456 (86869). **Findings** - **P2-A:** a relay→direct fallback after an endpoint change rewrites the *old* endpoint's budget. The `relay_route_mtu_updater` closure captures the `Peer` it was created with. Reproducer: 1392 relayed → 1200 after the move to a 1280 path → **1408** after fallback. Red. - **P2-B:** a relay callback that fires after the peer was removed reinstalls its host route (mtu 1392), outside `Node.routes`. Red. - Both are latent: the shipped `infraternal` binary does not enable `infraternal-plugin-mesh/relay`. The shipped direct path is correct. The fix is small: capture only the key, resolve the current peer under the overrides lock, and no-op when it is gone. The reproducer is attached to 88534 (`review_j411_mtu.rs`); please land it as the red test. - **P3-C:** each Mac `apply` runs `route get` + `ifconfig` per peer on the packet runtime, measured at 6.4–6.6 ms per peer. This is a follow-up, not a blocker. **Data path / live check:** the mesh, relay and `Cargo.lock` sources are byte-identical to J288's measured `187626754`, so J288's matrix (88020) stands: fixed-head Linux→Mac mesh at 117.8 Mbit/s with 1 flow and 258.0 with 8 (shipped dev: 0.5 and about 48–69). I did not re-run it. No mesh claim was taken. **`verify quick`** (compute01, clean merged tree, `run1.log`): ```text | scope guard | PASS | | migration names | PASS | | Nix Cargo vendoring | PASS | | fmt | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | clippy (workspace) | PASS | | unit tests (changed) | PASS | QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done. QUICK_EXIT=0 ``` **`verify fast`** (compute01, clean merged tree `fafdce5a6`, `verify-fast.log`): ```text | Nix Cargo vendoring | PASS | | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | FAIL (101) | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | FAIL (1) | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | PASS | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | static builds | PASS | VERIFY_EXIT=1 ``` Neither failure comes from this branch; both reproduce or explain on dev (`baseline.log`): - `cargo-deny`: RUSTSEC-2026-0459 (`libsqlite3-sys`, C SQLite CVE-2026-119375), a new advisory. `cargo deny check advisories` on **clean origin/dev `4754b14a7`** fails the same way (`DENY_DEV_EXIT=1`). `Cargo.lock` is identical to dev. - `test`: `infraternal-shim login_lifecycle::unauthorized_controller_is_refused_before_terminal_acceptance` failed with "worker retained a released terminal lease" (156 passed, 1 failed) under full-gate load. The same binary run alone passed 5/5 (about 3.0 s each). `crates/infraternal-shim` is byte-identical to dev, so this is a load-sensitive flake outside this branch, and dev needs an owner for it. All 103 privileged checks passed, including the three `review_j350_mtu` cases. Evidence on the control host: `~/.cache/codex-scratch/j411/evidence/` holds `run1.log`, `verify-fast.log`, `baseline.log`, `root-repros.log` and `mac-route-cost.log`. Cleanup: the compute01 target and worktrees are deleted, along with /root scratch and the netns; the Mac script is removed; no processes are left. No merge, push or deploy. Verdict: REQUEST CHANGES (P2-A, P2-B). Reviewed SHA: `52608564257450cd02ccc842ab67d015ef5e3ff7`. Gates: `verify quick` PASS; `verify fast` exit 1 on `cargo-deny` (fails identically on clean dev) and `test` (a shim flake in code identical to dev); every other row PASS.
j411-baseline.log.txt
7.6 KiB
kayg commented 2026-11-09 13:19:59 +00:00
Author
Owner
Copy link

J288c round 2 on J411's review (comments 88534, 88795). Branch mesh-mtu-381. I merged origin/dev as 8006a0531; the merge touches no mesh/relay files.

  • 1af129040: J411's review_j411_mtu.rs, landed unchanged as the red test. Red on dev-02 (root inside unshare -rn), dev-02:~/.cache/codex-scratch/j288c/j411-red.log:
    • fallback: relayed … mtu 1392 → after_endpoint_move … mtu 1200 → after_direct_fallback … mtu 1408, FAILED
    • removal: after_removal=[] late_callback=Ok(()) after_late_callback=[10.254.0.2 … mtu 1392], FAILED
  • 1d3b4fedd (P2-A + P2-B): the relay overrides map becomes PeerRouteState, which also holds the installed peers by key. They are updated inside the same lock as every full route install, restored on rollback, and cleared on retire. The callback keeps only the key: it resolves the current peer under the lock, or drops the override and writes nothing once the peer is gone. Green at a523607f6, dev-02:~/.cache/codex-scratch/j288c/j411-green.log:
    • fallback: after_direct_fallback … mtu 1200, ok
    • removal: after_late_callback=[], ok
    • all three review_j350_mtu cases still pass
  • a523607f6 (P3-C): Mac path MTUs are probed on the caller's thread before the snapshot reaches the packet worker. Each endpoint IP is looked up once, each underlay interface's ifconfig is read once, and the worker records the applied interface MTU, so rollback doesn't re-probe. Finding: on the old code each Mac apply probed every peer three times (old interface MTU, route MTUs, new interface MTU), so the stall was about 3× J411's per-peer estimate.

Next: a Mac stall measurement before/after (1d3b4fedd vs a523607f6) under lockf /tmp/mac-bench.lock. It uses 50 peers: one loopback GotaTun echo peer plus 49 TEST-NET endpoints, with a 10 ms ping through the utun during 10 endpoint-change updates, measuring the longest reply gap. Mac lib tests and clippy run at the new head. verify quick is queued on dev-02, then verify fast on compute01.

J288c round 2 on J411's review (comments 88534, 88795). Branch `mesh-mtu-381`. I merged `origin/dev` as `8006a0531`; the merge touches no mesh/relay files. - `1af129040`: J411's `review_j411_mtu.rs`, landed unchanged as the red test. **Red** on dev-02 (root inside `unshare -rn`), `dev-02:~/.cache/codex-scratch/j288c/j411-red.log`: - fallback: `relayed … mtu 1392` → `after_endpoint_move … mtu 1200` → `after_direct_fallback … mtu 1408`, FAILED - removal: `after_removal=[] late_callback=Ok(()) after_late_callback=[10.254.0.2 … mtu 1392]`, FAILED - `1d3b4fedd` (P2-A + P2-B): the relay overrides map becomes `PeerRouteState`, which also holds the installed peers by key. They are updated inside the same lock as every full route install, restored on rollback, and cleared on retire. The callback keeps only the key: it resolves the current peer under the lock, or drops the override and writes nothing once the peer is gone. **Green** at `a523607f6`, `dev-02:~/.cache/codex-scratch/j288c/j411-green.log`: - fallback: `after_direct_fallback … mtu 1200`, ok - removal: `after_late_callback=[]`, ok - all three `review_j350_mtu` cases still pass - `a523607f6` (P3-C): Mac path MTUs are probed on the caller's thread before the snapshot reaches the packet worker. Each endpoint IP is looked up once, each underlay interface's `ifconfig` is read once, and the worker records the applied interface MTU, so rollback doesn't re-probe. Finding: on the old code each Mac `apply` probed every peer **three** times (old interface MTU, route MTUs, new interface MTU), so the stall was about 3× J411's per-peer estimate. Next: a Mac stall measurement before/after (`1d3b4fedd` vs `a523607f6`) under `lockf /tmp/mac-bench.lock`. It uses 50 peers: one loopback GotaTun echo peer plus 49 TEST-NET endpoints, with a 10 ms ping through the utun during 10 endpoint-change updates, measuring the longest reply gap. Mac lib tests and clippy run at the new head. `verify quick` is queued on dev-02, then `verify fast` on compute01.
kayg commented 2026-11-09 13:55:33 +00:00
Author
Owner
Copy link

J288c P3-C: Mac stall before and after, measured on infraternal-macos-01-std under lockf -k /tmp/mac-bench.lock (13:48–13:54Z, one lock hold for build and measurement).

Harness (measurement only, not committed; attached as mac_stall_probe.rs): a real Mac Node (utun + GotaTun, agent,mesh-dns release build) with 50 peers:

  • one loopback echo peer, a userspace GotaTun device that answers ICMP echo;
  • 49 peers with distinct TEST-NET endpoints routed via en0.

ping -i 0.01 runs through the utun to the echo peer. After a 5 s control window, the harness applies 10 endpoint-change snapshots, one per second. The stall metric is the max RTT reported by ping for replies during each update; a packet delayed by a blocked worker shows up as RTT. Builds alternated twice: before = 1d3b4fedd (P2 fix only), after = 8c3b99eed.

control max RTT max RTT per update, median (range), n=20 caller apply median lost
before 1d3b4fedd 0.6–1.0 ms ≈580 ms (509–765) ≈610 ms 0
after 8c3b99eed 0.5–0.6 ms 0.5 ms (0.4–23.7) ≈120 ms 0
  • Before, packets wait about as long as the whole apply, because the probing ran on the packet runtime.
  • After, the probing (about 120 ms for 49 distinct endpoints) runs on the caller's thread. The worker only installs routes, so occasional spikes stay at or under 24 ms (a route change and ifconfig for the changed peer).
  • Initial 50-peer apply: 606/720 ms → 342/338 ms.
  • The reply-gap column in the raw log is not a reliable measure: ping output to a pipe is block-buffered, so even the control window shows about 85 ms gaps. Use the RTT column.

Mac correctness at a523607f6 (same Mac code as 8c3b99eed minus two lint rewrites): cargo test -p infraternal-plugin-mesh --lib --features relay,test-support,mesh-dns → 131 passed; 0 failed; 4 ignored. Mac clippy (-D warnings) flagged two lints in the new probe code, fixed in 8c3b99eed. The crate's other Mac clippy errors (28 lib, 37 lib test) are on lines this branch doesn't change; macOS clippy is not a gate row.

Evidence: control host ~/.cache/codex-scratch/j288c/evidence/ (mac-stall2.log, mac-build2.log, mac-build-tests.log, mac_stall_probe.rs). The Mac target, binaries, state directories and scripts are removed, and no 10.254 addresses remain.

J288c P3-C: Mac stall before and after, measured on `infraternal-macos-01-std` under `lockf -k /tmp/mac-bench.lock` (13:48–13:54Z, one lock hold for build and measurement). **Harness** (measurement only, not committed; attached as `mac_stall_probe.rs`): a real Mac `Node` (utun + GotaTun, `agent,mesh-dns` release build) with **50 peers**: - one loopback echo peer, a userspace GotaTun device that answers ICMP echo; - 49 peers with distinct TEST-NET endpoints routed via en0. `ping -i 0.01` runs through the utun to the echo peer. After a 5 s control window, the harness applies 10 endpoint-change snapshots, one per second. The stall metric is the max RTT reported by `ping` for replies during each update; a packet delayed by a blocked worker shows up as RTT. Builds alternated twice: before = `1d3b4fedd` (P2 fix only), after = `8c3b99eed`. | | control max RTT | max RTT per update, median (range), n=20 | caller `apply` median | lost | |---|---:|---:|---:|---:| | before `1d3b4fedd` | 0.6–1.0 ms | **≈580 ms** (509–765) | ≈610 ms | 0 | | after `8c3b99eed` | 0.5–0.6 ms | **0.5 ms** (0.4–23.7) | ≈120 ms | 0 | - Before, packets wait about as long as the whole apply, because the probing ran on the packet runtime. - After, the probing (about 120 ms for 49 distinct endpoints) runs on the caller's thread. The worker only installs routes, so occasional spikes stay at or under 24 ms (a `route change` and `ifconfig` for the changed peer). - Initial 50-peer apply: 606/720 ms → 342/338 ms. - The reply-gap column in the raw log is not a reliable measure: `ping` output to a pipe is block-buffered, so even the control window shows about 85 ms gaps. Use the RTT column. **Mac correctness** at `a523607f6` (same Mac code as `8c3b99eed` minus two lint rewrites): `cargo test -p infraternal-plugin-mesh --lib --features relay,test-support,mesh-dns` → `131 passed; 0 failed; 4 ignored`. Mac clippy (`-D warnings`) flagged two lints in the new probe code, fixed in `8c3b99eed`. The crate's other Mac clippy errors (28 lib, 37 lib test) are on lines this branch doesn't change; macOS clippy is not a gate row. Evidence: control host `~/.cache/codex-scratch/j288c/evidence/` (`mac-stall2.log`, `mac-build2.log`, `mac-build-tests.log`, `mac_stall_probe.rs`). The Mac target, binaries, state directories and scripts are removed, and no 10.254 addresses remain.
j288c-mac-stall2.log.txt
4 KiB
j288c-mac_stall_probe.rs.txt
11 KiB
j288c-mac-build-tests.log.txt
58 KiB
kayg commented 2026-11-09 18:40:58 +00:00
Author
Owner
Copy link

J288c round 2 finish. Head 8c3b99eeddf747f500e4b08654dcdbcf0fd6d936 on mesh-mtu-381, confirmed with git ls-remote origin mesh-mtu-381. Base: origin/dev merged as 8006a0531 (merge base 6df0f5bd5). No merge, no deploy.

Commits since 526085642:

  • 8006a0531: merge of origin/dev (no mesh/relay files)
  • 1af129040: J411's review_j411_mtu.rs as the red test
  • 1d3b4fedd: P2-A + P2-B, relay route callbacks resolve the accepted peer by key
  • a523607f6: P3-C, Mac path MTUs probed before the snapshot reaches the packet worker
  • 8c3b99eed: two macOS clippy lints in the new probe

Status

  • P2-A, fixed. Red at 1af129040: after_direct_fallback … mtu 1408. Green: mtu 1200 (dev-02 root netns, and verify fast privileged row [80/105] PASS).
  • P2-B, fixed. Red: after_late_callback=[… mtu 1392]. Green: after_late_callback=[] ([79/105] PASS). J375's three cases still pass ([76–78/105]).
  • P3-C, fixed and measured on the Mac (comment 88874). With 50 peers, max ping RTT during an update went from about 580 ms (509–765) to 0.5 ms (max 23.7). Mac lib tests: 131 passed, 0 failed.

verify quick (dev-02, a523607f6; 8c3b99eed changes macOS-only code since): all 11 rows PASS, VERIFY_EXIT=0.

verify fast (compute01 through build-slot, run from ~/worktrees/j288 at 8c3b99eed):

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | FAIL (1)     |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| privileged integration    | PASS         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
VERIFY_EXIT=1
  • cargo-deny fails only on RUSTSEC-2026-0459 (libsqlite3-sys). Cargo.lock and deny.toml are identical to merge base 6df0f5bd5, and plain dev at 6df0f5bd5 fails the same way (DENY_DEV_EXIT=1, deny-dev-6df0f5bd5.log). The #473 ignore landed on dev after my merge (in 547a48885).
  • TypeScript rows. verify stops after a Rust failure, so they did not run inside this gate. I ran ts_verify_fast alone on dev-02 at 8c3b99eed (ts-rows-8c3b99e.log): the first rows pass, including the tracer suite at 656 pass / 0 fail. bun test ./src/connection*.test.ts then fails 95 pass / 10 fail on this.clipboardWrites (#440's OSC 52 code; the refusal-test fixtures build a terminal without it). Plain dev at 6df0f5bd5 fails identically: 95 pass / 10 fail, same TypeError (ts-connection-dev-6df0f5bd5.log). This branch changes nothing under apps/ or packages/ against its merge base. The rows after that suite (terminal-query tests, tracer build, transfer budget, protocol tests) did not run.
  • The shim flake "worker retained a released terminal lease" did not occur in this run: the test row passed.
  • Static (report-only, against the gate's stored baseline): x86_64 28323736 bytes (+5905480), aarch64 25423560 bytes (+4970280).

Evidence: control host ~/.cache/codex-scratch/j288c/evidence/ holds verify-fast-8c3b99e.log, verify-quick-a523607.log, j411-red.log, j411-green.log, deny-dev-6df0f5bd5.log, ts-rows-8c3b99e.log, ts-connection-dev-6df0f5bd5.log, and the Mac logs. The targets on compute01, dev-02 and the Mac are deleted, along with the scratch worktrees and Mac /tmp files. No processes are left, and no mesh claim was taken.

Not changed: the legacy RelayPaths::relay() path's wireguard::Client route updater still captures its peer. Nothing in the tree calls it, and it has no node state to resolve against. Say if it should be removed or made route-free.

J288c round 2 finish. Head **`8c3b99eeddf747f500e4b08654dcdbcf0fd6d936`** on `mesh-mtu-381`, confirmed with `git ls-remote origin mesh-mtu-381`. Base: `origin/dev` merged as `8006a0531` (merge base `6df0f5bd5`). No merge, no deploy. Commits since `526085642`: - `8006a0531`: merge of origin/dev (no mesh/relay files) - `1af129040`: J411's `review_j411_mtu.rs` as the red test - `1d3b4fedd`: P2-A + P2-B, relay route callbacks resolve the accepted peer by key - `a523607f6`: P3-C, Mac path MTUs probed before the snapshot reaches the packet worker - `8c3b99eed`: two macOS clippy lints in the new probe **Status** - **P2-A, fixed.** Red at `1af129040`: `after_direct_fallback … mtu 1408`. Green: `mtu 1200` (dev-02 root netns, and `verify fast` privileged row [80/105] PASS). - **P2-B, fixed.** Red: `after_late_callback=[… mtu 1392]`. Green: `after_late_callback=[]` ([79/105] PASS). J375's three cases still pass ([76–78/105]). - **P3-C, fixed and measured on the Mac** (comment 88874). With 50 peers, max ping RTT during an update went from about **580 ms** (509–765) to **0.5 ms** (max 23.7). Mac lib tests: 131 passed, 0 failed. **`verify quick`** (dev-02, `a523607f6`; `8c3b99eed` changes macOS-only code since): all 11 rows PASS, `VERIFY_EXIT=0`. **`verify fast`** (compute01 through build-slot, run from `~/worktrees/j288` at `8c3b99eed`): ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | Nix Cargo vendoring | PASS | | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | FAIL (1) | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | PASS | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | static builds | PASS | +---------------------------+--------------+ VERIFY_EXIT=1 ``` - **`cargo-deny`** fails only on RUSTSEC-2026-0459 (`libsqlite3-sys`). `Cargo.lock` and `deny.toml` are identical to merge base `6df0f5bd5`, and plain dev at `6df0f5bd5` fails the same way (`DENY_DEV_EXIT=1`, `deny-dev-6df0f5bd5.log`). The #473 ignore landed on dev after my merge (in `547a48885`). - **TypeScript rows.** `verify` stops after a Rust failure, so they did not run inside this gate. I ran `ts_verify_fast` alone on dev-02 at `8c3b99eed` (`ts-rows-8c3b99e.log`): the first rows pass, including the tracer suite at 656 pass / 0 fail. `bun test ./src/connection*.test.ts` then fails 95 pass / 10 fail on `this.clipboardWrites` (#440's OSC 52 code; the refusal-test fixtures build a terminal without it). Plain dev at `6df0f5bd5` fails identically: 95 pass / 10 fail, same TypeError (`ts-connection-dev-6df0f5bd5.log`). This branch changes nothing under `apps/` or `packages/` against its merge base. The rows after that suite (terminal-query tests, tracer build, transfer budget, protocol tests) did not run. - The shim flake "worker retained a released terminal lease" did not occur in this run: the `test` row passed. - Static (report-only, against the gate's stored baseline): x86_64 28323736 bytes (+5905480), aarch64 25423560 bytes (+4970280). Evidence: control host `~/.cache/codex-scratch/j288c/evidence/` holds `verify-fast-8c3b99e.log`, `verify-quick-a523607.log`, `j411-red.log`, `j411-green.log`, `deny-dev-6df0f5bd5.log`, `ts-rows-8c3b99e.log`, `ts-connection-dev-6df0f5bd5.log`, and the Mac logs. The targets on compute01, dev-02 and the Mac are deleted, along with the scratch worktrees and Mac /tmp files. No processes are left, and no mesh claim was taken. Not changed: the legacy `RelayPaths::relay()` path's `wireguard::Client` route updater still captures its peer. Nothing in the tree calls it, and it has no node state to resolve against. Say if it should be removed or made route-free.
kayg commented 2026-11-09 18:45:07 +00:00
Author
Owner
Copy link

J381r (independent review of J288c round 2) start. Reviewing mesh-mtu-381 at head 859c96154: J288c's 381b4ae22 (removes the legacy relay() and the wireguard::Client EndpointUpdater impl) plus a merge of origin/dev 547a48885. It also covers 1af129040, 1d3b4fedd, a523607f6 and 8c3b99eed.

Plan, about 2 h:

  1. Run the red/green pair for review_j411_mtu (revert 1d3b4fedd on a scratch copy) in unshare -Urn on compute01. Add a scratch stress test (not committed): concurrent relay Some/None callbacks from threads while the main thread moves the endpoint, removes the peer and re-adds it, with invariants checked after each step. Read the code for lock order.
  2. Re-measure the Mac stall at 50 peers under ping, under lockf -k /tmp/mac-bench.lock, and delete the target before releasing the lock.
  3. Re-check the #381 contract (IPv4/IPv6 route MTU, padding budget). Re-run the live matrix only if it's cheap.
  4. Run verify quick and verify fast at the final head on compute01 (dev-02 is below its 40G disk floor).
J381r (independent review of J288c round 2) start. Reviewing `mesh-mtu-381` at head **`859c96154`**: J288c's `381b4ae22` (removes the legacy `relay()` and the `wireguard::Client` `EndpointUpdater` impl) plus a merge of origin/dev `547a48885`. It also covers `1af129040`, `1d3b4fedd`, `a523607f6` and `8c3b99eed`. Plan, about 2 h: 1. Run the red/green pair for `review_j411_mtu` (revert `1d3b4fedd` on a scratch copy) in `unshare -Urn` on compute01. Add a scratch stress test (not committed): concurrent relay `Some`/`None` callbacks from threads while the main thread moves the endpoint, removes the peer and re-adds it, with invariants checked after each step. Read the code for lock order. 2. Re-measure the Mac stall at 50 peers under ping, under `lockf -k /tmp/mac-bench.lock`, and delete the target before releasing the lock. 3. Re-check the #381 contract (IPv4/IPv6 route MTU, padding budget). Re-run the live matrix only if it's cheap. 4. Run `verify quick` and `verify fast` at the final head on compute01 (dev-02 is below its 40G disk floor).
kayg commented 2026-11-09 18:59:02 +00:00
Author
Owner
Copy link

J381r progress: Mac re-measure (P3-C) confirms the fix. Ran on infraternal-macos-01-std under lockf -k /tmp/mac-bench.lock from 18:51 to 18:58Z, one hold for build and measurement. The target, worktrees and binaries are deleted, 10.254 addresses: 0, lock released.

The harness is J288c's mac_stall_probe.rs (50 peers: 1 loopback echo peer plus 49 TEST-NET endpoints via en0, ping -i 0.01 through the utun, 10 endpoint-change updates). My one addition prints route -n get for one en0 peer in both families. I ran 3 alternating rounds per build: before = 1d3b4fedd (P2 fix, probing on the worker), after = the final head 859c96154.

max ping RTT per update, median (min–max), n=30 caller apply median initial 50-peer apply control max RTT lost
before 1d3b4fedd 448.2 ms (321.4–1524.6) 491.4 ms 965 / 531 / 544 ms 0.9–2.3 ms 0
after 859c96154 0.4 ms (0.3–14.6) 83.7 ms 242 / 240 / 225 ms 0.6–1.0 ms 0

Route MTU on the Mac (en0 at 1500): 10.254.0.3 → mtu 1408, fd01::3 → mtu 1408. That is 1500−80 = 1420, rounded down to the 16-byte step, the same in both families.

Evidence: control host ~/.cache/codex-scratch/j381r/evidence/mac-stall.log and mac-build.log. The Linux red/green, stress and gate run is still queued on compute01.

J381r progress: **Mac re-measure (P3-C) confirms the fix.** Ran on `infraternal-macos-01-std` under `lockf -k /tmp/mac-bench.lock` from 18:51 to 18:58Z, one hold for build and measurement. The target, worktrees and binaries are deleted, `10.254` addresses: 0, lock released. The harness is J288c's `mac_stall_probe.rs` (50 peers: 1 loopback echo peer plus 49 TEST-NET endpoints via en0, `ping -i 0.01` through the utun, 10 endpoint-change updates). My one addition prints `route -n get` for one en0 peer in both families. I ran 3 alternating rounds per build: before = `1d3b4fedd` (P2 fix, probing on the worker), after = the final head `859c96154`. | | max ping RTT per update, median (min–max), n=30 | caller `apply` median | initial 50-peer apply | control max RTT | lost | |---|---:|---:|---:|---:|---:| | before `1d3b4fedd` | **448.2 ms** (321.4–1524.6) | 491.4 ms | 965 / 531 / 544 ms | 0.9–2.3 ms | 0 | | after `859c96154` | **0.4 ms** (0.3–14.6) | 83.7 ms | 242 / 240 / 225 ms | 0.6–1.0 ms | 0 | Route MTU on the Mac (en0 at 1500): `10.254.0.3` → `mtu 1408`, `fd01::3` → `mtu 1408`. That is 1500−80 = 1420, rounded down to the 16-byte step, the same in both families. Evidence: control host `~/.cache/codex-scratch/j381r/evidence/mac-stall.log` and `mac-build.log`. The Linux red/green, stress and gate run is still queued on compute01.
kayg commented 2026-11-09 19:29:09 +00:00
Author
Owner
Copy link

J288c follow-up on the relay() answer: done and pushed. Head 859c9615470c1aa8eb332ab9cbc765a1cfb0dd8b, confirmed with git ls-remote.

  • 381b4ae22 removes RelayPaths::relay() and the EndpointUpdater for wireguard::Client impl, including its peer-copy route updater. Nothing depended on either: no caller, test or plugin API surface. The relay fixture's relay() is a local function unrelated to this. start_kernel_handshake and prepare_kernel_handshake stay, because the Node kernel engine uses them.
  • 859c96154 merges origin/dev 547a48885, which includes ba79e6746 (the RUSTSEC-2026-0459 ignore). No mesh or relay files change in the merge.

verify quick is queued on compute01 (log compute01:~/.cache/codex-scratch/j288c/verify-quick-859c961.log). dev-02 is at its 40G disk floor, so I didn't queue there. I'll post the result here.

J288c follow-up on the `relay()` answer: done and pushed. Head `859c9615470c1aa8eb332ab9cbc765a1cfb0dd8b`, confirmed with `git ls-remote`. - `381b4ae22` removes `RelayPaths::relay()` and the `EndpointUpdater for wireguard::Client` impl, including its peer-copy route updater. Nothing depended on either: no caller, test or plugin API surface. The relay fixture's `relay()` is a local function unrelated to this. `start_kernel_handshake` and `prepare_kernel_handshake` stay, because the Node kernel engine uses them. - `859c96154` merges origin/dev `547a48885`, which includes `ba79e6746` (the RUSTSEC-2026-0459 ignore). No mesh or relay files change in the merge. `verify quick` is queued on compute01 (log `compute01:~/.cache/codex-scratch/j288c/verify-quick-859c961.log`). dev-02 is at its 40G disk floor, so I didn't queue there. I'll post the result here.
kayg commented 2026-11-09 23:42:23 +00:00
Author
Owner
Copy link

J288c: verify quick passed at head 859c9615470c1aa8eb332ab9cbc765a1cfb0dd8b, on compute01 through build-slot. Log: compute01:~/.cache/codex-scratch/j288c/verify-quick-859c961.log.

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| scope guard               | PASS         |
| migration names           | PASS         |
| Nix Cargo vendoring       | PASS         |
| VT source recipe          | PASS         |
| fmt                       | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| clippy (workspace)        | PASS         |
| unit tests (changed)      | PASS         |
+---------------------------+--------------+

QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done.
VERIFY_EXIT=0

Delta for J458, from 8c3b99eed:

  • 381b4ae22: removes RelayPaths::relay() and the EndpointUpdater for wireguard::Client impl (the route updater that kept its own peer copy), plus the now-unused BTreeMap import. That's 40 lines in relay_path.rs.
  • 859c96154: merges origin/dev 547a48885, which contains ba79e6746 (the cargo-deny ignore). No mesh or relay files change.

I did not re-run verify fast on this delta. The compute01 target is deleted.

J288c: `verify quick` passed at head **`859c9615470c1aa8eb332ab9cbc765a1cfb0dd8b`**, on compute01 through build-slot. Log: `compute01:~/.cache/codex-scratch/j288c/verify-quick-859c961.log`. ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | scope guard | PASS | | migration names | PASS | | Nix Cargo vendoring | PASS | | VT source recipe | PASS | | fmt | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | clippy (workspace) | PASS | | unit tests (changed) | PASS | +---------------------------+--------------+ QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done. VERIFY_EXIT=0 ``` Delta for J458, from `8c3b99eed`: - `381b4ae22`: removes `RelayPaths::relay()` and the `EndpointUpdater for wireguard::Client` impl (the route updater that kept its own peer copy), plus the now-unused `BTreeMap` import. That's 40 lines in `relay_path.rs`. - `859c96154`: merges origin/dev `547a48885`, which contains `ba79e6746` (the cargo-deny ignore). No mesh or relay files change. I did not re-run `verify fast` on this delta. The compute01 target is deleted.
kayg commented 2026-11-09 23:53:26 +00:00
Author
Owner
Copy link

J381r re-review finish: APPROVE. Reviewed SHA 859c9615470c1aa8eb332ab9cbc765a1cfb0dd8b, the final head per git ls-remote origin mesh-mtu-381 just now. It covers J288c's 1af129040, 1d3b4fedd, a523607f6, 8c3b99eed and 381b4ae22 (legacy relay() removed), plus the merge of origin/dev 547a48885. No merge, push or deploy.

1. The fix (P2-A, P2-B): red/green confirmed, no new race found

review_j411_mtu ran in unshare -Urn on compute01, one namespace per test.

  • Green at 859c96154: P2-A gives relayed … mtu 1392 → after_endpoint_move … mtu 1200 → after_direct_fallback … mtu 1200, ok. P2-B gives after_removal=[] late_callback=Ok(()) after_late_callback=[], ok.
  • Red (1d3b4fedd reverted on a scratch copy of 859c96154): P2-A gives after_direct_fallback … mtu 1408, FAILED. P2-B gives after_late_callback=[… mtu 1392], FAILED. Both match J411.
  • A gotcha for future red runs: my first red build came out byte-identical to green. The repo's --remap-path-prefix makes a second worktree's build reuse the first one's units. The real red was built by applying the revert inside the same worktree (build-red2.err, sha256 differs from green).

Lock order: no cycle. The relay side takes Peer.route_mtu_generation, then PeerRouteState (in update_peer_route_mtu's spawn_blocking). settle takes Peer.state, then route_mtu_generation, and releases both before any route write. The node side takes only PeerRouteState, and while holding it calls only route netlink (routes_with_mtu_overrides → route_mtu); it never takes a relay lock. The callback runs on a blocking thread, so a node-side hold never parks a tokio worker. The packet path takes neither lock.

Removal ordering: the full snapshot path runs apply_peer_changes → refresh_peer_routes (drops the removed key from peers under the lock) → retain_recorded_routes. A callback that wins the lock before the refresh writes a route that the retain step then deletes. After the refresh, the callback is a no-op.

Stress (scratch test, not committed: review_j381r_stress.rs). Four threads call the updater in a tight loop, alternating Some(1392)/None (65.3 M calls, 0 errors). Meanwhile the coordinator applies 300 snapshots cycling narrow endpoint → wide → remove → re-add. Route MTUs were sampled for v4 and v6 after each snapshot: 3000 samples, 0 oversize, 0 routes for a removed peer, and quiesced final None → v4 1408 / v6 1408. Three v6 samples read mtu 1280 in the narrow step, where my oracle allowed only {1200, 1392}. This is the kernel clamping a concurrently written relay 1392 route down to the device MTU (1408 → 1280) when the interface shrinks. It is safe (smaller, never larger); see P3-1. Under that tight loop, snapshot latency was p50 277 ms and max 13.6 s because std::Mutex is unfair. Real callbacks fire only on a path change or a ≤1 Hz MTU-poll change, so this is a test artifact, not a finding.

Stale underlay on the Mac: the probe now runs on the caller's thread immediately before WorkerCommand::Apply. The caller (node_actions → Node::chunk) already blocked on the reply, so it adds no wait and moves the read point by microseconds. A network change in that window was already a race before (the read was also one point-in-time sample), and nothing on the Mac re-probes on an underlay change either before or after this branch. Not a regression. On Linux the budget is still read at write time, under the lock.

2. The Mac (P3-C): re-measured, fixed

This is in comment 89256: 50 peers, update under ping -i 0.01, lockf -k /tmp/mac-bench.lock, 3 alternating rounds per build. Max RTT per update went from 448.2 ms median (321–1525) at 1d3b4fedd to 0.4 ms (0.3–14.6) at 859c96154. Caller apply went from 491 to 84 ms, initial apply from ~540 to ~235 ms, 0 lost. The target and binaries were deleted before the lock was released.
Linux does no probing, and its change is only the extended lock scope. The test and privileged rows pass, and the Linux route budget is still per write.

3. The #381 contract

  • Route MTU follows the underlay in both families. On Linux, the J375/J411 privileged cases pass in verify fast's privileged row (1200 on a 1280 path, 1408 on 1500). On the Mac, en0 1500 gives 10.254.0.3 … mtu 1408 and fd01::3 … mtu 1408.
  • Padding: the mixed_path_direct_ipv6_budget_accounts_for_wireguard_padding case still passes. The invariant (route MTU a multiple of 16 and ≤ underlay−80) is unchanged by these commits.
  • Linux→Mac throughput: not re-run. It needs the mesh claim plus J288's two-host kit, so it is not cheap. These commits leave the data path alone apart from where the Mac computes the utun MTU (interface_mtu over the same probe results as for_config). The Mac utun and route values above match the pre-change arithmetic, so J288's 117.8 / 258 Mbit/s still stands by construction, not by measurement.

Findings

  • P1: none. P2: none.
  • P3-1 (latent, not from this round): the Linux interface MTU (for_config) ignores relay overrides. When a peer is relayed at 1392 and the direct paths are narrower, the device sits at the 1280 floor: the kernel clamps the v6 route to 1280 and caps v4 packets at 1280. It's safe, but relayed peers lose ~100 bytes of payload in that case. Relay isn't shipped yet, so this can be a follow-up.
  • P3-2 (pre-existing since J375): on a failed snapshot, restore_peer_route_state(previous) overwrites the whole state, including any override a relay callback changed during the transaction. If the relay dropped mid-transaction, the rollback reinstalls a stale 1392 override on a direct path. The window is narrow and only on failure.

Gates (compute01, clean tree at 859c96154, through build-slot)

verify quick, VERIFY_QUICK_BASE=origin/dev (verify-quick.log): all 11 rows PASS, QUICK_EXIT=0.

verify fast (verify-fast.log):

| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| privileged integration    | PASS         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| nightly browser suite reporter | PASS         |
| static builds             | PASS         |
...
 655 pass
 1 fail
(fail) a closing notice cannot hide its replacement [1.37ms]
VERIFY_EXIT=1

Every Rust row passes, and cargo-deny now passes too (the #473 ignore came in with the dev merge). The single failure is in the TypeScript tracer suite: apps/tracer/src/canvas/sessions/startup-toast.test.ts hits TypeError: element.getAttribute is not a function at packages/ui/src/input-source.ts:83 (markMotion, from toast.ts:224), because the fake DOM's elements lack getAttribute. It doesn't come from this branch: git diff 547a48885 859c96154 -- apps packages is empty, so the TS tree is byte-identical to dev. It's the #427 input-source change meeting toast-fake-dom.ts, and dev needs an owner for it. The rows after the tracer suite did not run.

Evidence (control host ~/.cache/codex-scratch/j381r/evidence/): j411-red.log, green-and-stress.log, review_j381r_stress.rs, build-red2.err, verify-quick.log, verify-fast.log, mac-stall.log, mac-build.log. Cleanup: compute01 target, bins and worktrees deleted; Mac target, worktrees and /Users/Shared/j381r deleted; no 10.254 addresses left; no mesh claim taken.

Verdict: APPROVE at 859c9615470c1aa8eb332ab9cbc765a1cfb0dd8b.

J381r re-review finish: **APPROVE**. Reviewed SHA **`859c9615470c1aa8eb332ab9cbc765a1cfb0dd8b`**, the final head per `git ls-remote origin mesh-mtu-381` just now. It covers J288c's `1af129040`, `1d3b4fedd`, `a523607f6`, `8c3b99eed` and `381b4ae22` (legacy `relay()` removed), plus the merge of origin/dev `547a48885`. No merge, push or deploy. ## 1. The fix (P2-A, P2-B): red/green confirmed, no new race found `review_j411_mtu` ran in `unshare -Urn` on compute01, one namespace per test. - **Green at `859c96154`:** P2-A gives `relayed … mtu 1392` → `after_endpoint_move … mtu 1200` → `after_direct_fallback … mtu 1200`, ok. P2-B gives `after_removal=[] late_callback=Ok(()) after_late_callback=[]`, ok. - **Red (`1d3b4fedd` reverted on a scratch copy of `859c96154`):** P2-A gives `after_direct_fallback … mtu 1408`, FAILED. P2-B gives `after_late_callback=[… mtu 1392]`, FAILED. Both match J411. - A gotcha for future red runs: my first red build came out byte-identical to green. The repo's `--remap-path-prefix` makes a second worktree's build reuse the first one's units. The real red was built by applying the revert inside the same worktree (`build-red2.err`, sha256 differs from green). **Lock order:** no cycle. The relay side takes `Peer.route_mtu_generation`, then `PeerRouteState` (in `update_peer_route_mtu`'s `spawn_blocking`). `settle` takes `Peer.state`, then `route_mtu_generation`, and releases both before any route write. The node side takes only `PeerRouteState`, and while holding it calls only route netlink (`routes_with_mtu_overrides` → `route_mtu`); it never takes a relay lock. The callback runs on a blocking thread, so a node-side hold never parks a tokio worker. The packet path takes neither lock. **Removal ordering:** the full snapshot path runs `apply_peer_changes` → `refresh_peer_routes` (drops the removed key from `peers` under the lock) → `retain_recorded_routes`. A callback that wins the lock before the refresh writes a route that the retain step then deletes. After the refresh, the callback is a no-op. **Stress** (scratch test, not committed: `review_j381r_stress.rs`). Four threads call the updater in a tight loop, alternating `Some(1392)`/`None` (65.3 M calls, 0 errors). Meanwhile the coordinator applies 300 snapshots cycling narrow endpoint → wide → remove → re-add. Route MTUs were sampled for v4 and v6 after each snapshot: 3000 samples, 0 oversize, 0 routes for a removed peer, and quiesced final `None` → v4 1408 / v6 1408. Three v6 samples read `mtu 1280` in the narrow step, where my oracle allowed only {1200, 1392}. This is the kernel clamping a concurrently written relay 1392 route down to the device MTU (1408 → 1280) when the interface shrinks. It is safe (smaller, never larger); see P3-1. Under that tight loop, snapshot latency was p50 277 ms and max 13.6 s because `std::Mutex` is unfair. Real callbacks fire only on a path change or a ≤1 Hz MTU-poll change, so this is a test artifact, not a finding. **Stale underlay on the Mac:** the probe now runs on the caller's thread immediately before `WorkerCommand::Apply`. The caller (`node_actions` → `Node::chunk`) already blocked on the reply, so it adds no wait and moves the read point by microseconds. A network change in that window was already a race before (the read was also one point-in-time sample), and nothing on the Mac re-probes on an underlay change either before or after this branch. Not a regression. On Linux the budget is still read at write time, under the lock. ## 2. The Mac (P3-C): re-measured, fixed This is in comment 89256: 50 peers, update under `ping -i 0.01`, `lockf -k /tmp/mac-bench.lock`, 3 alternating rounds per build. Max RTT per update went from **448.2 ms** median (321–1525) at `1d3b4fedd` to **0.4 ms** (0.3–14.6) at `859c96154`. Caller apply went from 491 to 84 ms, initial apply from ~540 to ~235 ms, 0 lost. The target and binaries were deleted before the lock was released. **Linux** does no probing, and its change is only the extended lock scope. The `test` and privileged rows pass, and the Linux route budget is still per write. ## 3. The #381 contract - Route MTU follows the underlay in both families. On Linux, the J375/J411 privileged cases pass in `verify fast`'s privileged row (1200 on a 1280 path, 1408 on 1500). On the Mac, en0 1500 gives `10.254.0.3 … mtu 1408` and `fd01::3 … mtu 1408`. - Padding: the `mixed_path_direct_ipv6_budget_accounts_for_wireguard_padding` case still passes. The invariant (route MTU a multiple of 16 and ≤ underlay−80) is unchanged by these commits. - Linux→Mac throughput: **not re-run.** It needs the mesh claim plus J288's two-host kit, so it is not cheap. These commits leave the data path alone apart from *where* the Mac computes the utun MTU (`interface_mtu` over the same probe results as `for_config`). The Mac utun and route values above match the pre-change arithmetic, so J288's 117.8 / 258 Mbit/s still stands by construction, not by measurement. ## Findings - **P1:** none. **P2:** none. - **P3-1 (latent, not from this round):** the Linux interface MTU (`for_config`) ignores relay overrides. When a peer is relayed at 1392 and the direct paths are narrower, the device sits at the 1280 floor: the kernel clamps the v6 route to 1280 and caps v4 packets at 1280. It's safe, but relayed peers lose ~100 bytes of payload in that case. Relay isn't shipped yet, so this can be a follow-up. - **P3-2 (pre-existing since J375):** on a failed snapshot, `restore_peer_route_state(previous)` overwrites the whole state, including any override a relay callback changed during the transaction. If the relay dropped mid-transaction, the rollback reinstalls a stale 1392 override on a direct path. The window is narrow and only on failure. ## Gates (compute01, clean tree at `859c96154`, through build-slot) `verify quick`, `VERIFY_QUICK_BASE=origin/dev` (`verify-quick.log`): all 11 rows PASS, `QUICK_EXIT=0`. `verify fast` (`verify-fast.log`): ```text | Nix Cargo vendoring | PASS | | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | PASS | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | nightly browser suite reporter | PASS | | static builds | PASS | ... 655 pass 1 fail (fail) a closing notice cannot hide its replacement [1.37ms] VERIFY_EXIT=1 ``` Every Rust row passes, and `cargo-deny` now passes too (the #473 ignore came in with the dev merge). The single failure is in the TypeScript tracer suite: `apps/tracer/src/canvas/sessions/startup-toast.test.ts` hits `TypeError: element.getAttribute is not a function` at `packages/ui/src/input-source.ts:83` (`markMotion`, from `toast.ts:224`), because the fake DOM's elements lack `getAttribute`. It doesn't come from this branch: `git diff 547a48885 859c96154 -- apps packages` is empty, so the TS tree is byte-identical to dev. It's the #427 input-source change meeting `toast-fake-dom.ts`, and dev needs an owner for it. The rows after the tracer suite did not run. Evidence (control host `~/.cache/codex-scratch/j381r/evidence/`): `j411-red.log`, `green-and-stress.log`, `review_j381r_stress.rs`, `build-red2.err`, `verify-quick.log`, `verify-fast.log`, `mac-stall.log`, `mac-build.log`. Cleanup: compute01 target, bins and worktrees deleted; Mac target, worktrees and `/Users/Shared/j381r` deleted; no 10.254 addresses left; no mesh claim taken. **Verdict: APPROVE** at `859c9615470c1aa8eb332ab9cbc765a1cfb0dd8b`.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
j470b-server-mesh-rejoin
mesh-compat-382
j444b-d381-key-wins-flood
j461b-macos-utun-reader-thread
j475b-remove-host-motion
fix-reconnect-new-link
j481-d380
dev
j479-mesh-soak
fix/j476-host-facts
j475-motion-pill-split
j470-mac-node-join
root-upgrade-d376
j474-review-merge
latency-targets-120hz
nightly-suite-shards
gotatun-pin-j469
macos-utun-reader-thread
fix-flood-echo-stall
wip/j416
motion-fast-ease-out
mesh-addhost-j259
takeover-close-terminals-d368
kbd-instant-followups
fix-trust-key-double-click
nightly-suite-partial-report
appwin-icons-j207
rusqlite-slice1
swipe-content-first-d374
mesh-mtu-381
addhost-instant-open
glass-contrast-d365-mock
glass-contrast-d365
signout-followups-442
fix-startup-toast-red
research-gotatun-matrix-j317
themes-sync-j304
fix-load-flakes
fix-nix-zig-patch
fix-compute01-browser-baseline
ratchet-size-baseline-419
predict-auto-d130
revoke-coverage-d367
nightly-suite-queue-timeout
nightly-browser-suite
fix-rustsec-2026-0459
fonts-ioskeley-obel
log-retention-d375
fix-kbd-no-anim-427
fix-dev-red-specs-2
takeover-proof-305-j205
wip/j410-server-pump
ui-consistency-d372
fix-osc52-clipboard
libghostty-vt-1.0
forget-retired-host-d369
j404-merge-426
fix-setup-lock-race
fix-touch-select-426
revoke-access-report-ui
fix-predictive-echo-specs
fix-selinux-agent-exec
wip/j395-checkpoint2
wip/j395
j391-review-scratch
ui-consistency-j195-deferred-rebased
fix-rustc-paths-421
decisions-wording-341
fix-install-suite-red
fix-dev-rtt-specs
fix-pending-removal-align
fix-privileged-skip
fix-install-suite-404
review-j368
mesh-dns-j254
revoke-access-report-d356b
revoke-privilege-round5
fix-removal-confirm
review-j360
gotatun-rotation-367
deps-refresh-340
review-j354
review-J351
review-j350
addhost-takeover-d359
revoke-report-items-347
revoke-end-sessions-d362
review-j348
size-mesh-ssh-346
size-trim-337
revoke-access-diff-d356
j259-pre-rebase-b06c5a6
j305-ended-at-start-spec
addhost-sudo-consent
fix-layout-real-specs-411
fix-shell-open-411
fix-console-hosts-373
fix-shell-exit
fuzz-host-name-353
fix-host-details-410
macos-filter-j197
research-gotatun-tricks
revoke-p3-j216l
stale-state-pill
revoke-followups-j216k
revoke-progress-j216e
backup/macos01-20261106/lume/j197
backup/macos01-20261106/lume/appwin-icons-j207-stash0
backup/macos01-20261106/claude/stash-0
backup/macos01-20261106/claude/stash-1
backup/macos01-20261106/claude/J80-macos-s1
backup/macos01-20261106/claude/J80
backup/macos01-20261106/claude/J80-slice1-20261102
backup/macos01-20261106/claude/J16b-appwin-on-dev
backup/macos01-20261106/claude/appwin-launch-idle-146
backup/macos01-20261106/claude/J80-s1-run
wip/j259
trust-285-p3c
runtime-lock-j216f
mesh-ssh-p3-j175
tun-proof-p3b
trust-285-p3
terminal-polish-j271
tun-proof-p3
ssh-install-flakes-j216i
fuzz-fixes-j264
trust-285-a
trust-285-b
restart-oversized-fixture-j198
backup/macos01-20261106/lume/appwin-icons-j207
restart-bound-j216h
up-preflight-p3-j198
backup/macos01-20261106/lume/j251
up-preflight-j198
wip/up-preflight-j198-j245-p2
wip/up-preflight-j198-j245-p1
agent-fsync-offloop-j196
wip/up-preflight-j198-j245
settings-offscreen-j231
wip/up-preflight-j198-j226-inspection
wip/j103f
themes-j206
wip/up-preflight-j198-j226
flakes-j216c
backup/macos01-20261106/claude/j197
flakes-j216b
backup/macos01-20261106/claude/j197-red
wip/up-preflight-j198-review
flakes-j216
backup/macos01-20261106/claude/appwin-icons-j207
engineering-j208b
appwin-mock-j209
toast-stack-j165
engineering-j208
decisions-d351
decisions-d350
decisions-d349
appearance-presets-mock-j129
decisions-d348
docs-audit-spec
doc-coverage-baseline-raise
decisions-d347
decisions-d346b
decisions-d346
j88h-profile
ui-consistency-j195-deferred
ui-consistency-j195
engineering-practices-j194
preferred-login-ui-j166
signin-pools-263-j174
macos-node-265-s1
decisions-d345
decisions-d344
proxy-cgroup-j164
verify-quick-fuzz-329
proxy-start-io-j168
timing-flakes-j163
up-263-s1
mcp-events-j147
verify-quick-fuzz-j158
proxy-p3-j159
agent-resource-j157-detect
wip/j103d-final
remote-cancel-hang-j155
proxy-traefik-281-j62d
j149-verify-quick
proxy-traefik-281-j62b
j150-drop-unnamed
wip/j79d-red-r4
wip/j103d-d332
rootless-toast-j142
wip/j103d-head
wip/j103d
j125c-mcp
macos-build-j136
backup/macos01-20261106/claude/j136
wip/j79d-red-cli
wip/j79d-red-server
j121-link-liveness
j118c-tracer-failures
wip/j79c-head
release-desktop-j131
j124c-signed-macos
wip/j79c-red-p3
wip/j79c-red-p23
wip/j79c-red-p22
merge-j128-check
perf-deferred-j128
sidebar-toggle-260
j127-shortcut-changed-dot
j125-mcp
j118-tracer-failures
j126-account-summary
settings-sections-mock-j92
j120-pinch-lock
sidebar-292
backup/macos01-20261106/claude/j120
taste-2026-11-03
row-actions-242-215
backup/macos01-20261106/claude/stash-0-untracked
backup/macos01-20261106/claude/J80-265-s1-run
backup/macos01-20261106/claude/J80-macos-s1-20261102
j112-canvas-commands
ui-no-overshoot-scroll
fix-recovery-restart
backup/macos01-20261106/claude/stash-1-untracked
taste-2026-11-02
size-baseline-ratchet
machine-cards-structure-j108
j109-boot-copy-mockups
fix-dev-red
wip/up-263-s1
j95b-settings-rows
proxy-traefik-281
j95-option-rows
docs-human-j91
docs-keys-palette
fix-flood-1013-270-j38d
mesh-ssh-s1
mesh-ssh-s0
addhost-d291-tunnel
test-stability-load
mesh-relay-220
repro-296
shim-thp-off-24
mesh-pins-258-v2
mesh-dns
signin-d311-merged
docs-pages
ui-size-283
fix-ui-batch-specs
wip/j12b-edge-update
gate-on-arm
fix-rename-sync-293
trust-285-j60b
fix-focus-ring-clip
integ-ui-1101
edge-update-v2
signin-d311
taste-2026-11-01
layout-per-tab-250
ui-approved-splitopt-224
ui-approved-splitopt-224-on-dev
icon-text-centre-282
ui-approved-189-212-216
research-reverse-proxy
transfer-budget-320
review-mesh-pins-258-r2
sidebar-pill-259
review-mesh-relay-220
wip/j12-edge-update-v2
nix-package
status-crash-loop-237
fix-dev-test-failures
fix-fixture-socket-path
wip/mesh-relay-220-pre-dev-advance-rebase
fix-canvas-side-clip
research-head-to-head
frosted-real-256
review-mesh-pins-258
wip/mesh-relay-220-pre-final-rebase
review-layout-per-tab-250-r2
broad-suite-triage
nix-devshell
review-status-crash-loop-237-r3
wip/nix-devshell-pre-p3
taste-frosted-real
wip/code-serve-demo-2026-10-30
wip/code-serve-glass-2026-10-30
wip/code-serve-clusters-2026-10-30
wip/code-merge-release94-dev-2026-10-30
wip/rproxy-delta-scratch-2026-10-30
wip/stale-note-257-staged-2026-10-30
wip/d241-size-staged-2026-10-30
review-broad-suite-207
wip/mesh-pins-258-pre-split
wip/mesh-relay-220-pre-rebase
mesh-measure-220
plan-211-update
review-edge-autoupdate-r3
perf24-reconnect
review-stale-note-r2
release-prod-live
d241-shell-marker
review-mesh-pins-server-258
review-revocation-r3
review-setup-token-r2
review-sync-journal-r3
wip/review-mesh-pins-258-prune-review-1995fee46
frosted-scroll-256
sync-journal-r3
revocation-per-account-wip
review-tokio-asyncfd-0386
review-sync-p3s
wip/stash-8by64-stash0-2026-10-30
mesh-drag-r2
setup-token-data-wip
release-9-4
auth-busy-1013
mesh-view-overlap
sync-p3s
review-auth-busy-wake
review-readd-r8-wip
mesh-drag-bounds
audit-auth-93
app-windows-stream-start
review-authority-cap-r3
app-windows-r7-measure
readd-minimal-r6
readd-minimal-r5
app-windows-r7-soak
readd-minimal-r4
app-windows-r7
app-windows-r6
readd-minimal-r3
readd-minimal-r2
readd-minimal
merge-release8-dev
review-busy-page-liveness-r2
review-busy-page-liveness
mesh-view-trim
review-authority-read-retry-r2
review-remove-uninstall-r2
release-retry-reattach
release-retry
review-remove-uninstall
review-authority-read-retry
app-windows-ship
mesh-autoswitch-p2
mesh-ssh
mesh-relay
mesh-macos-node
screen-and-control
session-labels
review-real-logins-r5
review-real-logins-r4
review-real-logins-r3
shim-memory-r2
review-real-logins-r2
shim-memory
shadow-addhost-instant
shadow-real-logins
shadow-same-session-flood
shadow-prod-perf-fix
hosts-from-server
app-windows-real
app-windows-transport
release-prod-6w
mesh-policy-verify
quiet-close-toast
dev-merge-r4
review-mesh-agent-transport
review-app-windows-transport
settings-roundtrip
review-release-pipeline-r4
mesh-agent-transport
ended-strip-fix
review-audit-ca-update-d244
settings-rows
phone-width
signout-clear
fix-restart-tests
dev-merge-rp
signer-hardening
review-audit-ca-update
terminal-fidelity
review-audit-ca-update-wip
mesh-integrate
review-mesh-agent-transport-515d3712
review-mesh-dataplane-feec1988
mesh-dataplane
signout
research-canvas-node
wip/review-renderer-cost
wip/review-terminal-fidelity
padlock-glyph
review-release-pipeline-2e95b689
phone-polish
sidebar-polish
inbox-generic
review-mesh-dataplane-378d4055
prod-batch
ui-inbox
ui-ca-unlock
ui-machine-sections
review-mesh-dataplane-a5f744fb
tls-session-prod
session-bounds-prod
fix-dead-session
session-bounds
tls-session
palette-polish
shortcuts-d222
research-appmeta
phone-gutter
ui-centre-button
wip/ui-centre-button
wip/tls-session
review-addhost-breakit
ui-platform
renderer-cost
remote-agents
mesh-api
ssh-bootstrap
prod-hotfix2
prod-fixes
review-remote-agents
review-mesh-dataplane
session-life
research-browser
review-mesh-api
research-capture
review-mesh-api-round3
review-mesh-api-round1
gate-stability
review-gate-stability
review-relay-direct
review-mesh-api-findings
proto-sessions
font-picker
proto-addhost
proto-windows
mesh-core
latency-gate
research-bsd
research-discovery
keytray-mods
mock-core
ui-tweaks
fix-motion-sel
fix-motion
integrate2
fix-robust
t-web
t-server
vt-ghostty
fix-snap-echo
t-web-auth
review-vt2
t-agent
lane-protocol
lane-shim
lane-auth
s1-base
s1-contracts
s1-workspace
No results found.
Labels
Clear labels   
launch-qa
Findings from launch QA.

No labels
launch-qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/infraternal#381
No description provided.