kayg/infraternal
Watch
1
0
Fork
You've already forked infraternal
 0
Code Issues 356 Pull requests Projects Releases Packages Wiki Activity Actions

Prod: remote infraternal agents never reconnect after a long server outage #225

New issue
Open
opened 2026-10-15 15:03:07 +00:00 by kayg · 78 comments
kayg commented 2026-10-15 15:03:07 +00:00
Owner
Copy link

Production bug, found 2026-10-15 ~20:35 IST, after monitor01 went down (17:57–19:34 IST, VM disk I/O stall), the switch-over deploy at 20:01 IST, and the owner's CA unlock at 20:30 IST: no remote infraternal agent reconnected. The UI shows only monitor01, and Mesh shows 0 hosts.

Evidence:

  • Production DB: 3 non-revoked remote hosts (54236622…, bcf38240…, 0bc79da5…). Their certificates are valid until Oct 19–21, so expiry is not the cause.
  • monitor01: nothing is established on :7443. After the unlock, the server log shows no remote connection at all.
  • Host 0bc79da5 (on infraternal-dev, an owner add-host fixture; do not touch it): the agent process is alive (pid since Oct 14), its binary is the OLD 20,333,104 B build, it has 0 sockets, and :7443 is reachable from that host with nc. Its journal shows no error after "Preparing the enrolled host connection".
  • Likely class: the server-link task in crates/infraternal-agent/src/remote_server_path.rs run() can end (current(...)?, ok_or(BrokenPipe)?, initialize_paths()?) while the process stays alive, so systemd never restarts it and the host is gone until someone restarts it by hand.
Production bug, found 2026-10-15 ~20:35 IST, after monitor01 went down (17:57–19:34 IST, VM disk I/O stall), the switch-over deploy at 20:01 IST, and the owner's CA unlock at 20:30 IST: **no remote infraternal agent reconnected.** The UI shows only monitor01, and Mesh shows 0 hosts. Evidence: - Production DB: 3 non-revoked remote hosts (54236622…, bcf38240…, 0bc79da5…). Their certificates are valid until Oct 19–21, so expiry is not the cause. - monitor01: nothing is established on :7443. After the unlock, the server log shows no remote connection at all. - Host 0bc79da5 (on infraternal-dev, an owner add-host fixture; do not touch it): the agent process is alive (pid since Oct 14), its binary is the OLD 20,333,104 B build, it has **0 sockets**, and :7443 is reachable from that host with nc. Its journal shows no error after "Preparing the enrolled host connection". - Likely class: the server-link task in crates/infraternal-agent/src/remote_server_path.rs `run()` can end (`current(...)?`, `ok_or(BrokenPipe)?`, `initialize_paths()?`) while the process stays alive, so systemd never restarts it and the host is gone until someone restarts it by hand.
red-corrected.log
1.1 KiB
install-red.log
1.9 KiB
service-red.log
1.4 KiB
old-process-outage.log
1.4 KiB
old-tcp-outage.log
684 B
modules-green.log
7.8 KiB
old-stalled-tcp-blackhole.log
2.6 KiB
fixed-stalled-tcp-blackhole.log
2.7 KiB
old-final-tcp-blackhole.log
2.7 KiB
fixed-final-tcp-blackhole.log
2.5 KiB
fixed-long-stalled-tcp-blackhole.log
2.5 KiB
tcp-red.log
1.2 KiB
old-final-long-tcp-blackhole.log
2.7 KiB
perf-baseline.log
3.4 KiB
worker-red-corrected.log
1.1 KiB
worker-green.log
3.7 KiB
worker-control-tests.log
1.2 KiB
worker-clippy.log
327 B
cadence-red.log
2.9 KiB
cadence-green.log
3.7 KiB
cadence-clippy.log
593 B
verify-fast-interim.log
297 KiB
verify-fast-final.log
275 KiB
ts-check.log
29 KiB
perf-fixed-final.log
3.4 KiB
final-size-and-measurement.txt
1.6 KiB
author-review.txt
1.5 KiB
historical-harness.patch
12 KiB
perf-cleanup.log
58 B
local-cleanup.txt
232 B
worker-bound.log
58 KiB
six-hour.log
57 KiB
generated-service-units.txt
787 B
field-process-metadata.txt
519 B
link-exit-replay.log
713 B
link-exit-clippy.log
0 B
perf-native-candidate.log
3.7 KiB
perf-latency-candidate.log
1.9 KiB
superseded-state.log
5.6 KiB
incremental-size.log
967 B
baseline-static.txt
189 B
candidate-static.txt
189 B
size-section-attribution.txt
679 B
size-function-attribution.txt
768 B
size-function-deltas-normalized.txt
8.3 KiB
state-baseline.log
3.8 KiB
worker-bound-entrypoint.log
1.1 KiB
review-clippy-state.log
4 KiB
perf-latency-baseline.log
1.9 KiB
perf-recovery-baseline.log
3.2 KiB
perf-recovery-candidate.log
3.2 KiB
verify-fast-head.log
300 KiB
verify-ts-head.log
29 KiB
link-exit-replay-static.log
713 B
review-red.log
6.8 KiB
store-red.log
2.7 KiB
lifecycle-green.log
18 KiB
lifecycle-exact.log
65 KiB
store-green.log
5.6 KiB
cli-green.log
13 KiB
fence-comments.log
991 B
slice-green.log
5.1 KiB
delta-perf-latency-candidate.log
1.8 KiB
off-red.log
1.2 KiB
off-green.log
1.7 KiB
size-buffered-red.log
960 B
size-final-green.log
303 B
size-baseline-to-candidate.txt
5.4 KiB
baseline-static-build.log
19 KiB
perf-store-final.log
1.7 KiB
perf-store-vmhwm.log
1.9 KiB
measure-store.txt
2.4 KiB
short-path-check.log
3.8 KiB
verify-fast-final.log
273 KiB
verify-ts-final.log
29 KiB
final-artifacts.json
690 B
author-review-repair.txt
3.1 KiB
perf-cleanup.log
103 B
local-cleanup-final.txt
505 B
verify-fast-interim-alias.log
144 KiB
verify-fast-interim-tmpdir.log
159 KiB
delta-focused-store.log
1.3 KiB
delta-runtime.log
306 B
delta-supervisor.log
67 KiB
delta-state.log
5.9 KiB
delta-property.log
276 B
delta-link-exit-original.log
713 B
delta-link-exit-return.log
713 B
delta-link-exit-panic.log
713 B
delta-candidate-static.txt
201 B
delta-size-pair-final.log
303 B
delta-baseline-repeat-static.txt
166 B
delta-candidate-same-path-static.txt
188 B
delta-perf-long-stall.log
3.3 KiB
delta-link-exit-return-unique.log
713 B
delta-link-exit-panic-unique.log
713 B
delta-perf-store.log
1.3 KiB
delta-perf-latency-baseline.log
1.8 KiB
delta-verify-fast.log
276 KiB
delta-verify-ts.log
29 KiB
delta-final-static-hash.txt
182 B
delta-strengthened-return.log
713 B
delta-strengthened-panic.log
713 B
delta-review-cli-clippy.log
192 B
delta-perf-cleanup-check.log
42 B
delta-perf-cleanup.log
32 B
delta-property-final.log
665 B
delta-property-clippy.log
1.8 KiB
artifacts.json
1.2 KiB
web-comparison.json
560 B
source-provenance.json
699 B
release-build.log
26 KiB
browser-summary.json
222 B
browser-restart.log
1.5 KiB
browser-liveness.log
3 KiB
browser-three-window.log
4.3 KiB
browser-mock.log
1.2 KiB
browser-cleanup.json
2.6 KiB
browser-directory-cleanup.txt
38 B
fixture-source-hashes.json
984 B
service-harness.json
741 B
staging-proof.txt
7.5 KiB
staging-command.log
16 KiB
browser-captures.log
1.5 KiB
capture-cleanup.json
580 B
restart-locked-ready.png
53 KiB
restart-unlock-ready.png
35 KiB
service-proof-final.log
1.7 KiB
service-proof.log
858 B
service-proof-retry.log
2.1 KiB
mode-fixture-runtime.json
866 B
verify-fast-final.log
272 KiB
verify-fast.log
177 KiB
verify-typescript.log
29 KiB
socket-path-recheck.log
209 B
gate-artifact-identity.json
862 B
completion-receipt.json
2.7 KiB
author-review.txt
4 KiB
local-cleanup-phase1.json
1.4 KiB
browser-scopes-cleanup.json
451 B
vm-final-cleanup.json
286 B
release-SHA256SUMS.txt
399 B
perf-fixed.log
3.4 KiB
operational-sources.txt
31 KiB
service-cleanup.json
1.6 KiB
service-final-sources.txt
21 KiB
perf-native-baseline.log
2.7 KiB
restart-locked-ready.png restart-unlock-ready.png
kayg commented 2026-10-15 15:04:38 +00:00
Author
Owner
Copy link

Starting #225 on branch agent-reconnect at base fa904fdaa8. Read issue body and all comments (none; no parent linked), CLAUDE.md, decisions, architecture and process. Will reproduce isolated old-process outage/cold restart, add regression coverage before fixes, supervise reconnect lifecycle and document field recovery. No real hosts or owner fixtures will be touched; push only agent-reconnect.

Starting #225 on branch agent-reconnect at base fa904fdaa8d6e28c0ffeee8550ff1d462c169817. Read issue body and all comments (none; no parent linked), CLAUDE.md, decisions, architecture and process. Will reproduce isolated old-process outage/cold restart, add regression coverage before fixes, supervise reconnect lifecycle and document field recovery. No real hosts or owner fixtures will be touched; push only agent-reconnect.
kayg commented 2026-10-15 15:11:54 +00:00
Author
Owner
Copy link

Source findings: 9ab4329c has no remote_server_path.rs. Its remote_client.rs:70–154 owns reconnect directly; main.rs:520 awaits remote::run, so a detached link-task exit is not established as the field root cause. The old loop can return through renewal.current/require_current, TLS configuration, UDP bind/endpoint setup. fa904fdaa remote_server_path.rs additionally propagates initialize_paths/current and warm-option errors. Both installers use one shared rootful/rootless template at crates/plugins/ssh/src/install.rs:298 with Restart=on-failure, not Restart=always. D168 already separates public retained transport identity from the locked signer (remote_startup.rs:69–113); existing enrolled hosts should reconnect while locked. Building isolated process proof and running regressions before fixes; no production contact.

Source findings: 9ab4329c has no remote_server_path.rs. Its remote_client.rs:70–154 owns reconnect directly; main.rs:520 awaits remote::run, so a detached link-task exit is not established as the field root cause. The old loop can return through renewal.current/require_current, TLS configuration, UDP bind/endpoint setup. fa904fdaa remote_server_path.rs additionally propagates initialize_paths/current and warm-option errors. Both installers use one shared rootful/rootless template at crates/plugins/ssh/src/install.rs:298 with Restart=on-failure, not Restart=always. D168 already separates public retained transport identity from the locked signer (remote_startup.rs:69–113); existing enrolled hosts should reconnect while locked. Building isolated process proof and running regressions before fixes; no production contact.
kayg commented 2026-10-15 15:24:37 +00:00
Author
Owner
Copy link

Regressions before fixes: unavailable_credentials_do_not_end_server_link: 1 failed (real-time test; an initial paused-clock version was discarded because it timed out waiting for the worker). Installed service restart contract: 1 failed. Service completion/shutdown: 2 failed (a successful link return was accepted; runtime destruction waited for unfinished blocking work). Receipts below. The isolated 9ab4329c server + infraternal agent established a real terminal, then the private namespace loopback was disabled; the old server was abruptly stopped. Ten-minute outage is running, followed by fa904fdaa cold server replacement. This has not yet established the field root cause.

red-corrected.log
install-red.log
service-red.log

Regressions before fixes: unavailable_credentials_do_not_end_server_link: 1 failed (real-time test; an initial paused-clock version was discarded because it timed out waiting for the worker). Installed service restart contract: 1 failed. Service completion/shutdown: 2 failed (a successful link return was accepted; runtime destruction waited for unfinished blocking work). Receipts below. The isolated 9ab4329c server + infraternal agent established a real terminal, then the private namespace loopback was disabled; the old server was abruptly stopped. Ten-minute outage is running, followed by fa904fdaa cold server replacement. This has not yet established the field root cause. [red-corrected.log](https://git.kayg.org/attachments/6a301da8-a8bb-47d2-9186-811778aa63ef) [install-red.log](https://git.kayg.org/attachments/9c2a21b5-ffe8-4091-a9ba-ee5116f07dbc) [service-red.log](https://git.kayg.org/attachments/665cb132-7f89-4777-b4b5-63c65caa6303)
kayg commented 2026-10-15 15:28:13 +00:00
Author
Owner
Copy link

Step complete, pushed 923cab7ea: retry ownership now covers credential validation, local service startup and the full path loop. Five continuous offline minutes cancel/recreate the link and local plugin service; a healthy link has no watchdog timer. Network/local errors retry with 200 ms exponential backoff capped at 30 s + 0–99 ms jitter. Initial unavailable credentials retry without creating local state; uid checks and TLS verification remain enforced. fmt passed; touched-crate clippy --all-targets --features remote passed; remote_client tests: 26 passed, 0 failed, 3 ignored. Deterministic tests cover 120 virtual advances of 31 seconds, returned errors/success, watchdog cancellation, and a healthy link beyond its offline deadline. The ten-minute old-process outage is still running; no field root-cause claim yet.

Step complete, pushed 923cab7ea: retry ownership now covers credential validation, local service startup and the full path loop. Five continuous offline minutes cancel/recreate the link and local plugin service; a healthy link has no watchdog timer. Network/local errors retry with 200 ms exponential backoff capped at 30 s + 0–99 ms jitter. Initial unavailable credentials retry without creating local state; uid checks and TLS verification remain enforced. fmt passed; touched-crate clippy --all-targets --features remote passed; remote_client tests: 26 passed, 0 failed, 3 ignored. Deterministic tests cover 120 virtual advances of 31 seconds, returned errors/success, watchdog cancellation, and a healthy link beyond its offline deadline. The ten-minute old-process outage is still running; no field root-cause claim yet.
kayg commented 2026-10-15 15:31:00 +00:00
Author
Owner
Copy link

Isolated real-process result: the unmodified 9ab4329c infraternal agent connected to a 9ab4329c server, stayed alive through 600 seconds with its namespace loopback disabled and the server abruptly stopped, then reconnected to a cold-locked fa904fdaa server. Terminal open succeeded before any CA unlock. This default-dial sequence PASSED: it does not reproduce or establish the field root cause. A TCP-only attempt failed opening the first terminal before the outage and is a separate diagnostic, not an outage reproduction. The process test passed; its surrounding shell subsequently returned an error because the script was edited during execution. Future runs use a stable copied runner. No production or owner fixtures were touched.

old-process-outage.log
old-tcp-outage.log

Isolated real-process result: the unmodified 9ab4329c infraternal agent connected to a 9ab4329c server, stayed alive through 600 seconds with its namespace loopback disabled and the server abruptly stopped, then reconnected to a cold-locked fa904fdaa server. Terminal open succeeded before any CA unlock. This default-dial sequence PASSED: it does not reproduce or establish the field root cause. A TCP-only attempt failed opening the first terminal before the outage and is a separate diagnostic, not an outage reproduction. The process test passed; its surrounding shell subsequently returned an error because the script was edited during execution. Future runs use a stable copied runner. No production or owner fixtures were touched. [old-process-outage.log](https://git.kayg.org/attachments/b6f8aee8-fcc8-4796-913a-d0f25b06f090) [old-tcp-outage.log](https://git.kayg.org/attachments/ee04d6ce-5d7a-4a10-b653-2f51e85a3b1e)
kayg commented 2026-10-15 15:34:04 +00:00
Author
Owner
Copy link

Orchestrator: the field outage was NOT "server unreachable". Reproduce THIS instead:

  1. ~17:30–17:57 IST: monitor01's disk I/O stalled (VM storage). Busy 25% → 40%, tasks stalled on I/O 37% of the time, and the journal stopped writing at 17:57.
  2. ~17:57 to about 19:00: the server process was ALIVE but I/O-stuck. TCP was accepted, /healthz returned 200 from memory, / returned 500, and NetBird SSH failed. Agents could probably connect, or stay connected, to a server that never answered or answered with errors.
  3. ~19:00: the VM was hard-reset (no route), then booted at 19:34 with the CA cold/locked. At 20:01 there was a deploy restart, and at 20:30 the CA was unlocked.

Field agent on infraternal-dev (pid 1308894, old 20,333,104 B binary), inspected read-only:

  • it has one thread, sleeping, 9.9 MB RSS (a tokio multi-thread runtime would have several);
  • it is non-dumpable (fd/stack/syscall are denied);
  • its shim child, pid 1309045, is alive.

So the agent's async runtime looks finished or parked while main waits, for example on the shim child. Reproduce with a server that accepts TCP and then stalls: SIGSTOP the server, or block its disk with a FUSE or dm-delay device. Then SIGKILL it, and restart it cold. Check the thread count of the old agent afterwards.

Orchestrator: the field outage was NOT "server unreachable". Reproduce THIS instead: 1. **~17:30–17:57 IST:** monitor01's disk I/O stalled (VM storage). Busy 25% → 40%, tasks stalled on I/O 37% of the time, and the journal stopped writing at 17:57. 2. **~17:57 to about 19:00:** the server process was ALIVE but I/O-stuck. TCP was accepted, `/healthz` returned 200 from memory, `/` returned 500, and NetBird SSH failed. Agents could probably connect, or stay connected, to a server that never answered or answered with errors. 3. **~19:00:** the VM was hard-reset (no route), then booted at 19:34 with the CA cold/locked. At 20:01 there was a deploy restart, and at 20:30 the CA was unlocked. Field agent on infraternal-dev (pid 1308894, old 20,333,104 B binary), inspected read-only: - it has **one thread**, sleeping, 9.9 MB RSS (a tokio multi-thread runtime would have several); - it is non-dumpable (fd/stack/syscall are denied); - its shim child, pid 1309045, is alive. So the agent's async runtime looks finished or parked while main waits, for example on the shim child. Reproduce with a server that **accepts TCP and then stalls**: SIGSTOP the server, or block its disk with a FUSE or dm-delay device. Then SIGKILL it, and restart it cold. Check the thread count of the old agent afterwards.
kayg commented 2026-10-15 15:35:33 +00:00
Author
Owner
Copy link

Resuming #225 on agent-reconnect at 6f200539e (base fa904fdaa). Read all six issue comments; no parent linked. Continue existing uncommitted process harness with SIGSTOP of the connected old server, then SIGKILL/no route/cold replacement. Important correction: 9ab4329c main.rs:491 explicitly builds new_current_thread(), so one thread is expected and does not prove runtime destruction. No real host or owner fixture will be contacted.

Resuming #225 on agent-reconnect at 6f200539e (base fa904fdaa). Read all six issue comments; no parent linked. Continue existing uncommitted process harness with SIGSTOP of the connected old server, then SIGKILL/no route/cold replacement. Important correction: 9ab4329c main.rs:491 explicitly builds new_current_thread(), so one thread is expected and does not prove runtime destruction. No real host or owner fixture will be contacted.
kayg commented 2026-10-15 15:40:47 +00:00
Author
Owner
Copy link

Completed/pushed logical steps after 923cab7ea: a9b3c54ed exposes existing nix socket controls without a new dependency; b534c00bf uses try_read for reconnect credentials so blocked certificate publication cannot park the async executor; ba9873de3 turns any returned/panicked link into nonzero service exit with bounded runtime shutdown; 2846109f3 adds outgoing kernel TCP keepalive (30s idle, 10s interval, 3 probes; Linux unacknowledged writes limited to 60s); 6f200539e generates Restart=always for both install modes. Tests before fixes failed; current focused receipt: remote transport/renewal 16 passed, service shutdown 2 passed, installer 49 passed; touched-crate clippy passed. These prove defensive failure paths, not the production incident cause.

691c4e4e4 documents one-time field recovery in docs/verification/agent-reconnect-recovery.md. Rootful: sudo systemctl restart infraternal-agent.service. Rootless (installation account): systemctl --user restart infraternal-agent.service. A disconnected process cannot receive a server-side update; retain identity and shims, verify actual host admission after restart. No real host was touched.

Stalled-server proof: current-schema harness cannot bootstrap the old server (auth initialization refused); rebuilding the identical test at the historical revision so its initial schema matches. This startup-only failure is not an outage result.

modules-green.log

Completed/pushed logical steps after 923cab7ea: a9b3c54ed exposes existing nix socket controls without a new dependency; b534c00bf uses try_read for reconnect credentials so blocked certificate publication cannot park the async executor; ba9873de3 turns any returned/panicked link into nonzero service exit with bounded runtime shutdown; 2846109f3 adds outgoing kernel TCP keepalive (30s idle, 10s interval, 3 probes; Linux unacknowledged writes limited to 60s); 6f200539e generates Restart=always for both install modes. Tests before fixes failed; current focused receipt: remote transport/renewal 16 passed, service shutdown 2 passed, installer 49 passed; touched-crate clippy passed. These prove defensive failure paths, not the production incident cause. 691c4e4e4 documents one-time field recovery in docs/verification/agent-reconnect-recovery.md. Rootful: sudo systemctl restart infraternal-agent.service. Rootless (installation account): systemctl --user restart infraternal-agent.service. A disconnected process cannot receive a server-side update; retain identity and shims, verify actual host admission after restart. No real host was touched. Stalled-server proof: current-schema harness cannot bootstrap the old server (auth initialization refused); rebuilding the identical test at the historical revision so its initial schema matches. This startup-only failure is not an outage result. [modules-green.log](https://git.kayg.org/attachments/ff90783e-5711-46cb-9457-3d47f0544a9f)
kayg commented 2026-10-15 15:53:23 +00:00
Author
Owner
Copy link

Finding: discarding both directions of :7443 with private namespace route rules before terminating the server exposes a half-open TCP link that simple loopback-down tests can miss. Old 9ab4329c: after SIGSTOP then SIGKILL and 120s of packet discard, process alive, Threads=1, TCP still ESTABLISHED (127.0.0.1:45676 -> 127.0.0.1:7443). Fixed build: TimedOut logged, old established TCP entry absent. The old loop has not returned in this path: remote_client.rs awaits daemon dispatch -> remote_tls::TcpTls::read, which has no established-idle read deadline or TCP keepalive in that release. This is a reproduced failure mechanism, not proof of every field process state.

Both terminal-admission assertions in that first short run are INVALID: the fixture browser ignored the newer server protocol Ping and was closed. Updating the browser fixture to answer correlated Ping/Pong before accepting a red/green recovery verdict. Ten-minute stall runs and baseline static-size build continue. No production contact.

old-stalled-tcp-blackhole.log
fixed-stalled-tcp-blackhole.log

Finding: discarding both directions of :7443 with private namespace route rules before terminating the server exposes a half-open TCP link that simple loopback-down tests can miss. Old 9ab4329c: after SIGSTOP then SIGKILL and 120s of packet discard, process alive, Threads=1, TCP still ESTABLISHED (127.0.0.1:45676 -> 127.0.0.1:7443). Fixed build: TimedOut logged, old established TCP entry absent. The old loop has not returned in this path: remote_client.rs awaits daemon dispatch -> remote_tls::TcpTls::read, which has no established-idle read deadline or TCP keepalive in that release. This is a reproduced failure mechanism, not proof of every field process state. Both terminal-admission assertions in that first short run are INVALID: the fixture browser ignored the newer server protocol Ping and was closed. Updating the browser fixture to answer correlated Ping/Pong before accepting a red/green recovery verdict. Ten-minute stall runs and baseline static-size build continue. No production contact. [old-stalled-tcp-blackhole.log](https://git.kayg.org/attachments/e103540d-4cf8-4e16-b76b-43841b959918) [fixed-stalled-tcp-blackhole.log](https://git.kayg.org/attachments/871efb35-5ee9-467b-bab7-ab311865d560)
kayg commented 2026-10-15 16:08:08 +00:00
Author
Owner
Copy link

Valid process red/green now established with the corrected browser helper (answers protocol Ping/Pong). Both runs: old 9ab4329c server, connected infraternal agent, SIGSTOP while new TCP connects still succeed, discard both directions of :7443, SIGKILL, 120s without transport packets, cold old restart, then cold fa904fdaa replacement. Old client: process alive, Threads=1, stale ESTABLISHED TCP entry, then the explicit recovery deadline failed: “infraternal agent did not reconnect after cold server replacement: Elapsed(())”. Fixed client: kernel TimedOut, stale connection absent, reconnected and opened a terminal while the replacement CA remained locked; test 1 passed, 0 failed. Separate fixed 600s SIGSTOP + 120s packet-discard run also passed (1 passed, 0 failed). These are build-host functional fault-injection proofs, not latency benchmarks.

Root failure mechanism: in 9ab4329c remote_client.rs:123–141 awaits daemon dispatch on the TCP fallback. crates/plugins/remote/src/remote_tls.rs:60–71 waits for an idle read; dial:188–207 configures no kernel keepalive. A reset without FIN/RST leaves that await live indefinitely, so run() does not reach backoff and systemd sees no exit. One thread is normal for its current-thread runtime. The new socket keepalive/write timeout fixes this path; the supervisor/watchdog/process-exit layers also cover returned local errors. This reproduces the reported outage shape without asserting that the field’s inaccessible descriptors have been examined.

The socket-option regression failed before the change (KeepAlive false) and now passes; logs include that original red receipt. Still completing stripped-static size comparison and final fast gate; not reporting done yet.

old-final-tcp-blackhole.log
fixed-final-tcp-blackhole.log
fixed-long-stalled-tcp-blackhole.log
tcp-red.log

Valid process red/green now established with the corrected browser helper (answers protocol Ping/Pong). Both runs: old 9ab4329c server, connected infraternal agent, SIGSTOP while new TCP connects still succeed, discard both directions of :7443, SIGKILL, 120s without transport packets, cold old restart, then cold fa904fdaa replacement. Old client: process alive, Threads=1, stale ESTABLISHED TCP entry, then the explicit recovery deadline failed: “infraternal agent did not reconnect after cold server replacement: Elapsed(())”. Fixed client: kernel TimedOut, stale connection absent, reconnected and opened a terminal while the replacement CA remained locked; test 1 passed, 0 failed. Separate fixed 600s SIGSTOP + 120s packet-discard run also passed (1 passed, 0 failed). These are build-host functional fault-injection proofs, not latency benchmarks. Root failure mechanism: in 9ab4329c remote_client.rs:123–141 awaits daemon dispatch on the TCP fallback. crates/plugins/remote/src/remote_tls.rs:60–71 waits for an idle read; dial:188–207 configures no kernel keepalive. A reset without FIN/RST leaves that await live indefinitely, so run() does not reach backoff and systemd sees no exit. One thread is normal for its current-thread runtime. The new socket keepalive/write timeout fixes this path; the supervisor/watchdog/process-exit layers also cover returned local errors. This reproduces the reported outage shape without asserting that the field’s inaccessible descriptors have been examined. The socket-option regression failed before the change (KeepAlive false) and now passes; logs include that original red receipt. Still completing stripped-static size comparison and final fast gate; not reporting done yet. [old-final-tcp-blackhole.log](https://git.kayg.org/attachments/84507a44-0611-4ca1-95cd-368c963d7c4b) [fixed-final-tcp-blackhole.log](https://git.kayg.org/attachments/201a3ebe-913a-46e0-a4f2-bf66965deb87) [fixed-long-stalled-tcp-blackhole.log](https://git.kayg.org/attachments/9cfaddef-252f-4443-853f-c7977b7dc894) [tcp-red.log](https://git.kayg.org/attachments/76f77c3f-6706-4d32-ac59-b2954e93e00b)
kayg commented 2026-10-15 16:21:36 +00:00
Author
Owner
Copy link

Pushed 876d343c2 (packet-discard regression defaults, browser proof replies, portable fixture measurement hooks) and 52074bc68 (precise half-open read path/recovery instructions). fmt, touched-server clippy: passed; focused remote_demo suite: 6 passed, 0 failed, 6 ignored. Worktree clean; no deployment.

Baseline fa904fdaa static x86_64 release built with the repository recipe: 22,329,104 bytes, static PIE, stripped, SHA256 b9917316111e743f4b485b034011164691ebf5125b51403e90f93c7e6a16cdf6. This is already 124,800 bytes above the unchanged 22,204,304-byte ceiling. Head delta is still being built by scripts/verify fast through build-slot; no ceiling increased. Performance-VM trial is queued on the shared lock; no build-host runtime numbers will be used as acceptance evidence.

Line correction to the prior comment: the old dial configures nodelay at remote_tls.rs:179 (not the later accept path); the indefinitely awaited read remains remote_tls.rs:70.

Pushed 876d343c2 (packet-discard regression defaults, browser proof replies, portable fixture measurement hooks) and 52074bc68 (precise half-open read path/recovery instructions). fmt, touched-server clippy: passed; focused remote_demo suite: 6 passed, 0 failed, 6 ignored. Worktree clean; no deployment. Baseline fa904fdaa static x86_64 release built with the repository recipe: 22,329,104 bytes, static PIE, stripped, SHA256 b9917316111e743f4b485b034011164691ebf5125b51403e90f93c7e6a16cdf6. This is already 124,800 bytes above the unchanged 22,204,304-byte ceiling. Head delta is still being built by scripts/verify fast through build-slot; no ceiling increased. Performance-VM trial is queued on the shared lock; no build-host runtime numbers will be used as acceptance evidence. Line correction to the prior comment: the old dial configures nodelay at remote_tls.rs:179 (not the later accept path); the indefinitely awaited read remains remote_tls.rs:70.
kayg commented 2026-10-15 16:38:03 +00:00
Author
Owner
Copy link

The corrected ten-minute old-release run now fails the intended reconnect deadline (not a browser lease): 600s SIGSTOP, 120s transport packet discard, cold old/new server restarts; process remains alive with Threads=1 and a stale established TCP socket. Explicit recovery deadline: Elapsed(()).

Independent production-static baseline trial on the perf VM also fails the intended recovery deadline. Artifact: fa904fdaa, SHA256 b9917316111e743f4b485b034011164691ebf5125b51403e90f93c7e6a16cdf6. Machine infraternal-perf-test, 4 cores, MemTotal 8,069,112 KiB; load at start 1.00/0.84/0.72, finish 0.23/0.50/0.61. Full-daemon RSS 15,876 KiB in this workflow (two unviewed PTY opens); one measured daemon × 1,000 distributed nodes = 15,876,000 KiB extrapolated, not a 1,000-daemon concurrency measurement or a bare-idle budget claim. Candidate measurement awaits its static artifact. The shared VM lock was held only for the trial and has been released.

Native process-test fixtures needed their exact existing C runtime bundled under the job directory because their Nix loader path is absent on the VM; only those fixture ELF loader/rpath fields were relocated. The production-static daemon was copied unchanged. The measurement runs in private user/network/mount namespaces with a synthetic UID/account file and private shell home, so it does not read an existing account profile or alter host account files.

Full fast gate on 52074bc68: fmt, native fmt, compile-out and workspace clippy have passed; full tests are compiling/running next. Final gate output pending.

old-final-long-tcp-blackhole.log
perf-baseline.log

The corrected ten-minute old-release run now fails the intended reconnect deadline (not a browser lease): 600s SIGSTOP, 120s transport packet discard, cold old/new server restarts; process remains alive with Threads=1 and a stale established TCP socket. Explicit recovery deadline: Elapsed(()). Independent production-static baseline trial on the perf VM also fails the intended recovery deadline. Artifact: fa904fdaa, SHA256 b9917316111e743f4b485b034011164691ebf5125b51403e90f93c7e6a16cdf6. Machine infraternal-perf-test, 4 cores, MemTotal 8,069,112 KiB; load at start 1.00/0.84/0.72, finish 0.23/0.50/0.61. Full-daemon RSS 15,876 KiB in this workflow (two unviewed PTY opens); one measured daemon × 1,000 distributed nodes = 15,876,000 KiB extrapolated, not a 1,000-daemon concurrency measurement or a bare-idle budget claim. Candidate measurement awaits its static artifact. The shared VM lock was held only for the trial and has been released. Native process-test fixtures needed their exact existing C runtime bundled under the job directory because their Nix loader path is absent on the VM; only those fixture ELF loader/rpath fields were relocated. The production-static daemon was copied unchanged. The measurement runs in private user/network/mount namespaces with a synthetic UID/account file and private shell home, so it does not read an existing account profile or alter host account files. Full fast gate on 52074bc68: fmt, native fmt, compile-out and workspace clippy have passed; full tests are compiling/running next. Final gate output pending. [old-final-long-tcp-blackhole.log](https://git.kayg.org/attachments/e360b670-aa01-451d-a894-539bebc67b48) [perf-baseline.log](https://git.kayg.org/attachments/5c71a379-3b36-4dbc-ae2b-250c40ba2eeb)
kayg commented 2026-10-15 16:45:23 +00:00
Author
Owner
Copy link

Author review finding at remote_server_path.rs:570–575 / plugin_control.rs:99–121: swallowing initialize_paths failure can retain a dead local control worker while the fallback link is marked connected. Registry consumes its one-shot factory before spawning; a spawn/factory failure or disconnected worker cannot heal inside that registry. The existing supervisor recreates the local service only when the attempt returns or stays offline. Adding a regression before changing the path loop to return this local failure to its retry owner. This stays within #225 all-error supervision; no wire or foundational dependency change.

Author review finding at remote_server_path.rs:570–575 / plugin_control.rs:99–121: swallowing initialize_paths failure can retain a dead local control worker while the fallback link is marked connected. Registry consumes its one-shot factory before spawning; a spawn/factory failure or disconnected worker cannot heal inside that registry. The existing supervisor recreates the local service only when the attempt returns or stays offline. Adding a regression before changing the path loop to return this local failure to its retry owner. This stays within #225 all-error supervision; no wire or foundational dependency change.
kayg commented 2026-10-15 16:53:02 +00:00
Author
Owner
Copy link

Pushed a32a403244: corrected our best-effort initialization handling so a failed one-shot local worker returns to the retry owner, which recreates the service. No preference remains a successful discovery result. Regression first failed with “failed worker must return to the owner that recreates its service: Elapsed(())”; now remote-client suite 27 passed, 0 failed, 3 ignored; control suite 8 passed, 0 failed; fmt and touched-agent clippy passed. Receipts below.

The in-progress full gate has passed its original workspace lint, test and dependency phases (zero failing tests), and is compiling static binaries. Since author review added this correction, that run is interim: the final fast gate will be rerun on a32a40324 once these static dependencies are cached. Candidate size and perf-VM trial are still pending.

worker-red-corrected.log
worker-green.log
worker-control-tests.log
worker-clippy.log

Pushed a32a403244586fc31fdcd5148176da5a730482df: corrected our best-effort initialization handling so a failed one-shot local worker returns to the retry owner, which recreates the service. No preference remains a successful discovery result. Regression first failed with “failed worker must return to the owner that recreates its service: Elapsed(())”; now remote-client suite 27 passed, 0 failed, 3 ignored; control suite 8 passed, 0 failed; fmt and touched-agent clippy passed. Receipts below. The in-progress full gate has passed its original workspace lint, test and dependency phases (zero failing tests), and is compiling static binaries. Since author review added this correction, that run is interim: the final fast gate will be rerun on a32a40324 once these static dependencies are cached. Candidate size and perf-VM trial are still pending. [worker-red-corrected.log](https://git.kayg.org/attachments/bfcb807b-3bf0-4b74-9cc7-2f6fcbebcfae) [worker-green.log](https://git.kayg.org/attachments/f4487c13-090c-4837-89e7-086923c9f085) [worker-control-tests.log](https://git.kayg.org/attachments/5191ee29-f0a6-4eda-8f79-019a92a550c9) [worker-clippy.log](https://git.kayg.org/attachments/3c680fcd-b9d7-4685-9cc9-1915ecfed38b)
kayg commented 2026-10-15 17:14:05 +00:00
Author
Owner
Copy link

Perf-VM candidate trial passed terminal recovery, but listener-ready to terminal admission was 33,031,085 us (one trial, load 0.21/0.37/0.70), exceeding the requested one 30s+jitter backoff period. The loop can spend its dial deadline and then sleep a full capped backoff. Adding a virtual-time regression before charging attempt duration against the shared retry interval; will repeat the production-static VM trial. No budget or protocol change. Existing candidate artifact: SHA256 29c1e689981fbad5c5cddc9f9c84de45058b4adc6482561fcd3ca325b25f98fd. Stripped x86_64 size 22,361,296 B, +32,192 B versus fa904fdaa. The absolute ceiling remains unchanged and already fails on the baseline.

Perf-VM candidate trial passed terminal recovery, but listener-ready to terminal admission was 33,031,085 us (one trial, load 0.21/0.37/0.70), exceeding the requested one 30s+jitter backoff period. The loop can spend its dial deadline and then sleep a full capped backoff. Adding a virtual-time regression before charging attempt duration against the shared retry interval; will repeat the production-static VM trial. No budget or protocol change. Existing candidate artifact: SHA256 29c1e689981fbad5c5cddc9f9c84de45058b4adc6482561fcd3ca325b25f98fd. Stripped x86_64 size 22,361,296 B, +32,192 B versus fa904fdaa. The absolute ceiling remains unchanged and already fails on the baseline.
kayg commented 2026-10-15 17:29:27 +00:00
Author
Owner
Copy link

Pushed da0abf751 (charge attempt work against the retry interval, share cap/jitter calculation) and 8003e213 (tighten the process proof’s functional admission deadline). The new virtual-time regression failed before the fix with a 35.094s attempt interval; it and the remote-client suite now pass: 28 passed, 0 failed, 3 ignored. Touched-crate clippy and six focused server tests pass. The failed-attempt interval now includes dial work, with 0–99ms jitter added to its remaining wait; stable sessions retain the short reconnect cool-off.

Previous production-static candidate trial on perf VM: terminal admission succeeded while CA locked, but took 33,031,085 us from listener-ready, so this is NOT accepted as the final timing result. Matching after-stall RSS: baseline 15,876 KiB, prior candidate 16,040 KiB (one measured daemon each, two unviewed PTYs; +164 KiB). After reconnect the prior candidate was 16,180 KiB with one additional PTY. No bare-idle or 1,000-daemon concurrency claim. New static artifact and repeated VM trial pending while the interim fast run completes its ARM build. Final fast gate will run on the last committed head. No budget increased; no production action.

cadence-red.log
cadence-green.log
cadence-clippy.log
perf-fixed.log

Pushed da0abf751 (charge attempt work against the retry interval, share cap/jitter calculation) and 8003e213 (tighten the process proof’s functional admission deadline). The new virtual-time regression failed before the fix with a 35.094s attempt interval; it and the remote-client suite now pass: 28 passed, 0 failed, 3 ignored. Touched-crate clippy and six focused server tests pass. The failed-attempt interval now includes dial work, with 0–99ms jitter added to its remaining wait; stable sessions retain the short reconnect cool-off. Previous production-static candidate trial on perf VM: terminal admission succeeded while CA locked, but took 33,031,085 us from listener-ready, so this is NOT accepted as the final timing result. Matching after-stall RSS: baseline 15,876 KiB, prior candidate 16,040 KiB (one measured daemon each, two unviewed PTYs; +164 KiB). After reconnect the prior candidate was 16,180 KiB with one additional PTY. No bare-idle or 1,000-daemon concurrency claim. New static artifact and repeated VM trial pending while the interim fast run completes its ARM build. Final fast gate will run on the last committed head. No budget increased; no production action. [cadence-red.log](https://git.kayg.org/attachments/8a56f923-de19-4a73-b457-a30df5f81217) [cadence-green.log](https://git.kayg.org/attachments/18922d80-4319-47d8-9c14-ef14f59b4947) [cadence-clippy.log](https://git.kayg.org/attachments/0dd11455-a9bb-46ce-a43f-1d61e5915c0c) [perf-fixed.log](https://git.kayg.org/attachments/27fd1db3-9075-4b23-b5e0-f30f170f158d)
kayg commented 2026-10-15 17:45:14 +00:00
Author
Owner
Copy link

Interim full build-slot fast gate completed. fmt, native fmt, CLI plugin compile-out, workspace clippy/tests and cargo-deny PASS; static size checks FAIL. Interim x86: 22,361,296 B vs cap 22,204,304; independently built fa904fdaa baseline: 22,329,104 B (already 124,800 B over cap). Interim branch delta +32,192 B. ARM interim: 19,784,856 B vs cap 19,638,744 (no ARM baseline measured; no claim of pre-existing ARM debt). Ceilings unchanged. verify-fast-interim.log

Final full fast gate now running on pushed 8003e2134, including cadence fix; final production-static VM measurement will use that rebuilt binary. These are gate receipts, not build-host performance evidence.

Interim full build-slot fast gate completed. fmt, native fmt, CLI plugin compile-out, workspace clippy/tests and cargo-deny PASS; static size checks FAIL. Interim x86: 22,361,296 B vs cap 22,204,304; independently built fa904fdaa baseline: 22,329,104 B (already 124,800 B over cap). Interim branch delta +32,192 B. ARM interim: 19,784,856 B vs cap 19,638,744 (no ARM baseline measured; no claim of pre-existing ARM debt). Ceilings unchanged. [verify-fast-interim.log](https://git.kayg.org/attachments/7c82c2c6-10e7-4f83-bfcd-bd04a9aea48b) Final full fast gate now running on pushed 8003e2134, including cadence fix; final production-static VM measurement will use that rebuilt binary. These are gate receipts, not build-host performance evidence.
kayg commented 2026-10-15 18:06:24 +00:00
Author
Owner
Copy link

Implementation and validation finished on pushed head 8003e2134d (agent-reconnect only; no deployment). Static size gate remains blocked; ceilings unchanged.

Root cause reproduced with real 9ab4329c processes: old remote_client.rs:123 awaits daemon dispatch; crates/plugins/remote/src/remote_tls.rs:70 can wait forever on an idle TCP read, and old dial at :179 configures no keepalive. After SIGSTOP (TCP still accepts), packet discard, SIGKILL and cold replacement, the old process remains alive with stale ESTABLISHED TCP and fails recovery. One thread is normal for its current-thread runtime, not evidence of finished runtime. Long600s stopped+120s discard red/green receipts are in earlier comments. No forensic claim about inaccessible field descriptors.

Final production-static VM trial: strict31s recovery test PASS; real terminal admitted6,403,887us after replacement listener ready, CA signer still locked. Cold enrollment remains refused and invitation retained. Same-shape baseline fails recovery. Machine4cores/8,069,112KiB; load0.34/0.10/0.06→0.03/0.06/0.05. Single trial, no percentile claim. Matching two-PTY full-daemon RSS baseline15,876KiB→final15,976KiB (+100KiB);1000-instance arithmetic adds100,000KiB, not a concurrent-scale or bare-idle measurement. Final perf receipt and limits below.

Final build-slot fast:1,090 Rust tests passed,0failed,140ignored; every Rust row except static size PASS. TS phase skipped by failing Rust return was run separately through build-slot and PASS (check, production build, transfer budget and protocol tests37pass/0fail). Final x86 static22,361,616B vs identical merge-base22,329,104B: +32,512B. Baseline already exceeds22,204,304B ceiling by124,800B; branch exceeds by157,312B. ARM19,784,856B vs19,638,744B; ARM baseline unmeasured. Growth above16KiB justified by demonstrated recovery of a permanently disconnected production-static baseline plus lifecycle/watchdog/socket protections, no new crate; below64KiB. Existing ceilings were not raised.

Final fast summary, verbatim (complete transcript attached):

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Recovery for already-stuck old installations requires one restart over an existing administration channel; a server action cannot reach their dead link. Rootful: sudo systemctl restart infraternal-agent.service. Rootless, as installation account: systemctl --user restart infraternal-agent.service. Identity and independent shims retained; no re-enrollment/reinstall. Then verify host admission with an actual terminal. Exact privileged rootless command and checks: docs/verification/agent-reconnect-recovery.md. No real host restarted.

Author checklist attached. VM owned executable process count0 and job directory removed; local temporary target/checkouts cleanup recorded next. Issue remains open for static-size gate disposition and normal review/rollout.

verify-fast-final.log
ts-check.log
perf-fixed-final.log
final-size-and-measurement.txt
author-review.txt
historical-harness.patch
perf-cleanup.log

Implementation and validation finished on pushed head 8003e2134d188182b1c60ab18833a65c610b4ced (agent-reconnect only; no deployment). Static size gate remains blocked; ceilings unchanged. Root cause reproduced with real 9ab4329c processes: old remote_client.rs:123 awaits daemon dispatch; crates/plugins/remote/src/remote_tls.rs:70 can wait forever on an idle TCP read, and old dial at :179 configures no keepalive. After SIGSTOP (TCP still accepts), packet discard, SIGKILL and cold replacement, the old process remains alive with stale ESTABLISHED TCP and fails recovery. One thread is normal for its current-thread runtime, not evidence of finished runtime. Long600s stopped+120s discard red/green receipts are in earlier comments. No forensic claim about inaccessible field descriptors. Final production-static VM trial: strict31s recovery test PASS; real terminal admitted6,403,887us after replacement listener ready, CA signer still locked. Cold enrollment remains refused and invitation retained. Same-shape baseline fails recovery. Machine4cores/8,069,112KiB; load0.34/0.10/0.06→0.03/0.06/0.05. Single trial, no percentile claim. Matching two-PTY full-daemon RSS baseline15,876KiB→final15,976KiB (+100KiB);1000-instance arithmetic adds100,000KiB, not a concurrent-scale or bare-idle measurement. Final perf receipt and limits below. Final build-slot fast:1,090 Rust tests passed,0failed,140ignored; every Rust row except static size PASS. TS phase skipped by failing Rust return was run separately through build-slot and PASS (check, production build, transfer budget and protocol tests37pass/0fail). Final x86 static22,361,616B vs identical merge-base22,329,104B: +32,512B. Baseline already exceeds22,204,304B ceiling by124,800B; branch exceeds by157,312B. ARM19,784,856B vs19,638,744B; ARM baseline unmeasured. Growth above16KiB justified by demonstrated recovery of a permanently disconnected production-static baseline plus lifecycle/watchdog/socket protections, no new crate; below64KiB. Existing ceilings were not raised. Final fast summary, verbatim (complete transcript attached): ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` Recovery for already-stuck old installations requires one restart over an existing administration channel; a server action cannot reach their dead link. Rootful: `sudo systemctl restart infraternal-agent.service`. Rootless, as installation account: `systemctl --user restart infraternal-agent.service`. Identity and independent shims retained; no re-enrollment/reinstall. Then verify host admission with an actual terminal. Exact privileged rootless command and checks: docs/verification/agent-reconnect-recovery.md. No real host restarted. Author checklist attached. VM owned executable process count0 and job directory removed; local temporary target/checkouts cleanup recorded next. Issue remains open for static-size gate disposition and normal review/rollout. [verify-fast-final.log](https://git.kayg.org/attachments/10787fef-49ff-4dc4-bb5c-d771f64f59c5) [ts-check.log](https://git.kayg.org/attachments/9ca06cb2-6220-4b47-8ac5-6c46d1ffefaf) [perf-fixed-final.log](https://git.kayg.org/attachments/9a28b504-fbb3-4b4c-ba5c-bce3411d433a) [final-size-and-measurement.txt](https://git.kayg.org/attachments/59d9734e-a049-42e5-9e85-803f2fcc145d) [author-review.txt](https://git.kayg.org/attachments/1b4ea822-a6ac-427b-8ca7-8ad7580367c8) [historical-harness.patch](https://git.kayg.org/attachments/aa630f9d-9468-4c33-aac9-53a34f920bde) [perf-cleanup.log](https://git.kayg.org/attachments/60698099-6637-4702-8238-a68dd6efeac5)
kayg commented 2026-10-15 18:06:42 +00:00
Author
Owner
Copy link

local-cleanup.txt

[local-cleanup.txt](https://git.kayg.org/attachments/f808504b-cd78-404b-a839-87a3ad74ba1c)
kayg commented 2026-10-15 18:08:31 +00:00
Author
Owner
Copy link

Independent review started on review-agent-reconnect at target 8003e2134d; merge base with local dev is 8e7467c07c. Read all 19 comments and issue body; no parent linked. Checking zero-socket return paths, prolonged downtime restart behavior, D128 shim survival, binary growth and final gate. No implementation changes, pushes, deployments or owner-fixture access.

Independent review started on review-agent-reconnect at target 8003e2134d188182b1c60ab18833a65c610b4ced; merge base with local dev is 8e7467c07cd2ec1668a775a846076b278f68ef38. Read all 19 comments and issue body; no parent linked. Checking zero-socket return paths, prolonged downtime restart behavior, D128 shim survival, binary growth and final gate. No implementation changes, pushes, deployments or owner-fixture access.
kayg commented 2026-10-15 18:14:58 +00:00
Author
Owner
Copy link

Base correction: local dev is stale at 8e7467c07; origin/dev is 40cc9c5be and its merge base with target 8003e2134 is fa904fdaa8 (17 files, 910 insertions/60 deletions). Review and size comparison use this current development-line base. Pristine head fast gate is running from a detached scratch snapshot, separately from review regression tests. No parent is linked.

Base correction: local dev is stale at 8e7467c07; origin/dev is 40cc9c5be and its merge base with target 8003e2134 is fa904fdaa8d6e28c0ffeee8550ff1d462c169817 (17 files, 910 insertions/60 deletions). Review and size comparison use this current development-line base. Pristine head fast gate is running from a detached scratch snapshot, separately from review regression tests. No parent is linked.
kayg commented 2026-10-15 18:24:26 +00:00
Author
Owner
Copy link

Independent finding R1 (P2), committed 7cb076a3c on review-agent-reconnect: watchdog cancellation retains synchronous plugin workers. Test watchdog_bounds_workers_when_local_path_discovery_stalls failed with “offline recovery retained 3 synchronous workers; local work must stay bounded” after two virtual five-minute resets. Each attempt uses the real Registry dedicated OS worker; all three workers are released and completion awaited before assertion. Source: remote_supervisor.rs:87 / remote_client.rs:78–104 / plugin_control.rs:117–133; MeshNodePlugin::preferred_server performs synchronous Owner::node/Node::restore. Resource growth is unbounded during a persistent local stall. Evidence: ~/.cache/codex-scratch/agent-reconnect-review/evidence/worker-bound.log; durable entry appended to the review findings.md. No implementation fixes or push.

Independent finding R1 (P2), committed 7cb076a3c on review-agent-reconnect: watchdog cancellation retains synchronous plugin workers. Test watchdog_bounds_workers_when_local_path_discovery_stalls failed with “offline recovery retained 3 synchronous workers; local work must stay bounded” after two virtual five-minute resets. Each attempt uses the real Registry dedicated OS worker; all three workers are released and completion awaited before assertion. Source: remote_supervisor.rs:87 / remote_client.rs:78–104 / plugin_control.rs:117–133; MeshNodePlugin::preferred_server performs synchronous Owner::node/Node::restore. Resource growth is unbounded during a persistent local stall. Evidence: ~/.cache/codex-scratch/agent-reconnect-review/evidence/worker-bound.log; durable entry appended to the review findings.md. No implementation fixes or push.
kayg commented 2026-10-15 18:26:24 +00:00
Author
Owner
Copy link

Step complete c7312f2b7: independent six-hour virtual-clock refusal test passed, 728 attempts, supervisor never returned. This establishes bounded retries inside one process during prolonged ordinary server downtime; it does not measure latency. Both installation modes share Restart=always, RestartSec=2, StartLimitIntervalSec=0 and KillMode=process (install.rs:298). Focused review-test clippy passed after lint cleanup. R1 remains intentionally failing and recorded in 7cb076a3c. Pristine 8003e2134 full fast gate is still running; actual nonzero link-exit/mid-output replay proof is being added separately.

Step complete c7312f2b7: independent six-hour virtual-clock refusal test passed, 728 attempts, supervisor never returned. This establishes bounded retries inside one process during prolonged ordinary server downtime; it does not measure latency. Both installation modes share Restart=always, RestartSec=2, StartLimitIntervalSec=0 and KillMode=process (install.rs:298). Focused review-test clippy passed after lint cleanup. R1 remains intentionally failing and recorded in 7cb076a3c. Pristine 8003e2134 full fast gate is still running; actual nonzero link-exit/mid-output replay proof is being added separately.
kayg commented 2026-10-15 18:35:04 +00:00
Author
Owner
Copy link

Independent read-only field metadata check: original PIDs 1308894 and 1309045 remain present, sleeping, one thread each; shim parent remains 1308894. /proc//fd enumeration is PermissionDenied for both: an all-socket count cannot be established by that method. The parent network namespace tcp/tcp6 tables contain zero same-UID outbound destination-port 7443 entries at observation (namespace-wide, not process attribution). Thus the reproduced stale ESTABLISHED TCP failure remains distinct from the field state, and the exact field root cause is not independently established. No credential/argv/state-file reads, descriptor contents, process tracing, signals or owner-fixture modifications. Receipt: ~/.cache/codex-scratch/agent-reconnect-review/evidence/field-process-metadata.txt. Returned-error supervision and the new bounded shutdown path are being verified separately; do not treat the stale-TCP reproduction as proof of this field cause.

Independent read-only field metadata check: original PIDs 1308894 and 1309045 remain present, sleeping, one thread each; shim parent remains 1308894. /proc/<pid>/fd enumeration is PermissionDenied for both: an all-socket count cannot be established by that method. The parent network namespace tcp/tcp6 tables contain zero same-UID outbound destination-port 7443 entries at observation (namespace-wide, not process attribution). Thus the reproduced stale ESTABLISHED TCP failure remains distinct from the field state, and the exact field root cause is not independently established. No credential/argv/state-file reads, descriptor contents, process tracing, signals or owner-fixture modifications. Receipt: ~/.cache/codex-scratch/agent-reconnect-review/evidence/field-process-metadata.txt. Returned-error supervision and the new bounded shutdown path are being verified separately; do not treat the stale-TCP reproduction as proof of this field cause.
kayg commented 2026-10-15 18:45:41 +00:00
Author
Owner
Copy link

Step complete: committed the independent nonzero-exit replay fixture on review-agent-reconnect (head recorded below). It aborts the actual admitted dispatcher task mid-output, calls the unchanged production remote_service::stopped helper and exits the fixture process with status 1; a replacement process reattaches the same real shim and stable writer. Result: 8/8 status-1 exits, 2,097,152 payload bytes, same shim PID across all cycles, lost=0 duplicated=0. Reuses the existing mux codec/Binary and authenticated dispatcher; no extra parser or implementation change. Small test crate compiled and passed pedantic clippy against pristine head production libraries. This verifies the dispatcher/shutdown boundary, not a live systemd service or the unidentified original field wait. Receipt ~/.cache/codex-scratch/agent-reconnect-review/evidence/link-exit-replay.log.
a44224d36e

Step complete: committed the independent nonzero-exit replay fixture on review-agent-reconnect (head recorded below). It aborts the actual admitted dispatcher task mid-output, calls the unchanged production remote_service::stopped helper and exits the fixture process with status 1; a replacement process reattaches the same real shim and stable writer. Result: 8/8 status-1 exits, 2,097,152 payload bytes, same shim PID across all cycles, lost=0 duplicated=0. Reuses the existing mux codec/Binary and authenticated dispatcher; no extra parser or implementation change. Small test crate compiled and passed pedantic clippy against pristine head production libraries. This verifies the dispatcher/shutdown boundary, not a live systemd service or the unidentified original field wait. Receipt ~/.cache/codex-scratch/agent-reconnect-review/evidence/link-exit-replay.log. a44224d36e6adffb2a843a6676774572b20cfbfe
kayg commented 2026-10-15 19:12:04 +00:00
Author
Owner
Copy link

Independent perf VM candidate result (unchanged production-static SHA256 2cdfed0ff49f1433808d819afed88a47b6d56ce43956c9301c9f5768e45093c9): native LAN benchmark PASS, n=1000 confirmed echo p99=2.49ms; credited flood=6.243MiB/s; observed idle schedule-ins=0; idle RSS=4560KiB, peak after 50MiB=7720KiB (single inherited-fd daemon; not the full remote TLS workflow or a 1000-instance concurrency measurement). Perf VM 4 cores, load start 1.45/1.54/1.04, end 1.48/1.54/1.05.

Separate 300ms RTT +/-30ms native gate FAIL: n=100 same-PTY flood Ack p99=388.29ms, echo p99=388.37ms versus334ms limit; echo growth=60.46ms versus16.67ms limit. Idle echo p99=327.91ms and other-PTY flood echo p99=305.10ms PASS. Load start1.67/1.54/1.08, end0.60/1.19/1.00. Baseline pending; no attribution to this branch yet. Each run used its own CPUWeight10000 scope, synthetic account/profile, private init/network/mount namespaces, shared flock and owner marker; lock released between runs. No benchmark ran on the build host. Receipts: perf-native-candidate.log and perf-latency-candidate.log under the review evidence directory. ARM fast gate build is still running.

Independent perf VM candidate result (unchanged production-static SHA256 2cdfed0ff49f1433808d819afed88a47b6d56ce43956c9301c9f5768e45093c9): native LAN benchmark PASS, n=1000 confirmed echo p99=2.49ms; credited flood=6.243MiB/s; observed idle schedule-ins=0; idle RSS=4560KiB, peak after 50MiB=7720KiB (single inherited-fd daemon; not the full remote TLS workflow or a 1000-instance concurrency measurement). Perf VM 4 cores, load start 1.45/1.54/1.04, end 1.48/1.54/1.05. Separate 300ms RTT +/-30ms native gate FAIL: n=100 same-PTY flood Ack p99=388.29ms, echo p99=388.37ms versus334ms limit; echo growth=60.46ms versus16.67ms limit. Idle echo p99=327.91ms and other-PTY flood echo p99=305.10ms PASS. Load start1.67/1.54/1.08, end0.60/1.19/1.00. Baseline pending; no attribution to this branch yet. Each run used its own CPUWeight10000 scope, synthetic account/profile, private init/network/mount namespaces, shared flock and owner marker; lock released between runs. No benchmark ran on the build host. Receipts: perf-native-candidate.log and perf-latency-candidate.log under the review evidence directory. ARM fast gate build is still running.
kayg commented 2026-10-15 19:36:54 +00:00
Author
Owner
Copy link

P1 / HOLD — R1 strengthened and committed as 32e780043 (count R1 once at P1, replacing its earlier P2 severity). The watchdog can leave an old synchronous callback alive while a replacement Service accepts later intent. New failing test drives the actual remote_client::run entrypoint, real local control socket/Registry, and real mesh node_state::Store. After a callback reads version1/Allow and pauses, watchdog replacement acknowledges a local action that durably writes version2/Deny. Releasing the superseded callback commits version1/Allow over it. Exact receipt: “newer_ack_before_old_release=true superseded_write_committed=true persisted_version=1 persisted_access=Allow”; assertion left1/right2. Test: remote_client::supervisor::review_tests::durable_state::acknowledged_network_access_survives_superseded_worker_completion, cargo test -p infraternal-agent --lib --features remote,mesh acknowledged_network_access_survives_superseded_worker_completion -- --nocapture. Source lifetime: remote_client.rs:78–104; offline cancellation remote_supervisor.rs; dedicated worker plugin_control.rs. This proves acknowledged durable-intent loss; kernel packet acceptance is not claimed. Test-only scoped factory/deadline hooks leave shipped behavior unchanged; workers are released and completion drained before assertion. Finding appended to review findings.md. No implementation fix, push or deployment.

Pristine 8003e2134 fast gate completed: all Rust rows PASS except static builds FAIL; separate TypeScript phase passed. Baseline size/link-map build has completed and is being measured. Final exact gate output and remaining VM comparisons will follow.

P1 / HOLD — R1 strengthened and committed as 32e780043 (count R1 once at P1, replacing its earlier P2 severity). The watchdog can leave an old synchronous callback alive while a replacement Service accepts later intent. New failing test drives the actual remote_client::run entrypoint, real local control socket/Registry, and real mesh node_state::Store. After a callback reads version1/Allow and pauses, watchdog replacement acknowledges a local action that durably writes version2/Deny. Releasing the superseded callback commits version1/Allow over it. Exact receipt: “newer_ack_before_old_release=true superseded_write_committed=true persisted_version=1 persisted_access=Allow”; assertion left1/right2. Test: remote_client::supervisor::review_tests::durable_state::acknowledged_network_access_survives_superseded_worker_completion, cargo test -p infraternal-agent --lib --features remote,mesh acknowledged_network_access_survives_superseded_worker_completion -- --nocapture. Source lifetime: remote_client.rs:78–104; offline cancellation remote_supervisor.rs; dedicated worker plugin_control.rs. This proves acknowledged durable-intent loss; kernel packet acceptance is not claimed. Test-only scoped factory/deadline hooks leave shipped behavior unchanged; workers are released and completion drained before assertion. Finding appended to review findings.md. No implementation fix, push or deployment. Pristine 8003e2134 fast gate completed: all Rust rows PASS except static builds FAIL; separate TypeScript phase passed. Baseline size/link-map build has completed and is being measured. Final exact gate output and remaining VM comparisons will follow.
kayg commented 2026-10-15 19:39:51 +00:00
Author
Owner
Copy link

R2 (P2) independently confirmed and committed ef903d256: stripped static x86 candidate22,361,616B versus fa904fdaa22,329,104B, delta32,512B, exceeding the16,384B review allowance by16,128B. Failing Bun integration: scripts/agent-reconnect-size-review.test.ts / “reconnect growth stays within its 16 KiB allowance”; paired real artifacts provided through INFRATERNAL_RECONNECT_SIZE_BASELINE/CANDIDATE. Same pinned release settings and checkout path; baseline linker map is a sidecar only. Baseline absolute debt124,800B remains separate from this change; no ceiling raised. Section/function attribution follows from both linker maps. Size alone does not HOLD under D205; R1 P1 determines HOLD. Receipt incremental-size.log and baseline/candidate-static.txt. No implementation cuts, push or deployment.

R2 (P2) independently confirmed and committed ef903d256: stripped static x86 candidate22,361,616B versus fa904fdaa22,329,104B, delta32,512B, exceeding the16,384B review allowance by16,128B. Failing Bun integration: scripts/agent-reconnect-size-review.test.ts / “reconnect growth stays within its 16 KiB allowance”; paired real artifacts provided through INFRATERNAL_RECONNECT_SIZE_BASELINE/CANDIDATE. Same pinned release settings and checkout path; baseline linker map is a sidecar only. Baseline absolute debt124,800B remains separate from this change; no ceiling raised. Section/function attribution follows from both linker maps. Size alone does not HOLD under D205; R1 P1 determines HOLD. Receipt incremental-size.log and baseline/candidate-static.txt. No implementation cuts, push or deployment.
kayg commented 2026-10-15 19:57:19 +00:00
Author
Owner
Copy link

R2 size attribution completed from both stripped ELF files and linker maps. Head map rebuild is byte-identical to the tested production candidate. All32,512B accounted: .text +23,232; .rodata +4,096; .eh_frame +2,344; .eh_frame_hdr +272; .gcc_except_table +2,128; .data.rel.ro +448; .relr.dyn +8; headers/alignment -16. No dependency/version addition.

Physical text grouped by demangled symbols: CLI remote task/future specializations +10,805B; other application +8,605B (including remote::run +5,063 and unchanged NativeChannel::read code-generation growth +2,490); remote-client/path/supervisor +3,270; other shared async runtime +623; TCP/TLS -446; other language/runtime +377; renewal -222; alignment/unlabelled +220. The new remote task has both current-thread and multi-thread scheduler specializations even though this CLI entry builds a current-thread runtime (e.g. two raw::poll copies +1,400/+1,410B, two harness::complete copies +644B each). This attributes code-generation cost; no implementation cut is made in the reviewer lane.

R1 remains P1/HOLD. Negative control of its durable-state test on fa904fdaa is running with only the test factory/fixtures added; the baseline production runner does not call the new supervisor.

R2 size attribution completed from both stripped ELF files and linker maps. Head map rebuild is byte-identical to the tested production candidate. All32,512B accounted: .text +23,232; .rodata +4,096; .eh_frame +2,344; .eh_frame_hdr +272; .gcc_except_table +2,128; .data.rel.ro +448; .relr.dyn +8; headers/alignment -16. No dependency/version addition. Physical text grouped by demangled symbols: CLI remote task/future specializations +10,805B; other application +8,605B (including remote::run +5,063 and unchanged NativeChannel::read code-generation growth +2,490); remote-client/path/supervisor +3,270; other shared async runtime +623; TCP/TLS -446; other language/runtime +377; renewal -222; alignment/unlabelled +220. The new remote task has both current-thread and multi-thread scheduler specializations even though this CLI entry builds a current-thread runtime (e.g. two raw::poll copies +1,400/+1,410B, two harness::complete copies +644B each). This attributes code-generation cost; no implementation cut is made in the reviewer lane. R1 remains P1/HOLD. Negative control of its durable-state test on fa904fdaa is running with only the test factory/fixtures added; the baseline production runner does not call the new supervisor.
kayg commented 2026-10-15 19:59:16 +00:00
Author
Owner
Copy link

R1 negative control completed: same durable-state test PASS on fa904fdaa, target FAIL. Baseline receipt: newer_ack_before_old_release=false, superseded_write_committed=true, persisted_version=2, persisted_access=Deny. Only test factory/fixtures were added; baseline production runner remains unchanged and does not invoke the new supervisor. This establishes the lifecycle regression rather than a pre-existing Store ordering failure. Actual-entrypoint worker counter also fails with retained3 workers. Review hooks/regressions pass touched-crate pedantic clippy.

Perf baseline300ms RTT also fails same-PTY flood: echo p99363.35ms (baseline idle323.70; growth39.65) versus334ms/16.67ms limits. Candidate same-PTY388.37ms (idle327.91; growth60.46). Different loads and one pair do not establish timing causality; both are red. Baseline full native benchmark timed out in its existing before-forward restart oracle before latency/throughput measurement. Candidate LAN benchmark passed. Cold-recovery VM runs remain to complete. HOLD remains due R1 P1; R2 is P2.

R1 negative control completed: same durable-state test PASS on fa904fdaa, target FAIL. Baseline receipt: newer_ack_before_old_release=false, superseded_write_committed=true, persisted_version=2, persisted_access=Deny. Only test factory/fixtures were added; baseline production runner remains unchanged and does not invoke the new supervisor. This establishes the lifecycle regression rather than a pre-existing Store ordering failure. Actual-entrypoint worker counter also fails with retained3 workers. Review hooks/regressions pass touched-crate pedantic clippy. Perf baseline300ms RTT also fails same-PTY flood: echo p99363.35ms (baseline idle323.70; growth39.65) versus334ms/16.67ms limits. Candidate same-PTY388.37ms (idle327.91; growth60.46). Different loads and one pair do not establish timing causality; both are red. Baseline full native benchmark timed out in its existing before-forward restart oracle before latency/throughput measurement. Candidate LAN benchmark passed. Cold-recovery VM runs remain to complete. HOLD remains due R1 P1; R2 is P2.
kayg commented 2026-10-15 20:15:44 +00:00
Author
Owner
Copy link

Verdict: merge after fixes — HOLD (P0=0, P1=1, P2=1, P3=0). Independent review of 8003e2134d against fa904fdaa8. Local review head ef903d256f. Implementation unchanged; no push/deploy.

R1 P1: watchdog replacement leaves a synchronous plugin worker alive. The actual entrypoint/local-control/real Store fixture acknowledges version 2 / Deny, then the superseded callback restores version 1 / Allow. Target FAIL, identical test with merge-base production runner PASS. Test: remote_client::supervisor::review_tests::durable_state::acknowledged_network_access_survives_superseded_worker_completion (32e780043). Packet-level acceptance is not asserted.

R2 P2: independently built x86 static artifacts grow 22,329,104 -> 22,361,616 B (+32,512; allowance 16,384). Failing paired-artifact test committed ef903d256. Every byte attributed in earlier attachments. Size alone is not the HOLD reason under D205. No implementation cuts made in this review.

Requested production checks:

  • Field cause: original process and shim still have one sleeping thread each. FD enumeration is denied; namespace TCP observations cannot prove process-wide zero sockets. The stalled TCP test is not a forensic explanation of that field state. Content-free privileged wait/FD metadata is needed from owner/orchestrator.
  • Restart behavior: six virtual hours of refusal PASS (728 attempts, no supervisor exit). Both generated units: Restart=always, RestartSec=2, StartLimitIntervalSec=0, KillMode=process. No live systemd restart test.
  • D128: final static shim plus unchanged production dispatcher/shutdown helper PASS, eight real status-1 exits during output, 2,097,152 bytes replayed, lost=0, duplicated=0, same shim PID.
  • Independent perf VM cold recovery: after 120 s bidirectional discard and cold replacement, baseline misses the 31 s admission deadline; candidate PASS, listener-to-terminal 5,037,807 us, replacement CA locked. Full daemon RSS candidate 13,800 -> 13,808 KiB (one process); baseline 15,940 KiB. Single samples do not establish a causal memory delta or at-scale concurrency.
  • Perf VM native candidate: confirmed echo p99 2.49 ms, credited flood 6.243 MiB/s, bounded starved snapshot 1,499 B. Baseline native benchmark fails the before-forward restart oracle before throughput comparison. With 300±30 ms RTT, same-PTY echo p99 candidate 388.37 ms and baseline 363.35 ms both FAIL 334 ms / 16.67 ms growth limits. Different recorded loads and jitter prevent branch-specific attribution from this pair. Raw logs preserve machine/load; budgets not relaxed.

Pristine target scripts/verify fast exit 1; summary verbatim:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

All other Rust rows PASS; 1,090 native/compile-out tests passed. The fast script stops before TS after static failure; separately running its TS phase on the pristine head PASS. Review test hooks and tests pass touched-crate pedantic clippy and cargo fmt. Own perf VM directory/scopes, temporary worktree, build targets, fixture artifacts and dedicated compiler-cache server cleaned up; raw evidence retained outside checkout.

perf-recovery-baseline.log
perf-recovery-candidate.log
verify-fast-head.log
verify-ts-head.log
link-exit-replay-static.log

Verdict: merge after fixes — HOLD (P0=0, P1=1, P2=1, P3=0). Independent review of 8003e2134d188182b1c60ab18833a65c610b4ced against fa904fdaa8d6e28c0ffeee8550ff1d462c169817. Local review head ef903d256f3fec420fcdd2a2a9ba550be1afb3d2. Implementation unchanged; no push/deploy. R1 P1: watchdog replacement leaves a synchronous plugin worker alive. The actual entrypoint/local-control/real Store fixture acknowledges version 2 / Deny, then the superseded callback restores version 1 / Allow. Target FAIL, identical test with merge-base production runner PASS. Test: remote_client::supervisor::review_tests::durable_state::acknowledged_network_access_survives_superseded_worker_completion (32e780043). Packet-level acceptance is not asserted. R2 P2: independently built x86 static artifacts grow 22,329,104 -> 22,361,616 B (+32,512; allowance 16,384). Failing paired-artifact test committed ef903d256. Every byte attributed in earlier attachments. Size alone is not the HOLD reason under D205. No implementation cuts made in this review. Requested production checks: - Field cause: original process and shim still have one sleeping thread each. FD enumeration is denied; namespace TCP observations cannot prove process-wide zero sockets. The stalled TCP test is not a forensic explanation of that field state. Content-free privileged wait/FD metadata is needed from owner/orchestrator. - Restart behavior: six virtual hours of refusal PASS (728 attempts, no supervisor exit). Both generated units: Restart=always, RestartSec=2, StartLimitIntervalSec=0, KillMode=process. No live systemd restart test. - D128: final static shim plus unchanged production dispatcher/shutdown helper PASS, eight real status-1 exits during output, 2,097,152 bytes replayed, lost=0, duplicated=0, same shim PID. - Independent perf VM cold recovery: after 120 s bidirectional discard and cold replacement, baseline misses the 31 s admission deadline; candidate PASS, listener-to-terminal 5,037,807 us, replacement CA locked. Full daemon RSS candidate 13,800 -> 13,808 KiB (one process); baseline 15,940 KiB. Single samples do not establish a causal memory delta or at-scale concurrency. - Perf VM native candidate: confirmed echo p99 2.49 ms, credited flood 6.243 MiB/s, bounded starved snapshot 1,499 B. Baseline native benchmark fails the before-forward restart oracle before throughput comparison. With 300±30 ms RTT, same-PTY echo p99 candidate 388.37 ms and baseline 363.35 ms both FAIL 334 ms / 16.67 ms growth limits. Different recorded loads and jitter prevent branch-specific attribution from this pair. Raw logs preserve machine/load; budgets not relaxed. Pristine target scripts/verify fast exit 1; summary verbatim: ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` All other Rust rows PASS; 1,090 native/compile-out tests passed. The fast script stops before TS after static failure; separately running its TS phase on the pristine head PASS. Review test hooks and tests pass touched-crate pedantic clippy and cargo fmt. Own perf VM directory/scopes, temporary worktree, build targets, fixture artifacts and dedicated compiler-cache server cleaned up; raw evidence retained outside checkout. [perf-recovery-baseline.log](https://git.kayg.org/attachments/04f79845-e757-4e41-abce-9d45940cf47b) [perf-recovery-candidate.log](https://git.kayg.org/attachments/0e204c43-fc7f-4d56-a3de-8991485d5b6d) [verify-fast-head.log](https://git.kayg.org/attachments/0b81701a-ace8-4352-8ea4-ac2a941b4db6) [verify-ts-head.log](https://git.kayg.org/attachments/2fb0b31d-86ec-42fa-aed1-75960847c4b0) [link-exit-replay-static.log](https://git.kayg.org/attachments/2f2842ba-6865-4d9d-9aed-ac124df2178d)
kayg commented 2026-10-15 20:16:47 +00:00
Author
Owner
Copy link

Resuming #225 on agent-reconnect at 8003e2134d; merge base fa904fdaa. Read all 31 issue comments and review findings through ef903d256; no parent linked. Addressing superseded-worker acknowledged intent rollback with failing review coverage first, bounded worker lifecycle and monotonic application, then attributed static-size reduction and final build-slot fast gate. Field cause remains unverified. No real host changes or deployment; pushes only to agent-reconnect.

Resuming #225 on agent-reconnect at 8003e2134d188182b1c60ab18833a65c610b4ced; merge base fa904fdaa. Read all 31 issue comments and review findings through ef903d256; no parent linked. Addressing superseded-worker acknowledged intent rollback with failing review coverage first, bounded worker lifecycle and monotonic application, then attributed static-size reduction and final build-slot fast gate. Field cause remains unverified. No real host changes or deployment; pushes only to agent-reconnect.
kayg commented 2026-10-15 20:19:33 +00:00
Author
Owner
Copy link

R1 repair choice: retain one local Service/Registry across transport attempts and offline-watchdog resets. Replacement is allowed only after its mpsc receiver closes, which proves the dedicated worker has ended; a live synchronous worker is never superseded. This bounds ownership to one worker and makes the old-worker fence structural, including kernel operations that cannot be cancelled safely. The existing Node staging owner then retains its acknowledged revision across link restarts.
Defence in depth: serialize durable snapshot check+replace with the existing nix directory flock and refuse older or conflicting same-version configs; allow identical replay and the existing exact legacy-key metadata upgrade. No schema/dependency change. Endpoint-install failure must retain the newer durable desired endpoints for retry rather than writing an older revision back; kernel endpoint rollback still runs and no success is acknowledged. Add exhaustive start/stop/apply schedule coverage plus stale/conflicting Store regressions before that persistence change. No frozen wire change or new foundational engine.

R1 repair choice: retain one local Service/Registry across transport attempts and offline-watchdog resets. Replacement is allowed only after its mpsc receiver closes, which proves the dedicated worker has ended; a live synchronous worker is never superseded. This bounds ownership to one worker and makes the old-worker fence structural, including kernel operations that cannot be cancelled safely. The existing Node staging owner then retains its acknowledged revision across link restarts. Defence in depth: serialize durable snapshot check+replace with the existing nix directory flock and refuse older or conflicting same-version configs; allow identical replay and the existing exact legacy-key metadata upgrade. No schema/dependency change. Endpoint-install failure must retain the newer durable desired endpoints for retry rather than writing an older revision back; kernel endpoint rollback still runs and no success is acknowledged. Add exhaustive start/stop/apply schedule coverage plus stale/conflicting Store regressions before that persistence change. No frozen wire change or new foundational engine.
kayg commented 2026-10-15 20:23:31 +00:00
Author
Owner
Copy link

Cherry-picked/pushed review regressions as e9e2e516d, 741008de9, f3cc92b6b and 6980e2352. Re-ran R1 before production edits: FAIL (0 passed, 1 failed), newer_ack_before_old_release=true superseded_write_committed=true persisted_version=1 persisted_access=Allow; assertion left1/right2. Root introduced at remote_client.rs:78–104: a Service inside each watchdog-cancellable attempt creates overlapping synchronous owners. Implementing the single-live-owner fix plus durable stale/equal-conflict checks. Field cause remains distinct/unverified.

review-red.log

Cherry-picked/pushed review regressions as e9e2e516d, 741008de9, f3cc92b6b and 6980e2352. Re-ran R1 before production edits: FAIL (0 passed, 1 failed), newer_ack_before_old_release=true superseded_write_committed=true persisted_version=1 persisted_access=Allow; assertion left1/right2. Root introduced at remote_client.rs:78–104: a Service inside each watchdog-cancellable attempt creates overlapping synchronous owners. Implementing the single-live-owner fix plus durable stale/equal-conflict checks. Field cause remains distinct/unverified. [review-red.log](https://git.kayg.org/attachments/c3209bb9-c0f1-411b-9338-3fc474352b3f)
kayg commented 2026-10-15 20:25:19 +00:00
Author
Owner
Copy link

Pushed b655a19d7: failing durable-order regressions before the persistence repair. Actual Store checks: 0 passed, 3 failed, 1 ignored. Covers stale version1 after acknowledged version2, conflicting equal-version content, every one of 729 six-operation start/stop/apply schedules, and concurrent independent writers (current same-PID temporary-file collision also reproduced). Ignored 1000-peer persistence benchmark added for the repair's cost; it has not run on the build host.

store-red.log

Pushed b655a19d7: failing durable-order regressions before the persistence repair. Actual Store checks: 0 passed, 3 failed, 1 ignored. Covers stale version1 after acknowledged version2, conflicting equal-version content, every one of 729 six-operation start/stop/apply schedules, and concurrent independent writers (current same-PID temporary-file collision also reproduced). Ignored 1000-peer persistence benchmark added for the repair's cost; it has not run on the build host. [store-red.log](https://git.kayg.org/attachments/9f24b264-f372-4790-ad31-875d54a3d83d)
kayg commented 2026-10-15 20:29:43 +00:00
Author
Owner
Copy link

Pushed 1aca17567: Service/Registry now outlives all link attempts/watchdog resets. A live worker is never replaced; only closed incoming receiver permits a new owner, fencing old handles and callbacks. Original durable regression passes WITHOUT the Store repair: newer_ack_before_old_release=false superseded_write_committed=true persisted_version=2 persisted_access=Deny. Worker-count regression passes (bound 1 live worker per process; at 1000 distributed nodes, <=1000 such workers). New terminated-factory test proves failed local startup is still recreated. Six-hour virtual outage: 728 attempts, service_returned=false. No runtime timing claim.
Gates: agent library 106 passed/0 failed/14 ignored before the new factory case; exact supervisor run 8 passed/0 failed including it; touched-agent pedantic clippy and cargo fmt pass. Separate Store stale/equal/conflicting/concurrent schedule repair is in progress.

lifecycle-green.log
lifecycle-exact.log

Pushed 1aca17567: Service/Registry now outlives all link attempts/watchdog resets. A live worker is never replaced; only closed incoming receiver permits a new owner, fencing old handles and callbacks. Original durable regression passes WITHOUT the Store repair: newer_ack_before_old_release=false superseded_write_committed=true persisted_version=2 persisted_access=Deny. Worker-count regression passes (bound 1 live worker per process; at 1000 distributed nodes, <=1000 such workers). New terminated-factory test proves failed local startup is still recreated. Six-hour virtual outage: 728 attempts, service_returned=false. No runtime timing claim. Gates: agent library 106 passed/0 failed/14 ignored before the new factory case; exact supervisor run 8 passed/0 failed including it; touched-agent pedantic clippy and cargo fmt pass. Separate Store stale/equal/conflicting/concurrent schedule repair is in progress. [lifecycle-green.log](https://git.kayg.org/attachments/22ac5f5f-f31d-4c72-a73c-4bf2a81fc2bf) [lifecycle-exact.log](https://git.kayg.org/attachments/292127d3-aaea-4310-99a0-0e731d2677ed)
kayg commented 2026-10-15 20:32:43 +00:00
Author
Owner
Copy link

Pushed 1d3b4d57f: durable public snapshot save serializes check+replace with the existing directory flock and refuses older/conflicting equal revisions. Identical replay and the exact existing legacy-key-only upgrade remain valid. No schema, dependency, feature or retained per-peer cache added. The endpoint-only failure path keeps newer desired intent durable after kernel rollback, acknowledges no success and permits retry; it no longer writes an older revision back. Existing no-follow/ownership/mode checks retained; reader buffered for the new per-update check.
Tests before fix: 3 failed. After: mesh agent library 29 passed/0 failed/12 ignored; includes all 729 six-operation schedules over two alternating worker slots, 32 concurrent-writer rounds and binding compatibility. fmt and touched-crate pedantic clippy pass. Performance-VM 1000-peer persistence cost and final static-size/gate work remain.
Pushed 62d265801 documents field-cause evidence. Exact field wait remains UNVERIFIED. After owner-led restart with the fixed binary, correlate transport TimedOut / five-minute offline watchdog / unexpected link completion and nonzero service restart with server admission. A watchdog line alone does not diagnose an established idle TCP wait. If no line appears, content-free privileged wait/thread/socket ownership metadata before restart is needed from the owner/orchestrator. docs/verification/agent-reconnect-recovery.md retains exact rootful/rootless commands. No field host touched.

store-green.log

Pushed 1d3b4d57f: durable public snapshot save serializes check+replace with the existing directory flock and refuses older/conflicting equal revisions. Identical replay and the exact existing legacy-key-only upgrade remain valid. No schema, dependency, feature or retained per-peer cache added. The endpoint-only failure path keeps newer desired intent durable after kernel rollback, acknowledges no success and permits retry; it no longer writes an older revision back. Existing no-follow/ownership/mode checks retained; reader buffered for the new per-update check. Tests before fix: 3 failed. After: mesh agent library 29 passed/0 failed/12 ignored; includes all 729 six-operation schedules over two alternating worker slots, 32 concurrent-writer rounds and binding compatibility. fmt and touched-crate pedantic clippy pass. Performance-VM 1000-peer persistence cost and final static-size/gate work remain. Pushed 62d265801 documents field-cause evidence. Exact field wait remains UNVERIFIED. After owner-led restart with the fixed binary, correlate transport TimedOut / five-minute offline watchdog / unexpected link completion and nonzero service restart with server admission. A watchdog line alone does not diagnose an established idle TCP wait. If no line appears, content-free privileged wait/thread/socket ownership metadata before restart is needed from the owner/orchestrator. docs/verification/agent-reconnect-recovery.md retains exact rootful/rootless commands. No field host touched. [store-green.log](https://git.kayg.org/attachments/25794b0a-439e-4e6d-a13c-80d7c4b55963)
kayg commented 2026-10-15 20:41:45 +00:00
Author
Owner
Copy link

Pushed a24762389: eliminate the extra CLI link task; catch polling panics at the owning runtime boundary and always consume the runtime through the unchanged bounded-shutdown helper. This targets the review's +10,805B task/future specialization group (including both scheduler harnesses); exact net static bytes are being rebuilt, not estimated. No global panic-profile, runtime-engine, dependency or wire change.
Gates: CLI all-target pedantic clippy/fmt PASS; all 3 success/error/panic service tests PASS, including background work held until after bounded shutdown. Pushed daf62bde2 documents the receiver-drop replacement fence and corrects the initializer comment; agent clippy/fmt and focused worker retry-owner test PASS (1/0).
Building stripped static x86 candidate and paired Store test with the repository release recipe. Linker-map sidecars only are enabled temporarily for attribution; the linker script change will be restored and never committed. Baseline fa904fdaa will be rebuilt with the same recipe. No ceilings changed.

cli-green.log
fence-comments.log

Pushed a24762389: eliminate the extra CLI link task; catch polling panics at the owning runtime boundary and always consume the runtime through the unchanged bounded-shutdown helper. This targets the review's +10,805B task/future specialization group (including both scheduler harnesses); exact net static bytes are being rebuilt, not estimated. No global panic-profile, runtime-engine, dependency or wire change. Gates: CLI all-target pedantic clippy/fmt PASS; all 3 success/error/panic service tests PASS, including background work held until after bounded shutdown. Pushed daf62bde2 documents the receiver-drop replacement fence and corrects the initializer comment; agent clippy/fmt and focused worker retry-owner test PASS (1/0). Building stripped static x86 candidate and paired Store test with the repository release recipe. Linker-map sidecars only are enabled temporarily for attribution; the linker script change will be restored and never committed. Baseline fa904fdaa will be rebuilt with the same recipe. No ceilings changed. [cli-green.log](https://git.kayg.org/attachments/2ccf9c15-f383-45d9-b5e7-314cb37ae482) [fence-comments.log](https://git.kayg.org/attachments/1d8fb5b8-11c2-45e4-ad84-6120e5d13324)
kayg commented 2026-10-15 20:58:03 +00:00
Author
Owner
Copy link

First new static x86 result (daf62bde2): 22,347,184B, SHA256 be0fff5c1a05b08dc6468a3dd302ec8c888762918a8b46faa09007624d3030ff. This cuts 14,432B from reviewed 8003e2134 (22,361,616B), but versus independently measured fa904fdaa 22,329,104B it is still +18,080B: 1,696B over the 16,384B allowance. Not accepted as the final size.
Linker-map inspection of candidate.map attributes 48,769 physical .text bytes to distinct IoRead<BufReader> JSON symbols (deduplicated address/size ranges). Largest: BTreeMap<NodeId, PeerPolicy> decode 6,587B; Option decode 5,617B; Addresses decode 3,913B. This public reader is the repository's only serde_json::from_reader call; the protocol already uses byte-slice decoding of the same peer/access types.
Next cut: read the already-opened, ownership/mode/no-follow-checked public file into bytes, then use the existing slice decoder. No new parser/library or validation bypass. It trades one transient snapshot-sized byte buffer for shared decoder code and faster parsing; measure 1000-peer CPU/time and full test-process peak RSS on the perf VM before accepting it. First buffered Store test artifact is compiling now so its cost can also be retained as a control. Full paired baseline rebuild and final gate still pending; no ceilings changed.

First new static x86 result (daf62bde2): 22,347,184B, SHA256 be0fff5c1a05b08dc6468a3dd302ec8c888762918a8b46faa09007624d3030ff. This cuts 14,432B from reviewed 8003e2134 (22,361,616B), but versus independently measured fa904fdaa 22,329,104B it is still +18,080B: 1,696B over the 16,384B allowance. Not accepted as the final size. Linker-map inspection of candidate.map attributes 48,769 physical .text bytes to distinct IoRead<BufReader<File>> JSON symbols (deduplicated address/size ranges). Largest: BTreeMap<NodeId, PeerPolicy> decode 6,587B; Option<Config> decode 5,617B; Addresses decode 3,913B. This public reader is the repository's only serde_json::from_reader call; the protocol already uses byte-slice decoding of the same peer/access types. Next cut: read the already-opened, ownership/mode/no-follow-checked public file into bytes, then use the existing slice decoder. No new parser/library or validation bypass. It trades one transient snapshot-sized byte buffer for shared decoder code and faster parsing; measure 1000-peer CPU/time and full test-process peak RSS on the perf VM before accepting it. First buffered Store test artifact is compiling now so its cost can also be retained as a control. Full paired baseline rebuild and final gate still pending; no ceilings changed.
kayg commented 2026-10-15 21:07:38 +00:00
Author
Owner
Copy link

Pushed d918d360b: public snapshots now read through the already-validated descriptor into a transient byte buffer and reuse the protocol's slice JSON decoder. This targets the measured 48,769B reader-specialization group; final net bytes remain pending. Ownership/mode/no-follow, revision checks and legacy binding semantics are unchanged. Gates: 29 passed/0 failed/12 ignored; pedantic clippy and fmt PASS.
Initial buffered-guard VM timing is retained only as a diagnostic until the paired run. Its whole-command RSS included the scope launcher, so that reading is NOT accepted as test-process memory. Instrumentation now runs inside the scope and collects only the direct test child's CPU/RSS. Baseline, buffered guard and final slice guard will run together under one short lock after artifacts are ready. No performance acceptance claim from build-host timing, and no budget changed.

slice-green.log

Pushed d918d360b: public snapshots now read through the already-validated descriptor into a transient byte buffer and reuse the protocol's slice JSON decoder. This targets the measured 48,769B reader-specialization group; final net bytes remain pending. Ownership/mode/no-follow, revision checks and legacy binding semantics are unchanged. Gates: 29 passed/0 failed/12 ignored; pedantic clippy and fmt PASS. Initial buffered-guard VM timing is retained only as a diagnostic until the paired run. Its whole-command RSS included the scope launcher, so that reading is NOT accepted as test-process memory. Instrumentation now runs inside the scope and collects only the direct test child's CPU/RSS. Baseline, buffered guard and final slice guard will run together under one short lock after artifacts are ready. No performance acceptance claim from build-host timing, and no budget changed. [slice-green.log](https://git.kayg.org/attachments/41a7ddd8-9419-4c57-acc9-b280c42c3a11)
kayg commented 2026-10-15 21:14:58 +00:00
Author
Owner
Copy link

Author-review finding and fix: d918d360b's Store::save unconditionally decoded the old file even for disabled/unconfigured intent. A running component could no longer persist explicit Off after public JSON damage. Committed/pushed failing real-file regression 8f87862c9 (0 passed, 1 failed), then repair 8cd49d505: only access-bearing configs require old-revision decoding; directory locking and all active monotonic checks remain. No schema/authority change. State tests now 9 passed/0 failed/1 measurement ignored; pedantic clippy/fmt PASS.
Baseline fa904fdaa x86 rebuilt independently: 22,329,104B, exactly matching both earlier baseline counts, SHA256 f93d487ba79fe83ab5ffeee65e8d509977eb017598172778a3efe90a85b5f025. The actual paired-artifact Bun test against the buffered first-cut binary fails as expected: +18,080B >16,384B (0 pass/1 fail). Slice-decoder head bytes, ARM baseline, corrected three-way VM measurement and final full gate pending.

off-red.log
off-green.log
size-buffered-red.log

Author-review finding and fix: d918d360b's Store::save unconditionally decoded the old file even for disabled/unconfigured intent. A running component could no longer persist explicit Off after public JSON damage. Committed/pushed failing real-file regression 8f87862c9 (0 passed, 1 failed), then repair 8cd49d505: only access-bearing configs require old-revision decoding; directory locking and all active monotonic checks remain. No schema/authority change. State tests now 9 passed/0 failed/1 measurement ignored; pedantic clippy/fmt PASS. Baseline fa904fdaa x86 rebuilt independently: 22,329,104B, exactly matching both earlier baseline counts, SHA256 f93d487ba79fe83ab5ffeee65e8d509977eb017598172778a3efe90a85b5f025. The actual paired-artifact Bun test against the buffered first-cut binary fails as expected: +18,080B >16,384B (0 pass/1 fail). Slice-decoder head bytes, ARM baseline, corrected three-way VM measurement and final full gate pending. [off-red.log](https://git.kayg.org/attachments/fdf3c619-f0de-4dd2-9139-bedb52ff7878) [off-green.log](https://git.kayg.org/attachments/6812fb3c-699c-4655-9b59-02f370a29cd1) [size-buffered-red.log](https://git.kayg.org/attachments/4e5be575-87a1-47ba-93cb-d47aa03b648c)
kayg commented 2026-10-15 21:30:32 +00:00
Author
Owner
Copy link

P2 incremental size resolved at pushed 8cd49d505. Stripped static x86 candidate 22,299,504B, SHA256 1ee932e29556160fd665a035df5878ceb24d57f19e0d270f347e4cebb7069a7e. Independently rebuilt fa904fdaa baseline 22,329,104B: branch delta -29,600B. Versus reviewed 8003e2134 (22,361,616B), cut 62,112B. Real paired-artifact Bun regression now PASS (1 pass/0 fail), after the recorded buffered first-cut FAIL (+18,080B).
All -29,600 file bytes accounted: .text -31,552; .eh_frame -2,680; .eh_frame_hdr -224; .gcc_except_table +588; .rodata +4,096; .data.rel.ro +224; .relr.dyn +8; headers/alignment -60. Function/alias attribution counts overlapping physical intervals once: old JSON I/O reader 50,924 -> 0B; shared slice reader +9,615B; CLI link boundary/future -8,459B. Remaining groups and largest changed intervals are in the receipt; compiler inlining movement is reported separately from source changes.
Absolute ceiling debt is separate and unchanged: baseline x86 +124,800B, candidate +95,200B against 22,204,304B. ARM baseline now independently measured 19,752,920B, +114,176B against 19,638,744B; ARM head will be measured by the final full gate. No budget/profile change. Final VM CPU/memory cost and branch-head fast gate remain pending.

size-final-green.log
size-baseline-to-candidate.txt
baseline-static-build.log

P2 incremental size resolved at pushed 8cd49d505. Stripped static x86 candidate 22,299,504B, SHA256 1ee932e29556160fd665a035df5878ceb24d57f19e0d270f347e4cebb7069a7e. Independently rebuilt fa904fdaa baseline 22,329,104B: branch delta -29,600B. Versus reviewed 8003e2134 (22,361,616B), cut 62,112B. Real paired-artifact Bun regression now PASS (1 pass/0 fail), after the recorded buffered first-cut FAIL (+18,080B). All -29,600 file bytes accounted: .text -31,552; .eh_frame -2,680; .eh_frame_hdr -224; .gcc_except_table +588; .rodata +4,096; .data.rel.ro +224; .relr.dyn +8; headers/alignment -60. Function/alias attribution counts overlapping physical intervals once: old JSON I/O reader 50,924 -> 0B; shared slice reader +9,615B; CLI link boundary/future -8,459B. Remaining groups and largest changed intervals are in the receipt; compiler inlining movement is reported separately from source changes. Absolute ceiling debt is separate and unchanged: baseline x86 +124,800B, candidate +95,200B against 22,204,304B. ARM baseline now independently measured 19,752,920B, +114,176B against 19,638,744B; ARM head will be measured by the final full gate. No budget/profile change. Final VM CPU/memory cost and branch-head fast gate remain pending. [size-final-green.log](https://git.kayg.org/attachments/f2891c88-0b25-45c1-95b3-f7720bf504d7) [size-baseline-to-candidate.txt](https://git.kayg.org/attachments/b104d467-7abb-463d-bd48-b036470fde24) [baseline-static-build.log](https://git.kayg.org/attachments/03646cb0-b0e3-4b22-9fcb-fe03221f78e1)
kayg commented 2026-10-15 21:44:58 +00:00
Author
Owner
Copy link

Corrected perf-VM measurements complete: infraternal-perf-test, 4 cores, two sequential baseline/buffered/final batches; 101 durable saves per artifact per batch, actual 1000-peer snapshot 206,907B. Each batch held the shared lock only during measurement, each child ran in CPUWeight=10000 scope. Latest batch load 0.15/0.47/0.60 ->0.35/0.50/0.61.

Artifact p50 ms, two batches p99 ms, two batches Observed post-exec VmHWM KiB
fa904fdaa baseline 20.648 /21.322 32.720 /34.314 5368
buffered guard 27.999 /23.684 82.038 /84.238 5508
8cd49d505 final slice guard 21.934 /22.062 30.969 /60.096 5640

Latest whole-test child CPU (user+system): baseline0.412062s, buffered0.709761s, final0.606284s. Across both batches the added total child CPU amortizes to ~1.92–2.19ms/update at1000 peers. This pays for checking durable revision before acknowledging a write; it runs on the existing plugin worker, not terminal dispatch. Medians add0.740–1.286ms versus baseline. Filesystem tails vary; no tail-latency budget pass or causal p99 improvement is claimed.
Post-exec /proc memory counters exclude launcher's inherited pre-exec high-water value: final observed test-process VmHWM +272KiB vs baseline at1000 peers; ×1000 distributed instances is272,000KiB (~265.6MiB) arithmetic only. This is neither a concurrent-1000-daemon run nor bare-idle daemon RSS. The public input byte buffer and decoded old state are transient; no new retained per-peer cache/idle descriptor/timer.
Final local gate first reached a fixture setup failure: my49-byte state-root override generated a110-byte AF_UNIX rendezvous. No code failure or limit relaxation. A shorter job-owned alias points into the same scratch directory. Focused real-binary proof with it now PASS:24 abrupt stops,4,710,396 bytes, exact input/output replay at all4 stop positions×6. Will rerun the full head gate after the in-progress static builds complete; the current attempt is interim.

perf-store-final.log
perf-store-vmhwm.log
measure-store.txt
short-path-check.log

Corrected perf-VM measurements complete: infraternal-perf-test, 4 cores, two sequential baseline/buffered/final batches; 101 durable saves per artifact per batch, actual 1000-peer snapshot 206,907B. Each batch held the shared lock only during measurement, each child ran in CPUWeight=10000 scope. Latest batch load 0.15/0.47/0.60 ->0.35/0.50/0.61. | Artifact | p50 ms, two batches | p99 ms, two batches | Observed post-exec VmHWM KiB | | --- | --- | --- | --- | | fa904fdaa baseline |20.648 /21.322 |32.720 /34.314 |5368 | | buffered guard |27.999 /23.684 |82.038 /84.238 |5508 | | 8cd49d505 final slice guard |21.934 /22.062 |30.969 /60.096 |5640 | Latest whole-test child CPU (user+system): baseline0.412062s, buffered0.709761s, final0.606284s. Across both batches the added total child CPU amortizes to ~1.92–2.19ms/update at1000 peers. This pays for checking durable revision before acknowledging a write; it runs on the existing plugin worker, not terminal dispatch. Medians add0.740–1.286ms versus baseline. Filesystem tails vary; no tail-latency budget pass or causal p99 improvement is claimed. Post-exec /proc memory counters exclude launcher's inherited pre-exec high-water value: final observed test-process VmHWM +272KiB vs baseline at1000 peers; ×1000 distributed instances is272,000KiB (~265.6MiB) arithmetic only. This is neither a concurrent-1000-daemon run nor bare-idle daemon RSS. The public input byte buffer and decoded old state are transient; no new retained per-peer cache/idle descriptor/timer. Final local gate first reached a fixture setup failure: my49-byte state-root override generated a110-byte AF_UNIX rendezvous. No code failure or limit relaxation. A shorter job-owned alias points into the same scratch directory. Focused real-binary proof with it now PASS:24 abrupt stops,4,710,396 bytes, exact input/output replay at all4 stop positions×6. Will rerun the full head gate after the in-progress static builds complete; the current attempt is interim. [perf-store-final.log](https://git.kayg.org/attachments/04e5d271-baed-410f-a861-69f8b0c6cf94) [perf-store-vmhwm.log](https://git.kayg.org/attachments/7c8c297e-fdd8-4a99-81f9-87da9fad2aad) [measure-store.txt](https://git.kayg.org/attachments/6992cc41-2076-4a65-be75-11e172b89209) [short-path-check.log](https://git.kayg.org/attachments/27e35fe8-8f11-4023-823a-ca31abe455ee)
kayg commented 2026-10-15 22:09:24 +00:00
Author
Owner
Copy link

Repair complete and pushed on agent-reconnect, head 8cd49d5057; base fa904fdaa8. Worktree clean; no deployment or real-host action. Issue remains open.

P1 resolved: one local Service/Registry survives transport/watchdog resets; a replacement starts only after the old receiver/registry is gone. Original review regression passes independently of the Store guard (newer_ack_before_old_release=false, final version2/Deny). Real worker bound and six-hour retry owner pass; durable ordering is additionally checked atomically across Store writers. All729 six-operation schedules,32 concurrent-writer rounds, legacy binding and explicit-off corruption recovery pass. Root cause introduced at reviewed remote_client.rs:78–104 was constructing a new Service inside each cancellable attempt; old synchronous callbacks could outlive it.

P2 resolved: x86 22,299,504B versus identical-recipe fa baseline22,329,104B = -29,600B; ARM19,726,904B versus baseline19,752,920B = -26,016B. The paired incremental regression passes; x86 is62,112B smaller than reviewed8003e2134. Attribution and VM1000-peer cost/limits are in prior comments. No size ceiling or runtime budget changed.

Final build-slot scripts/verify fast exit1:1100 Rust tests passed,0 failed,141 ignored. Every Rust row except existing absolute static-size ceilings PASS. Baselines exceed those ceilings too: x86 +124,800B and ARM +114,176B; head reduces remaining debt to95,200B/88,160B. Linkage, production trust and x86 locked/dump-excluded key-heap checks pass; ARM key-heap runtime is explicitly unmeasured on this x86 host. TypeScript phase skipped by the Rust return was run separately through build-slot: compiler/check, production build/transfer budgets,191 tracer tests and37 protocol tests PASS.

Final fast summary, verbatim:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Exact original field wait remains UNVERIFIED. docs/verification/agent-reconnect-recovery.md explains what post-restart timeout/watchdog/exit logs plus server admission would confirm, and when owner/orchestrator privileged content-free wait/socket metadata is still needed. One-time old-host recovery over an existing administration channel: rootful sudo systemctl restart infraternal-agent.service; rootless as installation account systemctl --user restart infraternal-agent.service. Do not re-enroll or remove identity/shims. No real host was touched.

All code commits pushed atomically. Own temporary baseline checkout, build target, artifacts and runtime fixtures deleted; raw receipts retained outside checkout. VM job executables0 and job directory removed; no shared compiler-cache service stopped. Setup-only intermediate failures (long socket roots, then symlink canonical-path mismatch, then long TMPDIR) are retained; the accepted full run uses a real34-character private runtime directory with no production limit relaxed. Author checklist completed.

verify-fast-final.log
verify-ts-final.log
final-artifacts.json
author-review-repair.txt
perf-cleanup.log
local-cleanup-final.txt
verify-fast-interim-alias.log
verify-fast-interim-tmpdir.log

Repair complete and pushed on agent-reconnect, head 8cd49d5057ca4df9b839402632892c0b286d78f2; base fa904fdaa8d6e28c0ffeee8550ff1d462c169817. Worktree clean; no deployment or real-host action. Issue remains open. P1 resolved: one local Service/Registry survives transport/watchdog resets; a replacement starts only after the old receiver/registry is gone. Original review regression passes independently of the Store guard (newer_ack_before_old_release=false, final version2/Deny). Real worker bound and six-hour retry owner pass; durable ordering is additionally checked atomically across Store writers. All729 six-operation schedules,32 concurrent-writer rounds, legacy binding and explicit-off corruption recovery pass. Root cause introduced at reviewed remote_client.rs:78–104 was constructing a new Service inside each cancellable attempt; old synchronous callbacks could outlive it. P2 resolved: x86 22,299,504B versus identical-recipe fa baseline22,329,104B = -29,600B; ARM19,726,904B versus baseline19,752,920B = -26,016B. The paired incremental regression passes; x86 is62,112B smaller than reviewed8003e2134. Attribution and VM1000-peer cost/limits are in prior comments. No size ceiling or runtime budget changed. Final build-slot scripts/verify fast exit1:1100 Rust tests passed,0 failed,141 ignored. Every Rust row except existing absolute static-size ceilings PASS. Baselines exceed those ceilings too: x86 +124,800B and ARM +114,176B; head reduces remaining debt to95,200B/88,160B. Linkage, production trust and x86 locked/dump-excluded key-heap checks pass; ARM key-heap runtime is explicitly unmeasured on this x86 host. TypeScript phase skipped by the Rust return was run separately through build-slot: compiler/check, production build/transfer budgets,191 tracer tests and37 protocol tests PASS. Final fast summary, verbatim: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` Exact original field wait remains UNVERIFIED. docs/verification/agent-reconnect-recovery.md explains what post-restart timeout/watchdog/exit logs plus server admission would confirm, and when owner/orchestrator privileged content-free wait/socket metadata is still needed. One-time old-host recovery over an existing administration channel: rootful `sudo systemctl restart infraternal-agent.service`; rootless as installation account `systemctl --user restart infraternal-agent.service`. Do not re-enroll or remove identity/shims. No real host was touched. All code commits pushed atomically. Own temporary baseline checkout, build target, artifacts and runtime fixtures deleted; raw receipts retained outside checkout. VM job executables0 and job directory removed; no shared compiler-cache service stopped. Setup-only intermediate failures (long socket roots, then symlink canonical-path mismatch, then long TMPDIR) are retained; the accepted full run uses a real34-character private runtime directory with no production limit relaxed. Author checklist completed. [verify-fast-final.log](https://git.kayg.org/attachments/7d372c92-fdad-4374-b52a-b869e2b4b605) [verify-ts-final.log](https://git.kayg.org/attachments/9ae052ed-36c2-4f16-b29a-d2e3e4e3e65c) [final-artifacts.json](https://git.kayg.org/attachments/1376e979-f297-47cf-b365-b46c41eedab0) [author-review-repair.txt](https://git.kayg.org/attachments/3158ebf9-fdb2-4f80-b279-d2b51cf493b4) [perf-cleanup.log](https://git.kayg.org/attachments/ede1535f-98e9-468c-8b96-9c65e7860389) [local-cleanup-final.txt](https://git.kayg.org/attachments/997be3f0-fc2f-4649-9e98-2e05c5f83d9f) [verify-fast-interim-alias.log](https://git.kayg.org/attachments/b70426e6-de78-4ebb-8885-13129762197e) [verify-fast-interim-tmpdir.log](https://git.kayg.org/attachments/6ffeff14-bc9e-4137-b245-d37496786b43)
kayg commented 2026-10-15 22:11:35 +00:00
Author
Owner
Copy link

Delta verification started: review-agent-reconnect at 8cd49d505; previous review 8003e2134; merge base with dev 8e7467c07. Read #225 body and all 44 comments; no parent linked. Earlier review history preserved at review-agent-reconnect-previous (ef903d256). Review only lane delta and prior regressions. No implementation edits, push, deployment or owner fixture changes.

Delta verification started: review-agent-reconnect at 8cd49d505; previous review 8003e2134; merge base with dev 8e7467c07. Read #225 body and all 44 comments; no parent linked. Earlier review history preserved at review-agent-reconnect-previous (ef903d256). Review only lane delta and prior regressions. No implementation edits, push, deployment or owner fixture changes.
kayg commented 2026-10-15 22:14:08 +00:00
Author
Owner
Copy link

Delta review source inspection complete; no new confirmed defect yet. Full pristine-head fast gate is queued through build-slot; all four shared slots are occupied. Focused single-crate durable-state tests are compiling separately. Performance VM is reachable; prior job executables were cleaned up, so ten-minute stall and exact shim replay await own rebuilt artifacts. No host load generators or production access.

Delta review source inspection complete; no new confirmed defect yet. Full pristine-head fast gate is queued through build-slot; all four shared slots are occupied. Focused single-crate durable-state tests are compiling separately. Performance VM is reachable; prior job executables were cleaned up, so ten-minute stall and exact shim replay await own rebuilt artifacts. No host load generators or production access.
kayg commented 2026-10-15 22:14:52 +00:00
Author
Owner
Copy link

Base correction, same as prior review: local dev is stale; current origin/dev merge base with 8cd49d505 is fa904fdaa8. Delta scope remains exactly 8003e2134..8cd49d505 (13 lane commits, no merge commits). Size comparison uses fa904fdaa, excluding inherited absolute-cap debt. Earlier regression commits are already present in the target.

Base correction, same as prior review: local dev is stale; current origin/dev merge base with 8cd49d505 is fa904fdaa8d6e28c0ffeee8550ff1d462c169817. Delta scope remains exactly 8003e2134..8cd49d505 (13 lane commits, no merge commits). Size comparison uses fa904fdaa, excluding inherited absolute-cap debt. Earlier regression commits are already present in the target.
kayg commented 2026-10-15 22:19:21 +00:00
Author
Owner
Copy link

Completed focused verification of the changed runtime boundary on pristine 8cd49d505: 3 passed, 0 failed (normal completion, returned error with blocked background work, polling panic with blocked background work). Compiled the exact production remote_service.rs directly against the focused target Tokio artifact, without implementation edits. Receipt delta-runtime.log. This is correctness evidence only, not a timing measurement or shim byte-preservation claim. Pristine static artifacts/full gate now hold one normal build slot.

Completed focused verification of the changed runtime boundary on pristine 8cd49d505: 3 passed, 0 failed (normal completion, returned error with blocked background work, polling panic with blocked background work). Compiled the exact production remote_service.rs directly against the focused target Tokio artifact, without implementation edits. Receipt delta-runtime.log. This is correctness evidence only, not a timing measurement or shim byte-preservation claim. Pristine static artifacts/full gate now hold one normal build slot.
kayg commented 2026-10-15 22:26:40 +00:00
Author
Owner
Copy link

Focused exact-production Store verification PASS: 11 passed, 0 failed, 1 performance test intentionally ignored. Includes all 729 existing six-operation lifecycle schedules, 32 concurrent writer rounds, stale/conflicting same-revision refusal, exact legacy binding upgrade, explicit Off over corrupt public data, and filesystem no-follow/temporary-file protections. Added independent scratch fixture passes 64 writer/reopen schedules ×32 operations (u64 boundary versions) and 47-sample reader/slice decoder differential corpus. The small harness includes the unchanged production node_state.rs/dataplane.rs and verbatim Backend enum; it omits unrelated kernel runtime code. This is not yet the full normal crate/gate result. Receipts delta-focused-store.log and delta-focused-store-build.log. Normal library checks and production artifacts remain compiling in the active build slot. No implementation edits.

Focused exact-production Store verification PASS: 11 passed, 0 failed, 1 performance test intentionally ignored. Includes all 729 existing six-operation lifecycle schedules, 32 concurrent writer rounds, stale/conflicting same-revision refusal, exact legacy binding upgrade, explicit Off over corrupt public data, and filesystem no-follow/temporary-file protections. Added independent scratch fixture passes 64 writer/reopen schedules ×32 operations (u64 boundary versions) and 47-sample reader/slice decoder differential corpus. The small harness includes the unchanged production node_state.rs/dataplane.rs and verbatim Backend enum; it omits unrelated kernel runtime code. This is not yet the full normal crate/gate result. Receipts delta-focused-store.log and delta-focused-store-build.log. Normal library checks and production artifacts remain compiling in the active build slot. No implementation edits.
kayg commented 2026-10-15 22:41:00 +00:00
Author
Owner
Copy link

Original R1 P1 is now independently green on pristine 8cd49d505: actual remote_client entrypoint regression PASS, newer_ack_before_old_release=false superseded_write_committed=true persisted_version=2 persisted_access=Deny. Worker bound <=1, terminated-factory replacement and six-hour refusal PASS (728 attempts, supervisor never returned). Exact supervisor run: 8 passed, 0 failed. Normal mesh state suite: 9 passed, 0 failed, 1 measurement ignored. These are real OS workers/local socket/real Store checks, not packet-acceptance assertions. No implementation changes. Raw receipt delta-supervisor.log; full gate, R2 paired static artifacts and VM long-stall/replay checks pending.

Original R1 P1 is now independently green on pristine 8cd49d505: actual remote_client entrypoint regression PASS, newer_ack_before_old_release=false superseded_write_committed=true persisted_version=2 persisted_access=Deny. Worker bound <=1, terminated-factory replacement and six-hour refusal PASS (728 attempts, supervisor never returned). Exact supervisor run: 8 passed, 0 failed. Normal mesh state suite: 9 passed, 0 failed, 1 measurement ignored. These are real OS workers/local socket/real Store checks, not packet-acceptance assertions. No implementation changes. Raw receipt delta-supervisor.log; full gate, R2 paired static artifacts and VM long-stall/replay checks pending.
kayg commented 2026-10-15 22:58:08 +00:00
Author
Owner
Copy link

D128 delta proof complete on independently built pristine static 8cd49d505 (22,299,504 B; SHA256 b5170c669a4ecb37eaa2ed08c57ebb7ad55ced59bf9aa0f34d300e2d1d8725e1). Exact original a442 shim fixture PASS: 8 real status-1 exits, 2,097,152 payload bytes, same shim, lost=0 duplicated=0. Additional production run-wrapper returned-error and polling-panic fixtures each PASS with the same 8 exits/2,097,152 bytes. Total 24 exits, 6,291,456 bytes exactly replayed; each run keeps its original shim PID. Real shipping shim + actual authenticated dispatcher/runtime helpers, no live service restart or field-cause claim. Logs delta-link-exit-original/return/panic.log. Server harness/full gate still building; VM ten-minute stall and paired baseline size pending.

D128 delta proof complete on independently built pristine static 8cd49d505 (22,299,504 B; SHA256 b5170c669a4ecb37eaa2ed08c57ebb7ad55ced59bf9aa0f34d300e2d1d8725e1). Exact original a442 shim fixture PASS: 8 real status-1 exits, 2,097,152 payload bytes, same shim, lost=0 duplicated=0. Additional production run-wrapper returned-error and polling-panic fixtures each PASS with the same 8 exits/2,097,152 bytes. Total 24 exits, 6,291,456 bytes exactly replayed; each run keeps its original shim PID. Real shipping shim + actual authenticated dispatcher/runtime helpers, no live service restart or field-cause claim. Logs delta-link-exit-original/return/panic.log. Server harness/full gate still building; VM ten-minute stall and paired baseline size pending.
kayg commented 2026-10-15 23:17:25 +00:00
Author
Owner
Copy link

R2 existing size regression currently FAILS on fresh artifacts: fa904fdaa baseline 22,277,680 B (887998b352d7df64d2a3825c85b81dec669c9fb39d570ebd57fa0d4cf5965201), head 22,299,504 B (b5170c669a4ecb37eaa2ed08c57ebb7ad55ced59bf9aa0f34d300e2d1d8725e1): +21,824 B, 5,440 B above its 16,384 B allowance. Original committed Bun test: 0 pass/1 fail. Rebuilt head at the identical temporary baseline checkout path: byte-identical to the independently built review-head artifact. The baseline count disagrees with the older 22,329,104 B receipt; repeating baseline at that same path before the final size finding. Same pinned default-feature stripped static recipe/wrappers; no cap/profile changes. Source pristine fast gate is in clippy; perf VM long-stall run awaits the shared lock. R1 remains resolved; no P0/P1 found.

R2 existing size regression currently FAILS on fresh artifacts: fa904fdaa baseline 22,277,680 B (887998b352d7df64d2a3825c85b81dec669c9fb39d570ebd57fa0d4cf5965201), head 22,299,504 B (b5170c669a4ecb37eaa2ed08c57ebb7ad55ced59bf9aa0f34d300e2d1d8725e1): +21,824 B, 5,440 B above its 16,384 B allowance. Original committed Bun test: 0 pass/1 fail. Rebuilt head at the identical temporary baseline checkout path: byte-identical to the independently built review-head artifact. The baseline count disagrees with the older 22,329,104 B receipt; repeating baseline at that same path before the final size finding. Same pinned default-feature stripped static recipe/wrappers; no cap/profile changes. Source pristine fast gate is in clippy; perf VM long-stall run awaits the shared lock. R1 remains resolved; no P0/P1 found.
kayg commented 2026-10-15 23:28:22 +00:00
Author
Owner
Copy link

R2 resolved with valid same-path controls: baseline fa904fdaa 22,329,104 B (df2b800889e7bffb5ffafdb0b17d38203636387c0c6fdbfeb44aff085417d2f7), head 8cd49d505 22,299,504 B (b5170c669a4ecb37eaa2ed08c57ebb7ad55ced59bf9aa0f34d300e2d1d8725e1): -29,600 B. Original committed paired-artifact Bun test PASS: 1 pass/0 fail. Head built at the baseline checkout path is byte-identical to the original independently built head.

Correction to the provisional +21,824 B result: I initially shared the target directory with a newer head build and an older checkout. Cargo reused newer dependencies (initial baseline log never compiles Mesh); that mixed baseline was invalid. Explicit head->baseline switching at the same path forces the changed sources to rebuild (repeat log includes Mesh) and reproduces the earlier 22,329,104 B baseline. The invalid first artifact/log is retained and excluded. No budget/profile/cap changes. R1 and R2 now independently pass; no new confirmed findings. Perf VM long test has entered its 600s SIGSTOP/TCP-accepting phase; pristine gate is in static builds.

R2 resolved with valid same-path controls: baseline fa904fdaa 22,329,104 B (df2b800889e7bffb5ffafdb0b17d38203636387c0c6fdbfeb44aff085417d2f7), head 8cd49d505 22,299,504 B (b5170c669a4ecb37eaa2ed08c57ebb7ad55ced59bf9aa0f34d300e2d1d8725e1): -29,600 B. Original committed paired-artifact Bun test PASS: 1 pass/0 fail. Head built at the baseline checkout path is byte-identical to the original independently built head. Correction to the provisional +21,824 B result: I initially shared the target directory with a newer head build and an older checkout. Cargo reused newer dependencies (initial baseline log never compiles Mesh); that mixed baseline was invalid. Explicit head->baseline switching at the same path forces the changed sources to rebuild (repeat log includes Mesh) and reproduces the earlier 22,329,104 B baseline. The invalid first artifact/log is retained and excluded. No budget/profile/cap changes. R1 and R2 now independently pass; no new confirmed findings. Perf VM long test has entered its 600s SIGSTOP/TCP-accepting phase; pristine gate is in static builds.
kayg commented 2026-10-15 23:36:34 +00:00
Author
Owner
Copy link

Independent perf VM long-stall proof PASS on saved pristine static 8cd49d505 (b5170c66..., 22,299,504 B). Real server SIGSTOP=600s while TCP still accepts, then 120s bidirectional transport discard/abrupt stop and cold replacement. Same infraternal agent stayed alive; listener-ready to terminal admission 20,911,430 us, within existing 31s functional deadline; replacement CA locked and new enrollment still refused. 1 passed/0 failed. Machine infraternal-perf-test, 4 cores; finish load 0.03/0.03/0.19. This does not identify the original field wait or prove a 300ms network-switch budget. RSS is one full-daemon workflow with two initial/three final PTYs, not an at-scale idle claim. Receipt delta-perf-long-stall.log.

The initial fast run is interim: shared-target reuse also exposed the baseline x86 binary there. Source timestamps were refreshed without content changes; a final full head gate is queued after the current ARM build, with all baseline controls finished. Store/latency VM measurements await the shared lock.

Independent perf VM long-stall proof PASS on saved pristine static 8cd49d505 (b5170c66..., 22,299,504 B). Real server SIGSTOP=600s while TCP still accepts, then 120s bidirectional transport discard/abrupt stop and cold replacement. Same infraternal agent stayed alive; listener-ready to terminal admission 20,911,430 us, within existing 31s functional deadline; replacement CA locked and new enrollment still refused. 1 passed/0 failed. Machine infraternal-perf-test, 4 cores; finish load 0.03/0.03/0.19. This does not identify the original field wait or prove a 300ms network-switch budget. RSS is one full-daemon workflow with two initial/three final PTYs, not an at-scale idle claim. Receipt delta-perf-long-stall.log. The initial fast run is interim: shared-target reuse also exposed the baseline x86 binary there. Source timestamps were refreshed without content changes; a final full head gate is queued after the current ARM build, with all baseline controls finished. Store/latency VM measurements await the shared lock.
kayg commented 2026-10-15 23:55:44 +00:00
Author
Owner
Copy link

Perf VM verification complete (4 cores, unchanged hardware). Exact-production isolated Store fixtures, 1000 peers/206907 B/101 durable saves per artifact, two batches with reversed order and lock released between batches:

Baseline/candidate p50 ms: 20.129/19.884, then16.722/18.675. p99 ms:29.940/29.220, then24.225/29.443. Added child CPU amortized1.86/1.54 ms per update; observed post-exec VmHWM delta380/372 KiB. At1000 distributed instances that is380000/372000 KiB arithmetic only, not concurrent fleet or whole-daemon idle memory. This isolates the durable revision check/read cost, not terminal dispatch. Batch1 load0.53/1.23/0.84->0.73/1.26/0.85; batch2 finish5.52/2.33/1.28. Tail variation is retained; no p99 improvement/budget claim.

Existing real-PTY 300ms RTT±30ms latency gate remains RED on both actual fa baseline and8cd head, same-PTY flood echo p99461.29/471.89 ms versus334; growth135.96/143.71 ms versus16.67. Idle echo p99325.33/328.19 ms PASS; other-PTY isolation PASS. One paired run with burst jitter/load variation does not establish a branch-specific regression or a latency budget pass. This repeats the already-recorded baseline failure; no threshold relaxed. Raw receipts delta-perf-store.log and delta-perf-latency-{baseline,candidate}.log.

Interim host test failure was fixture setup: my TMPDIR was too long for the existing Unix rendezvous. A private34-character scratch directory is prepared for the actual final gate; no code or test limit changed. Pristine final gate and local review-test commits still pending.

Perf VM verification complete (4 cores, unchanged hardware). Exact-production isolated Store fixtures, 1000 peers/206907 B/101 durable saves per artifact, two batches with reversed order and lock released between batches: Baseline/candidate p50 ms: 20.129/19.884, then16.722/18.675. p99 ms:29.940/29.220, then24.225/29.443. Added child CPU amortized1.86/1.54 ms per update; observed post-exec VmHWM delta380/372 KiB. At1000 distributed instances that is380000/372000 KiB arithmetic only, not concurrent fleet or whole-daemon idle memory. This isolates the durable revision check/read cost, not terminal dispatch. Batch1 load0.53/1.23/0.84->0.73/1.26/0.85; batch2 finish5.52/2.33/1.28. Tail variation is retained; no p99 improvement/budget claim. Existing real-PTY 300ms RTT±30ms latency gate remains RED on both actual fa baseline and8cd head, same-PTY flood echo p99461.29/471.89 ms versus334; growth135.96/143.71 ms versus16.67. Idle echo p99325.33/328.19 ms PASS; other-PTY isolation PASS. One paired run with burst jitter/load variation does not establish a branch-specific regression or a latency budget pass. This repeats the already-recorded baseline failure; no threshold relaxed. Raw receipts delta-perf-store.log and delta-perf-latency-{baseline,candidate}.log. Interim host test failure was fixture setup: my TMPDIR was too long for the existing Unix rendezvous. A private34-character scratch directory is prepared for the actual final gate; no code or test limit changed. Pristine final gate and local review-test commits still pending.
kayg commented 2026-10-16 00:11:20 +00:00
Author
Owner
Copy link

Pristine 8cd49d505 final scripts/verify fast completed, exit1:1100 Rust tests passed,0 failed,141 ignored. Every Rust row PASS except inherited absolute static-size caps. Actual head artifacts x8622299504 B (+95200 vs cap), ARM19726904 B (+88160); final x86 SHA b5170c669a4ecb37eaa2ed08c57ebb7ad55ced59bf9aa0f34d300e2d1d8725e1 matches the independently measured/replayed/VM-tested saved head. Source content stayed exactly8cd; final runtime root was a real private34-character path. Invalid shared-cache and long-TMPDIR interim runs retained but excluded. Separate TypeScript gate PASS:191 tracer/UI tests +37 protocol tests, compiler/build/checksum/transfer rows PASS. No cap or test limit changed.

Final fast summary verbatim:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Restoring missing independent a442 shim fixture now; normal crate lint/test checks precede local evidence commits. No implementation changes, push or deployment.

Pristine 8cd49d505 final scripts/verify fast completed, exit1:1100 Rust tests passed,0 failed,141 ignored. Every Rust row PASS except inherited absolute static-size caps. Actual head artifacts x8622299504 B (+95200 vs cap), ARM19726904 B (+88160); final x86 SHA b5170c669a4ecb37eaa2ed08c57ebb7ad55ced59bf9aa0f34d300e2d1d8725e1 matches the independently measured/replayed/VM-tested saved head. Source content stayed exactly8cd; final runtime root was a real private34-character path. Invalid shared-cache and long-TMPDIR interim runs retained but excluded. Separate TypeScript gate PASS:191 tracer/UI tests +37 protocol tests, compiler/build/checksum/transfer rows PASS. No cap or test limit changed. Final fast summary verbatim: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` Restoring missing independent a442 shim fixture now; normal crate lint/test checks precede local evidence commits. No implementation changes, push or deployment.
kayg commented 2026-10-16 00:20:47 +00:00
Author
Owner
Copy link

Review evidence step committed locally: dcea5cc3d1 restores prior a442 independent shim fixture on review-agent-reconnect (base8cd49d505). Normal CLI pedantic all-target lint and fmt PASS; normal crate test executable runtime units3 PASS; exact original driver8 status-1 exits/2097152 bytes/same shim/lost0/duplicated0 PASS. No implementation change or push. Strengthening run-wrapper return/panic replay next.

Review evidence step committed locally: dcea5cc3d127dfa0bc59c925be5470d12533aacc restores prior a442 independent shim fixture on review-agent-reconnect (base8cd49d505). Normal CLI pedantic all-target lint and fmt PASS; normal crate test executable runtime units3 PASS; exact original driver8 status-1 exits/2097152 bytes/same shim/lost0/duplicated0 PASS. No implementation change or push. Strengthening run-wrapper return/panic replay next.
kayg commented 2026-10-16 00:21:53 +00:00
Author
Owner
Copy link

Review evidence commit 3ee0044c6: normal CLI all-target clippy and formatting passed; 3 runtime unit tests passed. The real runtime return and polling-panic paths each survived 8 nonzero link exits and replayed 2,097,152 nonperiodic bytes exactly through the same shim (16 exits / 4 MiB total; zero loss or duplication). No implementation fix. Raw receipts: delta-strengthened-return.log and delta-strengthened-panic.log.

Review evidence commit 3ee0044c6: normal CLI all-target clippy and formatting passed; 3 runtime unit tests passed. The real runtime return and polling-panic paths each survived 8 nonzero link exits and replayed 2,097,152 nonperiodic bytes exactly through the same shim (16 exits / 4 MiB total; zero loss or duplication). No implementation fix. Raw receipts: delta-strengthened-return.log and delta-strengthened-panic.log.
kayg commented 2026-10-16 00:26:17 +00:00
Author
Owner
Copy link

Review evidence commit 068488eac: both independent normal Mesh integration tests pass (64 x32 durable writer/reopen operations including u64 boundaries; 47-sample reader/slice decoder corpus). All-target Mesh clippy with agent feature and cargo fmt pass. No implementation change or new confirmed finding. Receipts: delta-property-final.log and delta-property-clippy.log, attached.

Review evidence commit 068488eac: both independent normal Mesh integration tests pass (64 x32 durable writer/reopen operations including u64 boundaries; 47-sample reader/slice decoder corpus). All-target Mesh clippy with agent feature and cargo fmt pass. No implementation change or new confirmed finding. Receipts: delta-property-final.log and delta-property-clippy.log, attached.
kayg commented 2026-10-16 00:26:36 +00:00
Author
Owner
Copy link

MERGE (no P0/P1) — merge. Delta only: 8003e2134..8cd49d505, excluding merged dev changes. Findings remaining: P0=0, P1=0, P2=0, P3=0. Earlier R1 (P1) and R2 (P2) resolved; no new confirmed findings.

Target: 8cd49d5057. Local evidence branch: review-agent-reconnect, HEAD 068488eac9. No implementation fixes, push, deployment or issue closure.

R1 original actual-entrypoint regression passes: newer_ack_before_old_release=false; superseded_write_committed=true; persisted_version=2; persisted_access=Deny. Eight supervisor tests pass, including <=1 live OS worker and six virtual hours / 728 attempts without supervisor return. Independent property tests pass: 64 deterministic two-handle writer/reopen schedules x32 operations, versions through u64::MAX; 47 valid/invalid public snapshots agree between reader and slice decoders. These bounded schedules are not an exhaustive proof over all possible executions.

The normal CLI fixture passes 3 runtime tests. Real returned-error and polling-panic paths each survive 8 nonzero link exits and replay 2,097,152 nonperiodic bytes exactly through the same shim (16 exits / 4 MiB, zero loss or duplication). Restored original replay fixture independently passes 8 exits / 2 MiB. Touched-crate all-target clippy and formatting pass.

R2 controlled same-path stripped x86 release: merge base fa904fdaa=22,329,104 B; head=22,299,504 B; delta=-29,600 B. Committed size regression passes. Head SHA256 b5170c669a4ecb37eaa2ed08c57ebb7ad55ced59bf9aa0f34d300e2d1d8725e1, identical at both checkout paths. Exclude initial mixed-cache baseline and interim gate runs described in earlier comments.

Perf VM only (4 vCPU, MemTotal 8,069,112 KiB): real server stall 600 s, followed by 120 s packet discard and cold replacement, passes; original infraternal agent survives and reconnects with CA locked. Replacement listener to terminal recovery: 20,911,430 us. This is long-stall recovery evidence, not the 300 ms network-switch claim or proof of the owner's historical incident cause. VM start/finish loads 0.00/0.10/0.42 and 0.03/0.03/0.19.

At 1,000 peers / 206,907-B snapshot / 101 saves per artifact, two reverse-order Store batches show an additional 1.54–1.86 ms child CPU per update and +372–380 KiB peak memory for the isolated Store fixture. Batch1 p50/p99 ms base=20.129/29.940, head=19.884/29.220; batch2 base=16.722/24.225, head=18.675/29.443. No fleet-idle memory or tail-improvement claim. Existing same-PTY flood latency remains red on BOTH baseline and head (p99 461.29/471.89 ms, budget 334 ms; growth 135.96/143.71 ms, budget 16.67 ms). Idle echo and cross-PTY isolation pass. One paired loaded/jittered run does not establish a delta-specific regression; no performance budget relaxed.

Pristine target scripts/verify fast: 1,100 Rust tests pass, 0 fail, 141 ignored. Exit 1 solely for inherited absolute binary caps: x86 22,299,504 vs 22,204,304 B; ARM 19,726,904 vs 19,638,744 B. Judge by merge-base delta per brief. Verbatim final gate output:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

The separately executed normal TypeScript gate passes: 191 tracer/UI +37 protocol tests, plus its compiler, checksum, build and transfer rows. Raw valid gate: delta-verify-fast.log; TS: delta-verify-ts.log. Interim runtime-path/cache runs are excluded.

Reviewer checklist completed for this delta: retained intent and version fence, worker ownership, decoder compatibility, unchanged access rules and contracts, failure/restart paths, bounded worker count, real shim replay, size delta, perf VM receipts. No new engine/dependency or UI change. No new fuzz run or exhaustive concurrency proof claimed. Raw evidence and findings journal are retained outside checkout under the job paths; selected logs attached to #225. Own VM directory/scopes removed; local temporary artifacts cleaned; checkout clean at 068488eac9.

MERGE (no P0/P1) — merge. Delta only: 8003e2134..8cd49d505, excluding merged dev changes. Findings remaining: P0=0, P1=0, P2=0, P3=0. Earlier R1 (P1) and R2 (P2) resolved; no new confirmed findings. Target: 8cd49d5057ca4df9b839402632892c0b286d78f2. Local evidence branch: review-agent-reconnect, HEAD 068488eac942fd41be15e021db5c08996252d63c. No implementation fixes, push, deployment or issue closure. R1 original actual-entrypoint regression passes: newer_ack_before_old_release=false; superseded_write_committed=true; persisted_version=2; persisted_access=Deny. Eight supervisor tests pass, including <=1 live OS worker and six virtual hours / 728 attempts without supervisor return. Independent property tests pass: 64 deterministic two-handle writer/reopen schedules x32 operations, versions through u64::MAX; 47 valid/invalid public snapshots agree between reader and slice decoders. These bounded schedules are not an exhaustive proof over all possible executions. The normal CLI fixture passes 3 runtime tests. Real returned-error and polling-panic paths each survive 8 nonzero link exits and replay 2,097,152 nonperiodic bytes exactly through the same shim (16 exits / 4 MiB, zero loss or duplication). Restored original replay fixture independently passes 8 exits / 2 MiB. Touched-crate all-target clippy and formatting pass. R2 controlled same-path stripped x86 release: merge base fa904fdaa=22,329,104 B; head=22,299,504 B; delta=-29,600 B. Committed size regression passes. Head SHA256 b5170c669a4ecb37eaa2ed08c57ebb7ad55ced59bf9aa0f34d300e2d1d8725e1, identical at both checkout paths. Exclude initial mixed-cache baseline and interim gate runs described in earlier comments. Perf VM only (4 vCPU, MemTotal 8,069,112 KiB): real server stall 600 s, followed by 120 s packet discard and cold replacement, passes; original infraternal agent survives and reconnects with CA locked. Replacement listener to terminal recovery: 20,911,430 us. This is long-stall recovery evidence, not the 300 ms network-switch claim or proof of the owner's historical incident cause. VM start/finish loads 0.00/0.10/0.42 and 0.03/0.03/0.19. At 1,000 peers / 206,907-B snapshot / 101 saves per artifact, two reverse-order Store batches show an additional 1.54–1.86 ms child CPU per update and +372–380 KiB peak memory for the isolated Store fixture. Batch1 p50/p99 ms base=20.129/29.940, head=19.884/29.220; batch2 base=16.722/24.225, head=18.675/29.443. No fleet-idle memory or tail-improvement claim. Existing same-PTY flood latency remains red on BOTH baseline and head (p99 461.29/471.89 ms, budget 334 ms; growth 135.96/143.71 ms, budget 16.67 ms). Idle echo and cross-PTY isolation pass. One paired loaded/jittered run does not establish a delta-specific regression; no performance budget relaxed. Pristine target scripts/verify fast: 1,100 Rust tests pass, 0 fail, 141 ignored. Exit 1 solely for inherited absolute binary caps: x86 22,299,504 vs 22,204,304 B; ARM 19,726,904 vs 19,638,744 B. Judge by merge-base delta per brief. Verbatim final gate output: ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` The separately executed normal TypeScript gate passes: 191 tracer/UI +37 protocol tests, plus its compiler, checksum, build and transfer rows. Raw valid gate: delta-verify-fast.log; TS: delta-verify-ts.log. Interim runtime-path/cache runs are excluded. Reviewer checklist completed for this delta: retained intent and version fence, worker ownership, decoder compatibility, unchanged access rules and contracts, failure/restart paths, bounded worker count, real shim replay, size delta, perf VM receipts. No new engine/dependency or UI change. No new fuzz run or exhaustive concurrency proof claimed. Raw evidence and findings journal are retained outside checkout under the job paths; selected logs attached to #225. Own VM directory/scopes removed; local temporary artifacts cleaned; checkout clean at 068488eac942fd41be15e021db5c08996252d63c.
kayg commented 2026-10-16 00:28:21 +00:00
Author
Owner
Copy link

Orchestrator: delta review MERGE (068488eac). Merged to dev as 492e1505a (prod 036bc2bb0 + this fix; no other code change). Release build + signed staging proof running in job reconnect-release (branch reconnect-release), including production-binary → new-binary single restart and a ≥10-min staging server stall. No deploy yet: the deploy restarts the server (CA re-locks) and the three stuck agents need the owner's OK for one service restart each.

Orchestrator: delta review MERGE (068488eac). Merged to dev as **492e1505a** (prod 036bc2bb0 + this fix; no other code change). Release build + signed staging proof running in job `reconnect-release` (branch reconnect-release), including production-binary → new-binary single restart and a ≥10-min staging server stall. No deploy yet: the deploy restarts the server (CA re-locks) and the three stuck agents need the owner's OK for one service restart each.
kayg commented 2026-10-16 00:28:45 +00:00
Author
Owner
Copy link

Starting reconnect-release on branch reconnect-release at base/head 492e1505ab. Read all 61 prior comments; no parent linked. Confirmed origin/release-prod-live 036bc2bb0 is ancestor. Build exact release artifacts, signed staging plus production-binary single-restart and >=600s server-stall proof, then final fast gate. No deploy or owner-fixture access. Prior switchover-release artifacts directory is absent; recorded production checksums are available in prior job result. Existing staging-proof rebuilds temporary-trust bytes, which will be reported separately from exact release-image checks.

Starting reconnect-release on branch reconnect-release at base/head 492e1505ab6822078268344382f4d246d2c3d65e. Read all 61 prior comments; no parent linked. Confirmed origin/release-prod-live 036bc2bb0 is ancestor. Build exact release artifacts, signed staging plus production-binary single-restart and >=600s server-stall proof, then final fast gate. No deploy or owner-fixture access. Prior switchover-release artifacts directory is absent; recorded production checksums are available in prior job result. Existing staging-proof rebuilds temporary-trust bytes, which will be reported separately from exact release-image checks.
kayg commented 2026-10-16 00:30:10 +00:00
Author
Owner
Copy link

Release preparation finding: scripts/release/staging-proof:bundle/main necessarily recompiles x86 with per-run staging-trust, uses fixture web and an ARM placeholder; standard production builds intentionally reject that authority (scripts/release/check-production-trust.py). Thus its signed proof cannot be described as exercising exact production bytes. Will reproduce prior signed staging with explicit derived-image provenance and independently run exact frozen production binaries/web through restart and long-stall acceptance. No production signing material will be searched/read. Previous production artifact directory is absent; production x86 reference 29d65925/22329104B, ARM e120c1ad/19752920B, canonical web 99414fea/3543040B retained in switchover-release job result. Baseline needed for old->new service case will be rebuilt from exact 036bc2bb0 if no staging copy remains. Build currently waiting for required slot; priority slot held by size-ratchet-r2-fast-rebased, no other process interrupted.

Release preparation finding: scripts/release/staging-proof:bundle/main necessarily recompiles x86 with per-run staging-trust, uses fixture web and an ARM placeholder; standard production builds intentionally reject that authority (scripts/release/check-production-trust.py). Thus its signed proof cannot be described as exercising exact production bytes. Will reproduce prior signed staging with explicit derived-image provenance and independently run exact frozen production binaries/web through restart and long-stall acceptance. No production signing material will be searched/read. Previous production artifact directory is absent; production x86 reference 29d65925/22329104B, ARM e120c1ad/19752920B, canonical web 99414fea/3543040B retained in switchover-release job result. Baseline needed for old->new service case will be rebuilt from exact 036bc2bb0 if no staging copy remains. Build currently waiting for required slot; priority slot held by size-ratchet-r2-fast-rebased, no other process interrupted.
kayg commented 2026-10-16 00:37:38 +00:00
Author
Owner
Copy link

Exact-image proof refinement: keep the standard 12-handoff/refusal/rollback staging test unchanged. Add a supplemental one-way signed handoff from its ephemeral-trust image into E, containing the frozen requested x86/ARM/web bytes. The old temporary-trust process admits the ephemeral signature; the receiving exact production image retains no staging authority. A test-only scratch snapshot reuses all 128-shim/128-echo/6-retained-socket/real-passkey assertions, with one handoff and no ephemeral rollback after the production image is running. No production key or owner fixture is accessed. Supplemental fixture changes are outside the checkout and will be attached. Actual execution remains pending the build slot.

Exact-image proof refinement: keep the standard 12-handoff/refusal/rollback staging test unchanged. Add a supplemental one-way signed handoff from its ephemeral-trust image into E, containing the frozen requested x86/ARM/web bytes. The old temporary-trust process admits the ephemeral signature; the receiving exact production image retains no staging authority. A test-only scratch snapshot reuses all 128-shim/128-echo/6-retained-socket/real-passkey assertions, with one handoff and no ephemeral rollback after the production image is running. No production key or owner fixture is accessed. Supplemental fixture changes are outside the checkout and will be attached. Actual execution remains pending the build slot.
kayg commented 2026-10-16 01:06:06 +00:00
Author
Owner
Copy link

Build checkpoint: required priority slot acquired 2026-10-16T02:40:13+02:00 by reconnect-release PID4127859; x86 release still compiling, no error/artifact yet. Pinned Zig0.15.2 and exact nix make/perl recipe; one target directory. Source remains pristine492e1505a; git diff confirms apps/tracer, packages/ui, packages/protocol unchanged from036bc2bb0. Perf VM facts recorded:4 cores,8069112KiB, kernel6.12.111+deb13-amd64.

Resume state: all operational helpers in ~/.cache/codex-scratch/reconnect-release/tools/. validation.sh waits for initial release command to end, then freezes hashes/web tar, runs the exact production browser groups, rebuilds036bc2bb0, compiles test-only service/exact-handoff fixtures, runs >=600s SIGSTOP+120s bidirectional discard with unchanged new-service PID and original shim/session checks, standard signed staging plus one-way signed exact-image handoff, final pristine fast/TS gates. Evidence logs under evidence/. No acceptance result yet, no deployment or owner-fixture access. Temporary-trust derived images will be identified separately from exact release bytes.

Build checkpoint: required priority slot acquired 2026-10-16T02:40:13+02:00 by reconnect-release PID4127859; x86 release still compiling, no error/artifact yet. Pinned Zig0.15.2 and exact nix make/perl recipe; one target directory. Source remains pristine492e1505a; git diff confirms apps/tracer, packages/ui, packages/protocol unchanged from036bc2bb0. Perf VM facts recorded:4 cores,8069112KiB, kernel6.12.111+deb13-amd64. Resume state: all operational helpers in ~/.cache/codex-scratch/reconnect-release/tools/. validation.sh waits for initial release command to end, then freezes hashes/web tar, runs the exact production browser groups, rebuilds036bc2bb0, compiles test-only service/exact-handoff fixtures, runs >=600s SIGSTOP+120s bidirectional discard with unchanged new-service PID and original shim/session checks, standard signed staging plus one-way signed exact-image handoff, final pristine fast/TS gates. Evidence logs under evidence/. No acceptance result yet, no deployment or owner-fixture access. Temporary-trust derived images will be identified separately from exact release bytes.
kayg commented 2026-10-16 01:06:35 +00:00
Author
Owner
Copy link

Release x86 completed on pristine492e1505a, required priority slot/pinned Zig recipe. Artifact: /home/kayg/.cache/codex-scratch/reconnect-release/artifacts/infraternal-x86_64-musl; SHA256 23fc5e92df83e49da2d351cc19fbf24983475eebc2ce98c961e03bd29254b2c6;22299504B, delta -29600B versus recorded production22329104B. Stripped static PIE, no PT_INTERP; check-production-trust.py PASS (staging authority absent). ARM build has started. Web, staging and final gate pending; no deploy. Full receipt: evidence/x86-artifact.json under reconnect-release scratch.

Release x86 completed on pristine492e1505a, required priority slot/pinned Zig recipe. Artifact: /home/kayg/.cache/codex-scratch/reconnect-release/artifacts/infraternal-x86_64-musl; SHA256 23fc5e92df83e49da2d351cc19fbf24983475eebc2ce98c961e03bd29254b2c6;22299504B, delta -29600B versus recorded production22329104B. Stripped static PIE, no PT_INTERP; check-production-trust.py PASS (staging authority absent). ARM build has started. Web, staging and final gate pending; no deploy. Full receipt: evidence/x86-artifact.json under reconnect-release scratch.
kayg commented 2026-10-16 01:40:21 +00:00
Author
Owner
Copy link

Release artifacts frozen at pristine492e1505a; required build-slot/pinned Zig0.15.2 recipe, both production-trust/no-PT_INTERP checks PASS:

  • /home/kayg/.cache/codex-scratch/reconnect-release/artifacts/infraternal-x86_64-musl — 22299504 B; SHA256 23fc5e92df83e49da2d351cc19fbf24983475eebc2ce98c961e03bd29254b2c6
  • /home/kayg/.cache/codex-scratch/reconnect-release/artifacts/infraternal-aarch64-musl — 19726904 B; SHA256 f47a8b8396034686e91cc0ae862dc73a21be086a522ab925135669bf53b9806a
  • /home/kayg/.cache/codex-scratch/reconnect-release/artifacts/web-dist.tar — 3543040 B; SHA256 e783405b15bd91bcd8b3696f1d2c55564df259ff5ad04525219a941ce5be1a85
    Native deltas versus recorded036bc2bb0 production: x86 -29600B, ARM -26016B. Original switchover artifact directory is absent; both counts match the reviewed fix. Web:253 files,3333749B; per-file manifest /home/kayg/.cache/codex-scratch/reconnect-release/artifacts/web-SHA256SUMS (SHA256 53f513134ee07401c239c2e7809f64a55302877c06c5b07566e01db495141524).

Web is NOT byte-identical to recorded production: build.ts:19–33 embeds git HEAD and current ISO date; this dist embeds492e1505a and2026-10-16T01:38:46.041Z, versus production036bc2bb0. All tracer/UI/protocol/lock inputs are unchanged. The old dist is absent, so no complete per-file diff is claimed. Current compressed dist totals32B fewer than its recorded3333781B. Canonical tar uses sorted names,mtime0,owner/group0. Transfer PASS: critical gzip296286/307200B,Brotli246350/256000B; deferred47853/51200B,42406/45056B; largest on-demand15948/20480B,14188/18432B.

Exact-artifact browser restart group2/2 PASS on perf VM; other groups running. Rebuilding old036bc2bb0 for the enrolled-service test; signed staging and final gate pending. No deploy.
artifacts.json
web-comparison.json
source-provenance.json
release-build.log

Release artifacts frozen at pristine492e1505a; required build-slot/pinned Zig0.15.2 recipe, both production-trust/no-PT_INTERP checks PASS: - /home/kayg/.cache/codex-scratch/reconnect-release/artifacts/infraternal-x86_64-musl — 22299504 B; SHA256 23fc5e92df83e49da2d351cc19fbf24983475eebc2ce98c961e03bd29254b2c6 - /home/kayg/.cache/codex-scratch/reconnect-release/artifacts/infraternal-aarch64-musl — 19726904 B; SHA256 f47a8b8396034686e91cc0ae862dc73a21be086a522ab925135669bf53b9806a - /home/kayg/.cache/codex-scratch/reconnect-release/artifacts/web-dist.tar — 3543040 B; SHA256 e783405b15bd91bcd8b3696f1d2c55564df259ff5ad04525219a941ce5be1a85 Native deltas versus recorded036bc2bb0 production: x86 -29600B, ARM -26016B. Original switchover artifact directory is absent; both counts match the reviewed fix. Web:253 files,3333749B; per-file manifest /home/kayg/.cache/codex-scratch/reconnect-release/artifacts/web-SHA256SUMS (SHA256 53f513134ee07401c239c2e7809f64a55302877c06c5b07566e01db495141524). Web is NOT byte-identical to recorded production: build.ts:19–33 embeds git HEAD and current ISO date; this dist embeds492e1505a and2026-10-16T01:38:46.041Z, versus production036bc2bb0. All tracer/UI/protocol/lock inputs are unchanged. The old dist is absent, so no complete per-file diff is claimed. Current compressed dist totals32B fewer than its recorded3333781B. Canonical tar uses sorted names,mtime0,owner/group0. Transfer PASS: critical gzip296286/307200B,Brotli246350/256000B; deferred47853/51200B,42406/45056B; largest on-demand15948/20480B,14188/18432B. Exact-artifact browser restart group2/2 PASS on perf VM; other groups running. Rebuilding old036bc2bb0 for the enrolled-service test; signed staging and final gate pending. No deploy. [artifacts.json](https://git.kayg.org/attachments/271195ea-1520-4161-b164-0e6c6d500691) [web-comparison.json](https://git.kayg.org/attachments/fd2ff377-0f77-477c-9b6b-738379a651de) [source-provenance.json](https://git.kayg.org/attachments/b6ed3e72-d708-464f-93dc-24ba1fb3869b) [release-build.log](https://git.kayg.org/attachments/192d3626-ced9-4091-bc7e-bcc2420ff70f)
kayg commented 2026-10-16 01:53:18 +00:00
Author
Owner
Copy link

Browser acceptance complete:12/12 real systemd/PTY restart cases PASS (2 ordinary,4 retained-route/stalled-upstream,6 three-window isolation),5/5 mocks PASS,0 retries; frozen x8623fc5e92 and web e783405b. VM only, TRACER_TEST_PORT4525, Node22.23.3/bundled Playwright1.61.1. Each group releases the VM lock. All private browser units report MainPID0/inactive; four measurement scopes inactive; owned executable processes0 and browser fixture directory removed. Raw safe receipts attached.

Old production-source image rebuilt from exact036bc2bb0 with the full pinned release recipe:22329104B, SHA2565b17532eb0f9b716ef7893640b3740bc0649c456ac5eaccbd9c00b83891617b5, at ~/.cache/codex-scratch/reconnect-release/artifacts/production-rebuilt-x86_64-musl. Size matches production, but hash differs from the absent original29d65925 copy; this is a source-revision rebuild, not a claim that original production bytes were recovered. Native service/exact-handoff test harness compilation started. No deploy.
browser-summary.json
browser-restart.log
browser-liveness.log
browser-three-window.log
browser-mock.log
browser-cleanup.json
browser-directory-cleanup.txt

Browser acceptance complete:12/12 real systemd/PTY restart cases PASS (2 ordinary,4 retained-route/stalled-upstream,6 three-window isolation),5/5 mocks PASS,0 retries; frozen x8623fc5e92 and web e783405b. VM only, TRACER_TEST_PORT4525, Node22.23.3/bundled Playwright1.61.1. Each group releases the VM lock. All private browser units report MainPID0/inactive; four measurement scopes inactive; owned executable processes0 and browser fixture directory removed. Raw safe receipts attached. Old production-source image rebuilt from exact036bc2bb0 with the full pinned release recipe:22329104B, SHA2565b17532eb0f9b716ef7893640b3740bc0649c456ac5eaccbd9c00b83891617b5, at ~/.cache/codex-scratch/reconnect-release/artifacts/production-rebuilt-x86_64-musl. Size matches production, but hash differs from the absent original29d65925 copy; this is a source-revision rebuild, not a claim that original production bytes were recovered. Native service/exact-handoff test harness compilation started. No deploy. [browser-summary.json](https://git.kayg.org/attachments/cd8c1ae9-e04d-4098-ac29-4573497a23d8) [browser-restart.log](https://git.kayg.org/attachments/fbf9d825-8c99-407f-a49b-b23594f5f4a0) [browser-liveness.log](https://git.kayg.org/attachments/8097daea-721a-4700-9c64-0f34a4bc08a3) [browser-three-window.log](https://git.kayg.org/attachments/88bf8d8c-b9c0-4971-ae5a-a0752e4ab89c) [browser-mock.log](https://git.kayg.org/attachments/19fbede9-377a-493a-80fd-004c1290b2af) [browser-cleanup.json](https://git.kayg.org/attachments/a82ac99a-7f4c-48f4-882d-993fa1cbfb12) [browser-directory-cleanup.txt](https://git.kayg.org/attachments/59d67b1a-69dd-449d-8619-ba203416a0e6)
kayg commented 2026-10-16 02:22:03 +00:00
Author
Owner
Copy link

Native test-only harness and signed staging bundles are ready; production worktree remains pristine492e1505a. Supporting fixture sources/hashes attached. Service proof specifically reattaches a PTY created under the old managed service, checks every pre-upgrade shim PID across one upgrade restart, verifies /proc executable SHA for old5b17532e/new23fc5e92, then 600s stopped userspace plus120s bidirectional TCP discard with unchanged new-service PID and retained-session echo. No measured result yet. Standard signed12-handoff proof remains unchanged; supplemental one-way signed exact-image bundle reports23fc5e92 and includes the actual ARM/web artifacts.

VM queue: service waiting since02:08UTC, captures02:15UTC, signed staging~02:19UTC. Current owner process-per-user/ppu-soak-r8-dev, owner timestamp2026-10-16T02:01:31.241151Z. Waiting normally; no lock bypass or other process interruption. Browser12+5 PASS already attached. Temporary attachment HTTP500 cleared on retry.
fixture-source-hashes.json
service-harness.json
operational-sources.txt

Native test-only harness and signed staging bundles are ready; production worktree remains pristine492e1505a. Supporting fixture sources/hashes attached. Service proof specifically reattaches a PTY created under the old managed service, checks every pre-upgrade shim PID across one upgrade restart, verifies /proc executable SHA for old5b17532e/new23fc5e92, then 600s stopped userspace plus120s bidirectional TCP discard with unchanged new-service PID and retained-session echo. No measured result yet. Standard signed12-handoff proof remains unchanged; supplemental one-way signed exact-image bundle reports23fc5e92 and includes the actual ARM/web artifacts. VM queue: service waiting since~02:08UTC, captures~02:15UTC, signed staging~02:19UTC. Current owner process-per-user/ppu-soak-r8-dev, owner timestamp2026-10-16T02:01:31.241151Z. Waiting normally; no lock bypass or other process interruption. Browser12+5 PASS already attached. Temporary attachment HTTP500 cleared on retry. [fixture-source-hashes.json](https://git.kayg.org/attachments/3334421e-bf1a-447d-bc82-9937f2f8e0d1) [service-harness.json](https://git.kayg.org/attachments/958d7f74-ea63-4b43-8489-9843826ca8f6) [operational-sources.txt](https://git.kayg.org/attachments/70662e72-0473-4679-ad1e-5b045cf36afb)
kayg commented 2026-10-16 02:51:29 +00:00
Author
Owner
Copy link

Signed staging PASS on perf VM. Unchanged standard proof:12 sampled signed handoffs preserve128/128 shim PIDs and128/128 input echoes, CA remains unlocked after one real passkey ceremony,6/6 retained sockets responsive after idle; unusable-image refusal, warm rollback, backup integrity10000 nodes/50 users, and local polling/negative-control test PASS. Supplemental exact-image proof: one signed handoff from temporary-trust A into unmodified production x8623fc5e92, with actual ARM f47a8b83 and web e783405b staged, preserves128/128 shims+echoes, CA unlocked,6 sockets responsive. All3 ignored tests1/0 each; lock released and VM staging directory absent.

VM native listener-gap brackets (conservative, include probe resolution; not exact outage or a network-switch budget claim): standard p50=102.847ms,p99=306.183ms,max probe interval102.288ms; exact one-way bracket102.770448ms.

Production desktop/phone capture repetition2/2 PASS; screenshots attached, fixture units inactive and directory removed. Browser acceptance already12+5 PASS.

Service recovery proof currently INCOMPLETE: first run failed fixture start-old before upgrade/stall (release_service_test.rs:35;0passed/1failed in setup), so no reconnect result or fix failure is claimed. Its own transient unit was stopped; safe operation/manager diagnostics added to the adapter and retry queued. Final pristine492e1505a fast gate has started and is waiting for VM calibration. No shipping changes or deployment.
staging-proof.txt
staging-command.log
browser-captures.log
capture-cleanup.json
restart-locked-ready.png
restart-unlock-ready.png

Signed staging PASS on perf VM. Unchanged standard proof:12 sampled signed handoffs preserve128/128 shim PIDs and128/128 input echoes, CA remains unlocked after one real passkey ceremony,6/6 retained sockets responsive after idle; unusable-image refusal, warm rollback, backup integrity10000 nodes/50 users, and local polling/negative-control test PASS. Supplemental exact-image proof: one signed handoff from temporary-trust A into unmodified production x8623fc5e92, with actual ARM f47a8b83 and web e783405b staged, preserves128/128 shims+echoes, CA unlocked,6 sockets responsive. All3 ignored tests1/0 each; lock released and VM staging directory absent. VM native listener-gap brackets (conservative, include probe resolution; not exact outage or a network-switch budget claim): standard p50=102.847ms,p99=306.183ms,max probe interval102.288ms; exact one-way bracket102.770448ms. Production desktop/phone capture repetition2/2 PASS; screenshots attached, fixture units inactive and directory removed. Browser acceptance already12+5 PASS. Service recovery proof currently INCOMPLETE: first run failed fixture start-old before upgrade/stall (release_service_test.rs:35;0passed/1failed in setup), so no reconnect result or fix failure is claimed. Its own transient unit was stopped; safe operation/manager diagnostics added to the adapter and retry queued. Final pristine492e1505a fast gate has started and is waiting for VM calibration. No shipping changes or deployment. [staging-proof.txt](https://git.kayg.org/attachments/280ec54b-29fd-461e-a753-99c778beb808) [staging-command.log](https://git.kayg.org/attachments/a17d03fb-bca2-48f8-a5f0-06f4a7e5234d) [browser-captures.log](https://git.kayg.org/attachments/5acf20c4-c506-4704-afa9-bb02b9f41add) [capture-cleanup.json](https://git.kayg.org/attachments/d6ab60a0-583c-4c1b-9ddf-37c0888b8a19) ![restart-locked-ready.png](https://git.kayg.org/attachments/60f5b013-80f5-48ad-a192-7d48aa7b7961) ![restart-unlock-ready.png](https://git.kayg.org/attachments/5c7e95f8-985d-4b4e-8a0a-eebc372c8adc)
kayg commented 2026-10-16 03:02:32 +00:00
Author
Owner
Copy link

Service fixture investigation: its own unit journal confirms the old infraternal agent started and reached Preparing the enrolled host connection; initial failure was later in start-old adapter verification, not an observed release reconnect failure. Adapter originally checked MainPID/executable immediately after default Type=simple start. It now requests Type=exec and bounds actual /proc image-digest readiness to5s; it still refuses unless exact old5b17532e/new23fc5e92 bytes match. Safe operation/error/image-attempt receipts added; no shipping change. Diagnostic retry remains queued, no long-stall verdict yet.

VM owner currently process-per-user / ppu-soak-r8-dev, owner timestamp2026-10-16T02:48:01.027721Z. Current retry and fast-gate calibration waiting since~02:49UTC. Signed staging and12+5 browser acceptance already PASS; production captures2/2 PASS. Full fast has not yet reached Rust checks because calibration is queued. No deploy or owner-fixture action.

Service fixture investigation: its own unit journal confirms the old infraternal agent started and reached Preparing the enrolled host connection; initial failure was later in start-old adapter verification, not an observed release reconnect failure. Adapter originally checked MainPID/executable immediately after default Type=simple start. It now requests Type=exec and bounds actual /proc image-digest readiness to5s; it still refuses unless exact old5b17532e/new23fc5e92 bytes match. Safe operation/error/image-attempt receipts added; no shipping change. Diagnostic retry remains queued, no long-stall verdict yet. VM owner currently process-per-user / ppu-soak-r8-dev, owner timestamp2026-10-16T02:48:01.027721Z. Current retry and fast-gate calibration waiting since~02:49UTC. Signed staging and12+5 browser acceptance already PASS; production captures2/2 PASS. Full fast has not yet reached Rust checks because calibration is queued. No deploy or owner-fixture action.
kayg commented 2026-10-16 03:18:23 +00:00
Author
Owner
Copy link

Service upgrade step PASS on perf VM (load_before0.40/0.19/0.55): old036bc2bb0 rebuild5b17532e, PID403336 -> exact new23fc5e92, PID403348, via one service restart. Adapter verified actual /proc executable digests. Service-created PTY reattached with fresh UPGRADE_OK echo; every pre-upgrade shim PID survived. Restart-command to retained-session echo456627us (single trial, includes unit/image verification; not a network-switch percentile claim).

Fixture readiness repair: request Type=exec and bound exact digest readiness rather than immediately sampling a simple-unit launcher; no release source or acceptance predicate relaxed. Initial invalid setup failure remains recorded.

Current long case started~03:17:15UTC: real new staging server SIGSTOP, TCP still accepts,600s pause; then120s bidirectional TCP discard, abrupt stop/cold replacement and automatic reconnect. New infraternal agent service PID must stay403348 and retained shims/session must echo again. No edits to active runner. Long-case result pending. Final pristine492e1505a fast calibration completed on VM; host Rust checks compiling/running. Signed staging and12+5 browsers already PASS. No deploy.

Service upgrade step PASS on perf VM (load_before0.40/0.19/0.55): old036bc2bb0 rebuild5b17532e, PID403336 -> exact new23fc5e92, PID403348, via one service restart. Adapter verified actual /proc executable digests. Service-created PTY reattached with fresh UPGRADE_OK echo; every pre-upgrade shim PID survived. Restart-command to retained-session echo456627us (single trial, includes unit/image verification; not a network-switch percentile claim). Fixture readiness repair: request Type=exec and bound exact digest readiness rather than immediately sampling a simple-unit launcher; no release source or acceptance predicate relaxed. Initial invalid setup failure remains recorded. Current long case started~03:17:15UTC: real new staging server SIGSTOP, TCP still accepts,600s pause; then120s bidirectional TCP discard, abrupt stop/cold replacement and automatic reconnect. New infraternal agent service PID must stay403348 and retained shims/session must echo again. No edits to active runner. Long-case result pending. Final pristine492e1505a fast calibration completed on VM; host Rust checks compiling/running. Signed staging and12+5 browsers already PASS. No deploy.
kayg commented 2026-10-16 03:37:18 +00:00
Author
Owner
Copy link

Long-stall recovery assertions completed on perf VM:600s SIGSTOP while TCP accepts +120s bidirectional TCP discard/abrupt stop, then cold exact23fc5e92 server. Original new infraternal agent PID403348 stayed unchanged; all3 tracked prior shims survived; same service-created session produced STALL_OK. Listener-ready to retained echo21084919us; total stall-to-recovery741199221us. This is one trial, not a network-switch percentile.

Overall fixture still FAILED its final negative control, so this is NOT the clean acceptance run: remote_cold_tests.rs:80 preflight used request.mode Rootless even though the rootful fixture invitation is Rootful. The API403 was therefore not a valid locked-CA negative control. Correcting only test helper to request invitation.mode, keeping every existing refusal/preflight assertion, then repeating the full600+120s case. Source remains pristine492e1505a; correction is in external fixture snapshot. Focused test-crate rebuild uses exact retained dependency fingerprints, separate output under the one target directory, bounded to1CPU; it does not switch Cargo workspaces/cache while the full gate runs. No release binary changed.
Full fast is in workspace Clippy with no reported failure so far. Signed128/128 exact-image handoff and browser12+5 remain PASS. No deploy.

Long-stall recovery assertions completed on perf VM:600s SIGSTOP while TCP accepts +120s bidirectional TCP discard/abrupt stop, then cold exact23fc5e92 server. Original new infraternal agent PID403348 stayed unchanged; all3 tracked prior shims survived; same service-created session produced STALL_OK. Listener-ready to retained echo21084919us; total stall-to-recovery741199221us. This is one trial, not a network-switch percentile. Overall fixture still FAILED its final negative control, so this is NOT the clean acceptance run: remote_cold_tests.rs:80 preflight used request.mode Rootless even though the rootful fixture invitation is Rootful. The API403 was therefore not a valid locked-CA negative control. Correcting only test helper to request invitation.mode, keeping every existing refusal/preflight assertion, then repeating the full600+120s case. Source remains pristine492e1505a; correction is in external fixture snapshot. Focused test-crate rebuild uses exact retained dependency fingerprints, separate output under the one target directory, bounded to1CPU; it does not switch Cargo workspaces/cache while the full gate runs. No release binary changed. Full fast is in workspace Clippy with no reported failure so far. Signed128/128 exact-image handoff and browser12+5 remain PASS. No deploy.
kayg commented 2026-10-16 03:53:58 +00:00
Author
Owner
Copy link

Clean final service proof PASS:1 passed,0 failed, same full600s stopped-server +120s bidirectional TCP discard/abrupt-stop/cold-replacement case, all assertions retained. Perf VM4cores/8069112KiB; load0.08/0.72/0.95 ->0.00/0.05/0.41.

  • Full production-source036bc2bb0 rebuild5b17532e PID410944 -> frozen new23fc5e92 PID410957 through one service restart. Actual image digests checked, existing enrolled identity retained; same service-created PTY echoes after restart; every prior shim survives. Restart-to-retained echo349793us.
  • New exact23fc server stopped600s while TCP still accepts; discard both TCP directions120s before restoring. Agent PID410957 unchanged throughout;3 tracked prior shims and retained session survive. Cold-listener-ready to fresh retained-session echo21628511us; total stall-start to retained echo741702374us. One trial, not a percentile/network-switch claim.
  • Replacement CA locked; fresh enrollment403, token still valid/preflight retained; corrected request uses invitation.mode. Cleanup asserts0 shims. Own transient service inactive/MainPID0, owned executable processes0, VM service directory removed.

Earlier failed setup and invalid postcheck trials are excluded and attached alongside the passing run. Only external fixture helper changed (exec readiness and correct rootful invitation mode); no shipping code/binary changed. Exact signed handoff128/128, standard signed acceptance, browser12/12 + mocks5/5 + captures2/2 all PASS. Final fast gate still running workspace Clippy; native C dependency build has completed. No deploy.
service-proof-final.log
service-proof.log
service-proof-retry.log
service-cleanup.json
service-final-sources.txt
mode-fixture-runtime.json

Clean final service proof PASS:1 passed,0 failed, same full600s stopped-server +120s bidirectional TCP discard/abrupt-stop/cold-replacement case, all assertions retained. Perf VM4cores/8069112KiB; load0.08/0.72/0.95 ->0.00/0.05/0.41. - Full production-source036bc2bb0 rebuild5b17532e PID410944 -> frozen new23fc5e92 PID410957 through one service restart. Actual image digests checked, existing enrolled identity retained; same service-created PTY echoes after restart; every prior shim survives. Restart-to-retained echo349793us. - New exact23fc server stopped600s while TCP still accepts; discard both TCP directions120s before restoring. Agent PID410957 unchanged throughout;3 tracked prior shims and retained session survive. Cold-listener-ready to fresh retained-session echo21628511us; total stall-start to retained echo741702374us. One trial, not a percentile/network-switch claim. - Replacement CA locked; fresh enrollment403, token still valid/preflight retained; corrected request uses invitation.mode. Cleanup asserts0 shims. Own transient service inactive/MainPID0, owned executable processes0, VM service directory removed. Earlier failed setup and invalid postcheck trials are excluded and attached alongside the passing run. Only external fixture helper changed (exec readiness and correct rootful invitation mode); no shipping code/binary changed. Exact signed handoff128/128, standard signed acceptance, browser12/12 + mocks5/5 + captures2/2 all PASS. Final fast gate still running workspace Clippy; native C dependency build has completed. No deploy. [service-proof-final.log](https://git.kayg.org/attachments/00a97704-5298-4428-9e6c-c1d3b1d9c7b8) [service-proof.log](https://git.kayg.org/attachments/b74ecba8-0cfb-440f-992e-438a077980bd) [service-proof-retry.log](https://git.kayg.org/attachments/57dae3fa-8aed-4cf5-b187-1fcd882458d0) [service-cleanup.json](https://git.kayg.org/attachments/f10b8ca1-fe67-4c45-8632-10aeb97c9aab) [service-final-sources.txt](https://git.kayg.org/attachments/15ff6f56-9083-46a9-ac4b-d49e9d95006e) [mode-fixture-runtime.json](https://git.kayg.org/attachments/bcc9c2ea-8796-4d72-ae2d-0ebdd4acb119)
kayg commented 2026-10-16 04:02:28 +00:00
Author
Owner
Copy link

Gate finding: the first full gate on 492e1505a reached static builds but one Rust test failed with path must be shorter than SUN_LEN: crates/infraternal-agent/src/daemon_gone_tests.rs:111. TMPDIR=/home/kayg/.cache/codex-scratch/r225 produced a 109-character session socket path. The final rerun uses the job-owned /home/kayg/.cache/codex-scratch/r (106-character path), without source changes. The failed run is retained as verify-fast.log; final run will be verify-fast-final.log.

Gate finding: the first full gate on 492e1505a reached static builds but one Rust test failed with `path must be shorter than SUN_LEN`: crates/infraternal-agent/src/daemon_gone_tests.rs:111. TMPDIR=/home/kayg/.cache/codex-scratch/r225 produced a 109-character session socket path. The final rerun uses the job-owned /home/kayg/.cache/codex-scratch/r (106-character path), without source changes. The failed run is retained as verify-fast.log; final run will be verify-fast-final.log.
kayg commented 2026-10-16 04:20:37 +00:00
Author
Owner
Copy link

Staging evidence scope clarification: the existing standard harness reattaches the six browser connections after handoff/rollback, then verifies fresh PTY echoes and Ping/Pong on each of those retained sockets after a five-second idle sample (crates/infraternal-server/src/auth/deploy_staging_tests.rs:299, :378). It clears the initial connections at the first handoff (:276). Both standard and exact-image runs pass that acceptance; this is not proof that pre-handoff TCP sockets remain open across image replacement. The 128 original shim PIDs and sessions do survive the handoffs. Completion wording will state this scope.

Staging evidence scope clarification: the existing standard harness reattaches the six browser connections after handoff/rollback, then verifies fresh PTY echoes and Ping/Pong on each of those retained sockets after a five-second idle sample (crates/infraternal-server/src/auth/deploy_staging_tests.rs:299, :378). It clears the initial connections at the first handoff (:276). Both standard and exact-image runs pass that acceptance; this is not proof that pre-handoff TCP sockets remain open across image replacement. The 128 original shim PIDs and sessions do survive the handoffs. Completion wording will state this scope.
kayg commented 2026-10-16 04:36:47 +00:00
Author
Owner
Copy link

Finished on unchanged reconnect-release head 492e1505ab; production 036bc2bb06 is an ancestor. No commits, push or production deployment.

Retained release artifacts:

  • /home/kayg/.cache/codex-scratch/reconnect-release/artifacts/infraternal-x86_64-musl — 22,299,504 B; SHA-256 23fc5e92df83e49da2d351cc19fbf24983475eebc2ce98c961e03bd29254b2c6.
  • /home/kayg/.cache/codex-scratch/reconnect-release/artifacts/infraternal-aarch64-musl — 19,726,904 B; SHA-256 f47a8b8396034686e91cc0ae862dc73a21be086a522ab925135669bf53b9806a.
  • /home/kayg/.cache/codex-scratch/reconnect-release/artifacts/web-dist.tar — 3,543,040 B; SHA-256 e783405b15bd91bcd8b3696f1d2c55564df259ff5ad04525219a941ce5be1a85.

Web tree: /home/kayg/.cache/codex-scratch/reconnect-release/artifacts/web/ (253 files, 3,333,749 B); web-SHA256SUMS SHA-256 53f513134ee07401c239c2e7809f64a55302877c06c5b07566e01db495141524. /home/kayg/.cache/codex-scratch/reconnect-release/artifacts/SHA256SUMS covers the release files. Final hashes verified.

Compared with recorded production static sizes: x86 −29,600 B; ARM −26,016 B. Web is not byte-identical: build.ts embeds the new commit and date despite unchanged web/UI/protocol source. Original production artifacts/tree are absent; recorded old web tree 3,333,781 B. No complete per-file comparison claimed.

Acceptance:

  • PASS standard signed staging: 12 sampled handoffs, 128/128 real PTYs and echoes, 128 retained shims, CA unlocked after one real passkey ceremony, six retained responsive sockets after reattachment and five seconds idle; unusable-image refusal, warm rollback and 10,000-node/50-user backup integrity pass.
  • PASS supplemental signed handoff into exact frozen x86, actual ARM and frozen web artifacts above: 128 shims/echoes, unlocked CA, six responsive sockets after reattachment and five seconds idle. Temporary authority only in test sender images; exact release image retains production-only trust. Exact phase is one-way; standard proof separately covers rollback.
  • PASS real restart browser cases 12/12 (2 ordinary, 4 liveness, 6 three-window), mocks 5/5, no retries. Two additional production-build capture cases pass; screenshots attached; no physical-phone or author aesthetic claim.
  • PASS #225 single service restart: enrolled old-source infraternal agent into exact new x86; service-created PTY and all three tracked shims survive. Fresh retained echo 349,793 us (0.349793 s); PID 410944 → 410957, executable digests verified.
  • PASS automatic reconnect: server SIGSTOP 600 s, TCP discard 120 s, abrupt stop/cold replacement. Same new service PID and three prior shims retained. Listener-to-echo 21,628,511 us (21.628511 s); stall-start-to-echo 741,702,374 us. Replacement CA locked, enrollment refused, invitation retained, CA private key not persisted.

Measurements: infraternal-perf-test, 4 CPUs, load before 0.08/0.72/0.95 and after 0.00/0.05/0.41; shared VM lock, CPUWeight=10000. One reconnect trial, not a percentile or a switch-budget pass. Standard listener-gap brackets p50 102.847 ms/p99 306.183 ms; exact bracket 102.770448 ms, all polling bounds.

Limits: old-service binary is an exact 036bc2bb0 source rebuild, 22,329,104 B, SHA-256 5b17532eb0f9b716ef7893640b3740bc0649c456ac5eaccbd9c00b83891617b5; original production 29d65925… absent, not claimed byte-identical. ARM checked and included in staging, not executed natively. Excluded setup/mode-mismatched trials retained; corrected test-only fixture retains refusal assertions.

Final scripts/verify fast on pristine 492e1505a: exit 1, only inherited absolute size caps fail. Verbatim output summary and size diagnostics:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+
PT_INTERP: absent; size: 22299504 bytes; cap: 22204304 bytes; delta: +95200 bytes
binary exceeds the 22204304-byte size budget: 22299504 bytes
PT_INTERP: absent; size: 19726904 bytes; cap: 19638744 bytes; delta: +88160 bytes
binary exceeds the 19638744-byte size budget: 19726904 bytes
PT_INTERP: absent; size: 22299504 bytes; cap: 22204304 bytes; delta: +95200 bytes
binary exceeds the 22204304-byte size budget: 22299504 bytes

Rust workspace: 1,075 passed, 0 failed, 141 ignored; compile-out gate adds 25 passing tests. TypeScript separately PASS: checksums/compiler, 191 tracer/UI unit tests, build/transfer budgets, stream oracle and 37 protocol tests (dispatcher stops after inherited Rust size failure). Full logs attached verbatim. First full run's sole Rust failure was a 109-character Unix socket path; shortened 106-character scratch path passes focused recheck and final full gate without source changes.

Release artifacts use INFRATERNAL_RELEASE_BUILD=1. Ordinary fast-gate static files use marker 0 and have different hashes with identical sizes; only frozen marker-1 files above were used for exact staging/browser/service proofs. See gate-artifact-identity.json.

Author checklist completed. Private VM units/scopes inactive, owned executable processes zero, all private staging/browser/capture/service directories removed. Local temporary sources, upload archives and Cargo target removed after gate; final integrity/cleanup receipt attached. Checkout clean/head unchanged. Production and owner fixtures untouched; issue remains open.

Receipts:
web-comparison.json
browser-summary.json
staging-proof.txt
service-proof-final.log
service-final-sources.txt
verify-fast-final.log
verify-fast.log
verify-typescript.log
socket-path-recheck.log
gate-artifact-identity.json
completion-receipt.json
author-review.txt
local-cleanup-phase1.json
browser-scopes-cleanup.json
vm-final-cleanup.json
release-SHA256SUMS.txt

Finished on unchanged reconnect-release head **492e1505ab6822078268344382f4d246d2c3d65e**; production 036bc2bb067a231736f219f32897aaa7d8942b70 is an ancestor. No commits, push or production deployment. Retained release artifacts: - `/home/kayg/.cache/codex-scratch/reconnect-release/artifacts/infraternal-x86_64-musl` — 22,299,504 B; SHA-256 `23fc5e92df83e49da2d351cc19fbf24983475eebc2ce98c961e03bd29254b2c6`. - `/home/kayg/.cache/codex-scratch/reconnect-release/artifacts/infraternal-aarch64-musl` — 19,726,904 B; SHA-256 `f47a8b8396034686e91cc0ae862dc73a21be086a522ab925135669bf53b9806a`. - `/home/kayg/.cache/codex-scratch/reconnect-release/artifacts/web-dist.tar` — 3,543,040 B; SHA-256 `e783405b15bd91bcd8b3696f1d2c55564df259ff5ad04525219a941ce5be1a85`. Web tree: `/home/kayg/.cache/codex-scratch/reconnect-release/artifacts/web/` (253 files, 3,333,749 B); `web-SHA256SUMS` SHA-256 `53f513134ee07401c239c2e7809f64a55302877c06c5b07566e01db495141524`. `/home/kayg/.cache/codex-scratch/reconnect-release/artifacts/SHA256SUMS` covers the release files. Final hashes verified. Compared with recorded production static sizes: x86 −29,600 B; ARM −26,016 B. Web is not byte-identical: build.ts embeds the new commit and date despite unchanged web/UI/protocol source. Original production artifacts/tree are absent; recorded old web tree 3,333,781 B. No complete per-file comparison claimed. Acceptance: - PASS standard signed staging: 12 sampled handoffs, 128/128 real PTYs and echoes, 128 retained shims, CA unlocked after one real passkey ceremony, six retained responsive sockets after reattachment and five seconds idle; unusable-image refusal, warm rollback and 10,000-node/50-user backup integrity pass. - PASS supplemental signed handoff into exact frozen x86, actual ARM and frozen web artifacts above: 128 shims/echoes, unlocked CA, six responsive sockets after reattachment and five seconds idle. Temporary authority only in test sender images; exact release image retains production-only trust. Exact phase is one-way; standard proof separately covers rollback. - PASS real restart browser cases 12/12 (2 ordinary, 4 liveness, 6 three-window), mocks 5/5, no retries. Two additional production-build capture cases pass; screenshots attached; no physical-phone or author aesthetic claim. - PASS #225 single service restart: enrolled old-source infraternal agent into exact new x86; service-created PTY and all three tracked shims survive. Fresh retained echo **349,793 us (0.349793 s)**; PID 410944 → 410957, executable digests verified. - PASS automatic reconnect: server SIGSTOP **600 s**, TCP discard **120 s**, abrupt stop/cold replacement. Same new service PID and three prior shims retained. Listener-to-echo **21,628,511 us (21.628511 s)**; stall-start-to-echo **741,702,374 us**. Replacement CA locked, enrollment refused, invitation retained, CA private key not persisted. Measurements: infraternal-perf-test, 4 CPUs, load before 0.08/0.72/0.95 and after 0.00/0.05/0.41; shared VM lock, CPUWeight=10000. One reconnect trial, not a percentile or a switch-budget pass. Standard listener-gap brackets p50 102.847 ms/p99 306.183 ms; exact bracket 102.770448 ms, all polling bounds. Limits: old-service binary is an exact **036bc2bb0 source rebuild**, 22,329,104 B, SHA-256 `5b17532eb0f9b716ef7893640b3740bc0649c456ac5eaccbd9c00b83891617b5`; original production `29d65925…` absent, not claimed byte-identical. ARM checked and included in staging, not executed natively. Excluded setup/mode-mismatched trials retained; corrected test-only fixture retains refusal assertions. Final `scripts/verify fast` on pristine 492e1505a: exit 1, **only inherited absolute size caps fail**. Verbatim output summary and size diagnostics: ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ PT_INTERP: absent; size: 22299504 bytes; cap: 22204304 bytes; delta: +95200 bytes binary exceeds the 22204304-byte size budget: 22299504 bytes PT_INTERP: absent; size: 19726904 bytes; cap: 19638744 bytes; delta: +88160 bytes binary exceeds the 19638744-byte size budget: 19726904 bytes PT_INTERP: absent; size: 22299504 bytes; cap: 22204304 bytes; delta: +95200 bytes binary exceeds the 22204304-byte size budget: 22299504 bytes ``` Rust workspace: 1,075 passed, 0 failed, 141 ignored; compile-out gate adds 25 passing tests. TypeScript separately PASS: checksums/compiler, 191 tracer/UI unit tests, build/transfer budgets, stream oracle and 37 protocol tests (dispatcher stops after inherited Rust size failure). Full logs attached verbatim. First full run's sole Rust failure was a 109-character Unix socket path; shortened 106-character scratch path passes focused recheck and final full gate without source changes. Release artifacts use INFRATERNAL_RELEASE_BUILD=1. Ordinary fast-gate static files use marker 0 and have different hashes with identical sizes; only frozen marker-1 files above were used for exact staging/browser/service proofs. See gate-artifact-identity.json. Author checklist completed. Private VM units/scopes inactive, owned executable processes zero, all private staging/browser/capture/service directories removed. Local temporary sources, upload archives and Cargo target removed after gate; final integrity/cleanup receipt attached. Checkout clean/head unchanged. Production and owner fixtures untouched; issue remains open. Receipts: [web-comparison.json](https://git.kayg.org/attachments/fd2ff377-0f77-477c-9b6b-738379a651de) [browser-summary.json](https://git.kayg.org/attachments/cd8c1ae9-e04d-4098-ac29-4573497a23d8) [staging-proof.txt](https://git.kayg.org/attachments/280ec54b-29fd-461e-a753-99c778beb808) [service-proof-final.log](https://git.kayg.org/attachments/00a97704-5298-4428-9e6c-c1d3b1d9c7b8) [service-final-sources.txt](https://git.kayg.org/attachments/15ff6f56-9083-46a9-ac4b-d49e9d95006e) [verify-fast-final.log](https://git.kayg.org/attachments/472302f2-d986-4020-bb7e-b3c737060291) [verify-fast.log](https://git.kayg.org/attachments/55e127c4-9b6e-4906-934e-4df606aa9032) [verify-typescript.log](https://git.kayg.org/attachments/30bbc7cb-ba70-4327-bb03-bca847e23b6b) [socket-path-recheck.log](https://git.kayg.org/attachments/5d698568-0808-4427-a67f-6acce2d4629e) [gate-artifact-identity.json](https://git.kayg.org/attachments/ae194e0a-db4f-4be1-b9ae-359d7636ff99) [completion-receipt.json](https://git.kayg.org/attachments/788b70b4-9d48-4214-a480-864116f9c3e4) [author-review.txt](https://git.kayg.org/attachments/03bd32f9-7f71-41aa-8b9f-ac01c44b0a4e) [local-cleanup-phase1.json](https://git.kayg.org/attachments/c1feacd1-430c-4bce-9be6-c0214c399c30) [browser-scopes-cleanup.json](https://git.kayg.org/attachments/977b7263-a0f9-4bfe-a497-a39f6bb6969b) [vm-final-cleanup.json](https://git.kayg.org/attachments/fddfd368-1395-4358-a929-c6d36c0612e5) [release-SHA256SUMS.txt](https://git.kayg.org/attachments/5e5dfc09-f8c9-4d51-bd43-78788c1c2630)
kayg commented 2026-10-16 05:31:24 +00:00
Author
Owner
Copy link

#225 field evidence (post when Forgejo is back) — 2026-10-16 06:38–06:56 CEST

  • Prod deployed 492e1505a 06:37:55 CEST (x86 23fc5e92…), release-prod-live updated. CA locked by the restart (owner asked to unlock).
  • The 3 "stuck" DB hosts = one rootless agent on infraternal-dev (identity host_id 0bc79da5…, endpoint 10.69.69.50:7443). Old binary bin/live 20,333,104 B.
  • Restarted once 06:38:11 (owner OK). It dialed once at ~06:39:11 (TIME-WAIT seen), connection closed at once, no server log line. Then 0 SYN/FIN/RST to :7443 in a 15-min tcpdump; single thread in epoll_wait (~12 s timeouts), inotify on sessions dir only.
  • => Old agent tries once after start and never retries: the field bug, independent of the TCP-keepalive case. Unknown: why the one attempt closed (CA locked? cert?). Next: after the owner unlocks the CA, restart once more and see if it connects; it then needs the signed update to the fixed binary.
## #225 field evidence (post when Forgejo is back) — 2026-10-16 06:38–06:56 CEST - Prod deployed 492e1505a 06:37:55 CEST (x86 23fc5e92…), release-prod-live updated. CA locked by the restart (owner asked to unlock). - The 3 "stuck" DB hosts = one rootless agent on infraternal-dev (identity host_id 0bc79da5…, endpoint 10.69.69.50:7443). Old binary bin/live 20,333,104 B. - Restarted once 06:38:11 (owner OK). It dialed once at ~06:39:11 (TIME-WAIT seen), connection closed at once, no server log line. Then 0 SYN/FIN/RST to :7443 in a 15-min tcpdump; single thread in epoll_wait (~12 s timeouts), inotify on sessions dir only. - => Old agent tries once after start and never retries: the field bug, independent of the TCP-keepalive case. Unknown: why the one attempt closed (CA locked? cert?). Next: after the owner unlocks the CA, restart once more and see if it connects; it then needs the signed update to the fixed binary.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
j470b-server-mesh-rejoin
mesh-compat-382
j444b-d381-key-wins-flood
j461b-macos-utun-reader-thread
j475b-remove-host-motion
fix-reconnect-new-link
j481-d380
dev
j479-mesh-soak
fix/j476-host-facts
j475-motion-pill-split
j470-mac-node-join
root-upgrade-d376
j474-review-merge
latency-targets-120hz
nightly-suite-shards
gotatun-pin-j469
macos-utun-reader-thread
fix-flood-echo-stall
wip/j416
motion-fast-ease-out
mesh-addhost-j259
takeover-close-terminals-d368
kbd-instant-followups
fix-trust-key-double-click
nightly-suite-partial-report
appwin-icons-j207
rusqlite-slice1
swipe-content-first-d374
mesh-mtu-381
addhost-instant-open
glass-contrast-d365-mock
glass-contrast-d365
signout-followups-442
fix-startup-toast-red
research-gotatun-matrix-j317
themes-sync-j304
fix-load-flakes
fix-nix-zig-patch
fix-compute01-browser-baseline
ratchet-size-baseline-419
predict-auto-d130
revoke-coverage-d367
nightly-suite-queue-timeout
nightly-browser-suite
fix-rustsec-2026-0459
fonts-ioskeley-obel
log-retention-d375
fix-kbd-no-anim-427
fix-dev-red-specs-2
takeover-proof-305-j205
wip/j410-server-pump
ui-consistency-d372
fix-osc52-clipboard
libghostty-vt-1.0
forget-retired-host-d369
j404-merge-426
fix-setup-lock-race
fix-touch-select-426
revoke-access-report-ui
fix-predictive-echo-specs
fix-selinux-agent-exec
wip/j395-checkpoint2
wip/j395
j391-review-scratch
ui-consistency-j195-deferred-rebased
fix-rustc-paths-421
decisions-wording-341
fix-install-suite-red
fix-dev-rtt-specs
fix-pending-removal-align
fix-privileged-skip
fix-install-suite-404
review-j368
mesh-dns-j254
revoke-access-report-d356b
revoke-privilege-round5
fix-removal-confirm
review-j360
gotatun-rotation-367
deps-refresh-340
review-j354
review-J351
review-j350
addhost-takeover-d359
revoke-report-items-347
revoke-end-sessions-d362
review-j348
size-mesh-ssh-346
size-trim-337
revoke-access-diff-d356
j259-pre-rebase-b06c5a6
j305-ended-at-start-spec
addhost-sudo-consent
fix-layout-real-specs-411
fix-shell-open-411
fix-console-hosts-373
fix-shell-exit
fuzz-host-name-353
fix-host-details-410
macos-filter-j197
research-gotatun-tricks
revoke-p3-j216l
stale-state-pill
revoke-followups-j216k
revoke-progress-j216e
backup/macos01-20261106/lume/j197
backup/macos01-20261106/lume/appwin-icons-j207-stash0
backup/macos01-20261106/claude/stash-0
backup/macos01-20261106/claude/stash-1
backup/macos01-20261106/claude/J80-macos-s1
backup/macos01-20261106/claude/J80
backup/macos01-20261106/claude/J80-slice1-20261102
backup/macos01-20261106/claude/J16b-appwin-on-dev
backup/macos01-20261106/claude/appwin-launch-idle-146
backup/macos01-20261106/claude/J80-s1-run
wip/j259
trust-285-p3c
runtime-lock-j216f
mesh-ssh-p3-j175
tun-proof-p3b
trust-285-p3
terminal-polish-j271
tun-proof-p3
ssh-install-flakes-j216i
fuzz-fixes-j264
trust-285-a
trust-285-b
restart-oversized-fixture-j198
backup/macos01-20261106/lume/appwin-icons-j207
restart-bound-j216h
up-preflight-p3-j198
backup/macos01-20261106/lume/j251
up-preflight-j198
wip/up-preflight-j198-j245-p2
wip/up-preflight-j198-j245-p1
agent-fsync-offloop-j196
wip/up-preflight-j198-j245
settings-offscreen-j231
wip/up-preflight-j198-j226-inspection
wip/j103f
themes-j206
wip/up-preflight-j198-j226
flakes-j216c
backup/macos01-20261106/claude/j197
flakes-j216b
backup/macos01-20261106/claude/j197-red
wip/up-preflight-j198-review
flakes-j216
backup/macos01-20261106/claude/appwin-icons-j207
engineering-j208b
appwin-mock-j209
toast-stack-j165
engineering-j208
decisions-d351
decisions-d350
decisions-d349
appearance-presets-mock-j129
decisions-d348
docs-audit-spec
doc-coverage-baseline-raise
decisions-d347
decisions-d346b
decisions-d346
j88h-profile
ui-consistency-j195-deferred
ui-consistency-j195
engineering-practices-j194
preferred-login-ui-j166
signin-pools-263-j174
macos-node-265-s1
decisions-d345
decisions-d344
proxy-cgroup-j164
verify-quick-fuzz-329
proxy-start-io-j168
timing-flakes-j163
up-263-s1
mcp-events-j147
verify-quick-fuzz-j158
proxy-p3-j159
agent-resource-j157-detect
wip/j103d-final
remote-cancel-hang-j155
proxy-traefik-281-j62d
j149-verify-quick
proxy-traefik-281-j62b
j150-drop-unnamed
wip/j79d-red-r4
wip/j103d-d332
rootless-toast-j142
wip/j103d-head
wip/j103d
j125c-mcp
macos-build-j136
backup/macos01-20261106/claude/j136
wip/j79d-red-cli
wip/j79d-red-server
j121-link-liveness
j118c-tracer-failures
wip/j79c-head
release-desktop-j131
j124c-signed-macos
wip/j79c-red-p3
wip/j79c-red-p23
wip/j79c-red-p22
merge-j128-check
perf-deferred-j128
sidebar-toggle-260
j127-shortcut-changed-dot
j125-mcp
j118-tracer-failures
j126-account-summary
settings-sections-mock-j92
j120-pinch-lock
sidebar-292
backup/macos01-20261106/claude/j120
taste-2026-11-03
row-actions-242-215
backup/macos01-20261106/claude/stash-0-untracked
backup/macos01-20261106/claude/J80-265-s1-run
backup/macos01-20261106/claude/J80-macos-s1-20261102
j112-canvas-commands
ui-no-overshoot-scroll
fix-recovery-restart
backup/macos01-20261106/claude/stash-1-untracked
taste-2026-11-02
size-baseline-ratchet
machine-cards-structure-j108
j109-boot-copy-mockups
fix-dev-red
wip/up-263-s1
j95b-settings-rows
proxy-traefik-281
j95-option-rows
docs-human-j91
docs-keys-palette
fix-flood-1013-270-j38d
mesh-ssh-s1
mesh-ssh-s0
addhost-d291-tunnel
test-stability-load
mesh-relay-220
repro-296
shim-thp-off-24
mesh-pins-258-v2
mesh-dns
signin-d311-merged
docs-pages
ui-size-283
fix-ui-batch-specs
wip/j12b-edge-update
gate-on-arm
fix-rename-sync-293
trust-285-j60b
fix-focus-ring-clip
integ-ui-1101
edge-update-v2
signin-d311
taste-2026-11-01
layout-per-tab-250
ui-approved-splitopt-224
ui-approved-splitopt-224-on-dev
icon-text-centre-282
ui-approved-189-212-216
research-reverse-proxy
transfer-budget-320
review-mesh-pins-258-r2
sidebar-pill-259
review-mesh-relay-220
wip/j12-edge-update-v2
nix-package
status-crash-loop-237
fix-dev-test-failures
fix-fixture-socket-path
wip/mesh-relay-220-pre-dev-advance-rebase
fix-canvas-side-clip
research-head-to-head
frosted-real-256
review-mesh-pins-258
wip/mesh-relay-220-pre-final-rebase
review-layout-per-tab-250-r2
broad-suite-triage
nix-devshell
review-status-crash-loop-237-r3
wip/nix-devshell-pre-p3
taste-frosted-real
wip/code-serve-demo-2026-10-30
wip/code-serve-glass-2026-10-30
wip/code-serve-clusters-2026-10-30
wip/code-merge-release94-dev-2026-10-30
wip/rproxy-delta-scratch-2026-10-30
wip/stale-note-257-staged-2026-10-30
wip/d241-size-staged-2026-10-30
review-broad-suite-207
wip/mesh-pins-258-pre-split
wip/mesh-relay-220-pre-rebase
mesh-measure-220
plan-211-update
review-edge-autoupdate-r3
perf24-reconnect
review-stale-note-r2
release-prod-live
d241-shell-marker
review-mesh-pins-server-258
review-revocation-r3
review-setup-token-r2
review-sync-journal-r3
wip/review-mesh-pins-258-prune-review-1995fee46
frosted-scroll-256
sync-journal-r3
revocation-per-account-wip
review-tokio-asyncfd-0386
review-sync-p3s
wip/stash-8by64-stash0-2026-10-30
mesh-drag-r2
setup-token-data-wip
release-9-4
auth-busy-1013
mesh-view-overlap
sync-p3s
review-auth-busy-wake
review-readd-r8-wip
mesh-drag-bounds
audit-auth-93
app-windows-stream-start
review-authority-cap-r3
app-windows-r7-measure
readd-minimal-r6
readd-minimal-r5
app-windows-r7-soak
readd-minimal-r4
app-windows-r7
app-windows-r6
readd-minimal-r3
readd-minimal-r2
readd-minimal
merge-release8-dev
review-busy-page-liveness-r2
review-busy-page-liveness
mesh-view-trim
review-authority-read-retry-r2
review-remove-uninstall-r2
release-retry-reattach
release-retry
review-remove-uninstall
review-authority-read-retry
app-windows-ship
mesh-autoswitch-p2
mesh-ssh
mesh-relay
mesh-macos-node
screen-and-control
session-labels
review-real-logins-r5
review-real-logins-r4
review-real-logins-r3
shim-memory-r2
review-real-logins-r2
shim-memory
shadow-addhost-instant
shadow-real-logins
shadow-same-session-flood
shadow-prod-perf-fix
hosts-from-server
app-windows-real
app-windows-transport
release-prod-6w
mesh-policy-verify
quiet-close-toast
dev-merge-r4
review-mesh-agent-transport
review-app-windows-transport
settings-roundtrip
review-release-pipeline-r4
mesh-agent-transport
ended-strip-fix
review-audit-ca-update-d244
settings-rows
phone-width
signout-clear
fix-restart-tests
dev-merge-rp
signer-hardening
review-audit-ca-update
terminal-fidelity
review-audit-ca-update-wip
mesh-integrate
review-mesh-agent-transport-515d3712
review-mesh-dataplane-feec1988
mesh-dataplane
signout
research-canvas-node
wip/review-renderer-cost
wip/review-terminal-fidelity
padlock-glyph
review-release-pipeline-2e95b689
phone-polish
sidebar-polish
inbox-generic
review-mesh-dataplane-378d4055
prod-batch
ui-inbox
ui-ca-unlock
ui-machine-sections
review-mesh-dataplane-a5f744fb
tls-session-prod
session-bounds-prod
fix-dead-session
session-bounds
tls-session
palette-polish
shortcuts-d222
research-appmeta
phone-gutter
ui-centre-button
wip/ui-centre-button
wip/tls-session
review-addhost-breakit
ui-platform
renderer-cost
remote-agents
mesh-api
ssh-bootstrap
prod-hotfix2
prod-fixes
review-remote-agents
review-mesh-dataplane
session-life
research-browser
review-mesh-api
research-capture
review-mesh-api-round3
review-mesh-api-round1
gate-stability
review-gate-stability
review-relay-direct
review-mesh-api-findings
proto-sessions
font-picker
proto-addhost
proto-windows
mesh-core
latency-gate
research-bsd
research-discovery
keytray-mods
mock-core
ui-tweaks
fix-motion-sel
fix-motion
integrate2
fix-robust
t-web
t-server
vt-ghostty
fix-snap-echo
t-web-auth
review-vt2
t-agent
lane-protocol
lane-shim
lane-auth
s1-base
s1-contracts
s1-workspace
No results found.
Labels
Clear labels   
launch-qa
Findings from launch QA.

No labels
launch-qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/infraternal#225
No description provided.