kayg commented

Owner

Scheduled Ping admission treats a temporarily full native control FIFO as a dead transport, closing a healthy host primary connection. RLR-13 regression is already committed as 54d24be15 on ping-fifo-fix.

Track the fix, audit of server/host control sends, regression tests, base-versus-branch terminal-flood measurements at LAN and 150 ms RTT, final fast gate and D243 size comparison here.

Related: #172 (RLR-13), #227 (connection indicator flicker), #201 (production reliability). No merge or deploy.

Scheduled Ping admission treats a temporarily full native control FIFO as a dead transport, closing a healthy host primary connection. RLR-13 regression is already committed as 54d24be15 on ping-fifo-fix. Track the fix, audit of server/host control sends, regression tests, base-versus-branch terminal-flood measurements at LAN and 150 ms RTT, final fast gate and D243 size comparison here. Related: #172 (RLR-13), #227 (connection indicator flicker), #201 (production reliability). No merge or deploy.

red-scheduled-confirm.log

1.1 KiB

red-server.log

1.5 KiB

red-host.log

2.9 KiB

red-session.log

914 B

instant-readd.log

2.8 KiB

full-readd.log

2.4 KiB

full-clean.log

2.1 KiB

full-retry.log

2.9 KiB

remove-rootless.log

1.9 KiB

remove-rootless-readd.log

2.1 KiB

remove-root.log

2.6 KiB

remove-root-readd.log

2.1 KiB

remove-root-mesh.log

2 KiB

remove-root-mesh-readd.log

2.2 KiB

staging-proof.log

16 KiB

authority-stall-soak.log

1.5 KiB

MANIFEST.txt

1.2 KiB

addhost-wizard-playwright.log

6.4 KiB

suites.txt

6.5 KiB

verify-103815e70.log

327 KiB

kayg commented

2026-10-17 06:14:02 +00:00

Author

Owner

Start: branch ping-fifo-fix, HEAD 54d24be15, base 2a5302cc7 (origin/dev at branch creation). Read #234 (no comments), all #227 and #172 comments, and repository contracts. Continuing RLR-13 failing regression. Scope: queue-admission fix, control-send audit and individual regressions, perf-VM base/branch LAN and 150 ms RTT flood, fast gate and D243 sizes. Push ping-fifo-fix only; no merge/deploy.

Start: branch ping-fifo-fix, HEAD 54d24be15, base 2a5302cc7 (origin/dev at branch creation). Read #234 (no comments), all #227 and #172 comments, and repository contracts. Continuing RLR-13 failing regression. Scope: queue-admission fix, control-send audit and individual regressions, perf-VM base/branch LAN and 150 ms RTT flood, fast gate and D243 sizes. Push ping-fifo-fix only; no merge/deploy.

kayg commented

2026-10-17 06:16:32 +00:00

Author

Owner

Control-send audit: embedded.rs scheduled Ping bypasses writes.can_admit(); process_incoming Ping also propagates Pong enqueue refusal as primary failure. daemon.rs reply() maps Full to WouldBlock and dispatch propagates it for Pongs, action results, plugin events, terminal closes and the census/open-result fence. Existing daemon_readiness_tests even require retirement on 32 coalesced Pings. Added individual regressions for scheduled Ping (existing cherry-pick), server Pong, host Ping replies, and host reliable action replies.

Implementation choice: skip scheduled probes before issuing their correlation when capacity is unavailable; keep checking outstanding reply deadlines. On the host dispatcher, reserve room for the maximum two-message turn (census then open Result) before polling reply-producing sources, waking only on queue capacity. Preserve bounded FIFO/order and ongoing independent native terminal work. No new dependency, wire layout or resource limit. This replaces local-full retirement with producer backpressure; transport failure and reply expiry still retire connections.

Control-send audit: embedded.rs scheduled Ping bypasses writes.can_admit(); process_incoming Ping also propagates Pong enqueue refusal as primary failure. daemon.rs reply() maps Full to WouldBlock and dispatch propagates it for Pongs, action results, plugin events, terminal closes and the census/open-result fence. Existing daemon_readiness_tests even require retirement on 32 coalesced Pings. Added individual regressions for scheduled Ping (existing cherry-pick), server Pong, host Ping replies, and host reliable action replies. Implementation choice: skip scheduled probes before issuing their correlation when capacity is unavailable; keep checking outstanding reply deadlines. On the host dispatcher, reserve room for the maximum two-message turn (census then open Result) before polling reply-producing sources, waking only on queue capacity. Preserve bounded FIFO/order and ongoing independent native terminal work. No new dependency, wire layout or resource limit. This replaces local-full retirement with producer backpressure; transport failure and reply expiry still retire connections.

kayg referenced this issue

2026-10-17 06:22:47 +00:00

Machine pill indicator flickers while a full-screen app redraws (production) #227

kayg commented

2026-10-17 06:29:39 +00:00

Author

Owner

Proposed additive clarification to contracts/wire.md route liveness (#234): local queue pressure is not evidence of a dead peer. Scheduled Ping may be skipped before its reply lease is issued; an already outstanding lease is never renewed by skipping/coalescing. Best-effort Pong and terminal-command probes may be omitted under local pressure, leaving the requester’s correlated timeout authoritative. Reliable control messages remain ordered in bounded producer queues and wait for admission capacity. No frame layouts, authority checks or timeout budgets change.

kayg commented

2026-10-17 06:32:14 +00:00

Author

Owner

All five regression cases observed failing before their fixes on base implementation 54d24be15 / origin/dev 2a5302cc7:

Scheduled Ping: actual enrolled mTLS QUIC primary closes with application code 0 after active+queued control saturation (receipt).
Server Pong: process_incoming returns connection failure at full control FIFO (receipt).
Host Ping burst and reliable action replies: both dispatchers return WouldBlock/control reply queue full (receipt).
Terminal Ping at 8 accepted Input commands: Full would retire that terminal (receipt).

Fixes are in the worktree; no commit yet. Existing primary-output saturation control passed. New host tests also require all 32 replies to drain in order when the same blocked writer is released; queue bounds and reply timeouts remain unchanged. The natural flood fixture uses two real PTYs and memory-only enrolled identity; no expected close count is fabricated. Baseline static build is running under the shared build slot.

All five regression cases observed failing before their fixes on base implementation 54d24be15 / origin/dev 2a5302cc7: - Scheduled Ping: actual enrolled mTLS QUIC primary closes with application code 0 after active+queued control saturation ([receipt](https://git.kayg.org/attachments/7908a50d-7f40-4f18-a331-cc2f702235d9)). - Server Pong: process_incoming returns connection failure at full control FIFO ([receipt](https://git.kayg.org/attachments/d2d44224-c158-4f35-aad9-9b89de2bb2ab)). - Host Ping burst and reliable action replies: both dispatchers return WouldBlock/control reply queue full ([receipt](https://git.kayg.org/attachments/4b69f4a2-ce5a-4c2d-af62-acaabeac2a10)). - Terminal Ping at 8 accepted Input commands: Full would retire that terminal ([receipt](https://git.kayg.org/attachments/fc9641f6-407d-46f1-92f7-f89c21dc9245)). Fixes are in the worktree; no commit yet. Existing primary-output saturation control passed. New host tests also require all 32 replies to drain in order when the same blocked writer is released; queue bounds and reply timeouts remain unchanged. The natural flood fixture uses two real PTYs and memory-only enrolled identity; no expected close count is fabricated. Baseline static build is running under the shared build slot.

kayg commented

2026-10-17 06:42:20 +00:00

Author

Owner

Baseline static x86_64 artifact from origin/dev 2a5302cc76: 21,300,016 bytes, SHA-256 236e8496260f6de12d93ecd0e0ce1f920716e1029b6a3db0ecdf1bccfe055606. Recipe: CARGO_TARGET_DIR=$HOME/.cache/codex-scratch/ping-fifo-fix/target scripts/static-cargo build --locked --release --package infraternal --target x86_64-unknown-linux-musl. Same pinned toolchain/compiler wrappers/profile as final gate; no dependency changes. Build passed under build-slot. Post-fix fmt/Clippy/touched-crate library tests and matched optimized measurement images are queued. Perf-VM functional smoke check is waiting for flock -w 3600; owner currently process-per-user ppu-soak-d296-p2c, started 2026-10-17T06:36:06.712368+00:00. No flood results yet.

Baseline static x86_64 artifact from origin/dev 2a5302cc76ecc97c2b2c7b752641dc3f944f4575: 21,300,016 bytes, SHA-256 236e8496260f6de12d93ecd0e0ce1f920716e1029b6a3db0ecdf1bccfe055606. Recipe: CARGO_TARGET_DIR=$HOME/.cache/codex-scratch/ping-fifo-fix/target scripts/static-cargo build --locked --release --package infraternal --target x86_64-unknown-linux-musl. Same pinned toolchain/compiler wrappers/profile as final gate; no dependency changes. Build passed under build-slot. Post-fix fmt/Clippy/touched-crate library tests and matched optimized measurement images are queued. Perf-VM functional smoke check is waiting for flock -w 3600; owner currently process-per-user ppu-soak-d296-p2c, started 2026-10-17T06:36:06.712368+00:00. No flood results yet.

kayg commented

2026-10-17 06:46:17 +00:00

Author

Owner

Broader server audit found the same probe-only retirement at websocket_transport.rs:361 and :388: both transport Ping and wire Ping replies retire the browser socket when the 128-slot reply FIFO is full. Added one regression per reply type, covering preservation of the queued reply, continued admission after drain, and rejection of a closed writer. Extracted the unchanged try_send policy for direct regression coverage; behavior fix remains pending observation of both red tests. Plugin/action admission overflow in plugins.rs returns a scoped refusal; remote replacement/priming overflow refuses the candidate, preserving the healthy primary. Those are resource admission refusals, not probe-induced healthy-connection retirement.

kayg commented

2026-10-17 06:56:43 +00:00

Author

Owner

Both browser reply regressions now observed red: 0 passed / 2 failed, each at queue-full admission. Fix applied: Full skips the optional reply while Closed remains fatal; the reader continues processing correlated Pongs. Receipt saved at $HOME/.cache/codex-scratch/ping-fifo-fix/evidence/red-browser.log; issue-attach is still running.

First host-library validation stopped on 15 NotFound fixture setup failures, including pre-existing relative ../../target paths; no timeout/baseline waiver claimed. The checkout had no target fixture parent because Rust outputs live in the mandated external CARGO_TARGET_DIR. Created a temporary, untracked target symlink to this job’s private scratch fixture directory and set INFRATERNAL_TEST_STATE_ROOT; corrected the new reply fixture to use an existing parent. All fixture/build artifacts remain under the job scratch directory. Rerunning scoped lint/tests before committing.

Both browser reply regressions now observed red: 0 passed / 2 failed, each at queue-full admission. Fix applied: Full skips the optional reply while Closed remains fatal; the reader continues processing correlated Pongs. Receipt saved at $HOME/.cache/codex-scratch/ping-fifo-fix/evidence/red-browser.log; issue-attach is still running. First host-library validation stopped on 15 NotFound fixture setup failures, including pre-existing relative ../../target paths; no timeout/baseline waiver claimed. The checkout had no target fixture parent because Rust outputs live in the mandated external CARGO_TARGET_DIR. Created a temporary, untracked target symlink to this job’s private scratch fixture directory and set INFRATERNAL_TEST_STATE_ROOT; corrected the new reply fixture to use an existing parent. All fixture/build artifacts remain under the job scratch directory. Rerunning scoped lint/tests before committing.

kayg commented

2026-10-17 07:36:24 +00:00

Author

Owner

Takeover after interruption: branch ping-fifo-fix, base 2a5302cc7 (origin/dev), previous head 54d24be15. Kept the worktree fixes after review and committed them atomically (3ae395719..748343331, pushed): wire contract, server scheduled Ping/Pong, browser proof replies, host terminal Ping, host control-reply backpressure, ignored flood fixture. Fixture now counts retired terminals and relay drops and prints its result line before asserting, so the base run still reports numbers. Scoped host lib: 128 passed / 0 failed. Running scripts/verify fast and building matched base/branch release test images for the perf VM now.

Takeover after interruption: branch ping-fifo-fix, base 2a5302cc7 (origin/dev), previous head 54d24be15. Kept the worktree fixes after review and committed them atomically (3ae395719..748343331, pushed): wire contract, server scheduled Ping/Pong, browser proof replies, host terminal Ping, host control-reply backpressure, ignored flood fixture. Fixture now counts retired terminals and relay drops and prints its result line before asserting, so the base run still reports numbers. Scoped host lib: 128 passed / 0 failed. Running `scripts/verify fast` and building matched base/branch release test images for the perf VM now.

kayg commented

2026-10-17 13:04:41 +00:00

Author

Owner

Measurements and root cause (head `c179cdcff`, perf VM 10.69.69.208, 4 vCPU, under /var/lock/perf-bench.lock)

Short version: the queue fix in this branch is correct and has its tests, but it is not what drops connections under real load. Under load, the primary closes because the reply deadline is shorter than one QUIC loss recovery. One lost packet on the ordered control stream holds back every later Pong for about 2 round trips. At 150 ms that is about 300–450 ms, and the deadline is about 250 ms. Changing the deadline belongs to the owner (D265: "dead link detected within about one round trip"), so this branch does not change it. Experiment numbers are below.

Fixture

remote_runtime::tests::ping_flood::primary_ping_during_terminal_flood (ignored; release musl test image). It uses a real enrolled mTLS QUIC host client, real PTYs and a parked secondary, so the shipping bounded control writer is in use. Both the primary and the secondary go through a userspace UDP delay relay with optional random loss. Nothing else on the VM is touched; no host netem. Load: PTY 1 runs while :; do head -c 32MiB /dev/zero | tr '\000' x; done. PTY 3 takes a continuous 4000-byte-line paste (server→host). PTY 2 echoes once a second (echo = input → output round trip). Images: base = origin/dev 2a5302cc7 + fixture only; branch = this branch.

10-minute rows, base origin/dev vs branch (fixture `d505e8b35`)

link	variant	primary closes	reconnects	echo p50 ms	echo p99 ms	flood MB	upload MB
LAN	base	0	0	6.8	21.7	15221	1620
LAN	branch	0	0	5.9	18.3	19704	1825
150 ms	base	0	0	153.7	188.7	752	68
150 ms	branch	2	2	154.1	306.3	739	65
150 ms + 1 % loss	base	12	12	154.6	438.2	555	59
150 ms + 1 % loss	branch	12	12	154.1	666.3	572	59
150 ms + 1 % loss	experiment: deadline 2·srtt + 4·rttvar + 75 ms	1	1	154.1	327.9	558	65

No terminal was retired in any row. Echo samples were 600/600 in every row except base 150 ms + 1 % (599/600).

Which code closed the primary (cfg(test) caller lines, `c179cdcff`)

base 150 ms + 1 % (120 s): 3 closes. Every one is the tick branch with liveness.expired()=true, never a full-queue Ping (probe_ok=false never fired).
base 150 ms, no loss, no paste (120 s): 1 close, expired=true.
base 150 ms + 1 %, no paste (120 s): 1 close, expired=true. So the server's own control FIFO does not need to be full.
branch with an extra change that lets Pongs bypass the paused control read gate (patch kept as evidence, not committed): still 2 closes in 120 s, expired=true. It brought nothing, so it was dropped.

What the branch fixes (each red before, green after, all in the fast gate)

Server scheduled Ping at a full native control FIFO closed a healthy primary (54d24be15 regression). Now the probe is skipped and no lease is issued.
Server Pong at a full FIFO retired the primary. Now it is best effort.
Browser socket: transport and wire Ping replies at a full 128-slot reply FIFO retired the socket.
Host terminal Ping at 8 queued Input/Resize commands retired the terminal stream.
Host dispatcher: a full control lane retired the whole channel (Pongs, results, events, census). Now replies wait for capacity, in order.

These are real P1 gaps; the deterministic test reaches (1) in about 75 ms of saturation. But the realistic flood rows show that, under load, the deadline fires first.

Decision needed (owner; foundational, not decided in a job)

The liveness deadline at 150 ms is about 250 ms. One QUIC loss recovery on the ordered control stream costs about 2·RTT + ack delay. Options, measured or estimated:

A. Deadline covers one loss recovery: 2·srtt + 4·rttvar + 75 ms (floor 250 ms unchanged, so LAN detection is unchanged). Measured: 12 → 1 close per 10 min at 150 ms + 1 %. Cost: dead-link detection goes from about 1 RTT to about 2 RTT on long links (about 250 → 400 ms at 150 ms). No size or memory cost.
B. Doubt, then retire: on expiry, mark the route doubtful and hedge or prime the secondary; retire only after a second budget with no reply. Keeps the 1-RTT signal and costs more code. Not measured.
C. Probes in QUIC datagrams: a lost probe never blocks later Pongs, so isolated loss never expires the lease, and continuous silence still trips at about 1 RTT. This changes the wire and the transport, so it is a foundational choice. Not measured.

The one close left in A (actor_lines=[958] = tick/expiry) suggests a burst loss longer than one recovery. A + B together would cover it.

Raw result lines: $HOME/.cache/codex-scratch/ping-fifo-fix/evidence/vm-results.txt. Experiment tree: .../exp (liveness.rs only). Pong-bypass patch: .../evidence/pong-routing.patch.

Also seen: auth::tests::socket_tests::restart_open_liveness_review::responsive_socket_survives_a_held_terminal_open_with_an_existing_attachment is flaky on base too (1 of 6 base runs fails after 90 s, "socket test stopped before its observed event"; 1 of 5 on branch). It existed before this branch.

Finish: head `c179cdcff` (pushed to ping-fifo-fix), scripts/verify fast verbatim


+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Static builds fail on size only. Origin/dev builds to exactly the cap (x86_64 21,300,016 B). This branch: x86_64 21,301,008 B (+992 B), aarch64 19,501,672 B (+1,296 B). That is real growth from the fix, not the registry-path noise. Caps were not raised. Needs a trim or an owner waiver (D243). No merge, no deploy.

## Measurements and root cause (head c179cdcff, perf VM 10.69.69.208, 4 vCPU, under /var/lock/perf-bench.lock) **Short version:** the queue fix in this branch is correct and has its tests, but it is **not** what drops connections under real load. Under load, the primary closes because the **reply deadline is shorter than one QUIC loss recovery**. One lost packet on the ordered control stream holds back every later Pong for about 2 round trips. At 150 ms that is about 300–450 ms, and the deadline is about 250 ms. Changing the deadline belongs to the owner (D265: "dead link detected within about one round trip"), so this branch does not change it. Experiment numbers are below. ### Fixture `remote_runtime::tests::ping_flood::primary_ping_during_terminal_flood` (ignored; release musl test image). It uses a real enrolled mTLS QUIC host client, real PTYs and a parked secondary, so the shipping bounded control writer is in use. Both the primary and the secondary go through a userspace UDP delay relay with optional random loss. Nothing else on the VM is touched; no host netem. Load: PTY 1 runs `while :; do head -c 32MiB /dev/zero | tr '\000' x; done`. PTY 3 takes a continuous 4000-byte-line paste (server→host). PTY 2 echoes once a second (echo = input → output round trip). Images: base = origin/dev 2a5302cc7 + fixture only; branch = this branch. ### 10-minute rows, base origin/dev vs branch (fixture d505e8b35) | link | variant | primary closes | reconnects | echo p50 ms | echo p99 ms | flood MB | upload MB | |---|---|---|---|---|---|---|---| | LAN | base | 0 | 0 | 6.8 | 21.7 | 15221 | 1620 | | LAN | branch | 0 | 0 | 5.9 | 18.3 | 19704 | 1825 | | 150 ms | base | 0 | 0 | 153.7 | 188.7 | 752 | 68 | | 150 ms | branch | 2 | 2 | 154.1 | 306.3 | 739 | 65 | | 150 ms + 1 % loss | base | **12** | 12 | 154.6 | 438.2 | 555 | 59 | | 150 ms + 1 % loss | branch | **12** | 12 | 154.1 | 666.3 | 572 | 59 | | 150 ms + 1 % loss | **experiment**: deadline 2·srtt + 4·rttvar + 75 ms | **1** | 1 | 154.1 | 327.9 | 558 | 65 | No terminal was retired in any row. Echo samples were 600/600 in every row except base 150 ms + 1 % (599/600). ### Which code closed the primary (cfg(test) caller lines, c179cdcff) - base 150 ms + 1 % (120 s): 3 closes. Every one is the tick branch with `liveness.expired()=true`, never a full-queue Ping (`probe_ok=false` never fired). - base 150 ms, no loss, no paste (120 s): 1 close, `expired=true`. - base 150 ms + 1 %, no paste (120 s): 1 close, `expired=true`. So the server's own control FIFO does not need to be full. - branch with an extra change that lets Pongs bypass the paused control read gate (patch kept as evidence, **not committed**): still 2 closes in 120 s, `expired=true`. It brought nothing, so it was dropped. ### What the branch fixes (each red before, green after, all in the fast gate) 1. Server scheduled Ping at a full native control FIFO closed a healthy primary (54d24be15 regression). Now the probe is skipped and no lease is issued. 2. Server Pong at a full FIFO retired the primary. Now it is best effort. 3. Browser socket: transport and wire Ping replies at a full 128-slot reply FIFO retired the socket. 4. Host terminal Ping at 8 queued Input/Resize commands retired the terminal stream. 5. Host dispatcher: a full control lane retired the whole channel (Pongs, results, events, census). Now replies wait for capacity, in order. These are real P1 gaps; the deterministic test reaches (1) in about 75 ms of saturation. But the realistic flood rows show that, under load, the deadline fires first. ### Decision needed (owner; foundational, not decided in a job) The liveness deadline at 150 ms is about 250 ms. One QUIC loss recovery on the ordered control stream costs about 2·RTT + ack delay. Options, measured or estimated: - **A. Deadline covers one loss recovery:** 2·srtt + 4·rttvar + 75 ms (floor 250 ms unchanged, so LAN detection is unchanged). Measured: 12 → 1 close per 10 min at 150 ms + 1 %. Cost: dead-link detection goes from about 1 RTT to about 2 RTT on long links (about 250 → 400 ms at 150 ms). No size or memory cost. - **B. Doubt, then retire:** on expiry, mark the route doubtful and hedge or prime the secondary; retire only after a second budget with no reply. Keeps the 1-RTT signal and costs more code. Not measured. - **C. Probes in QUIC datagrams:** a lost probe never blocks later Pongs, so isolated loss never expires the lease, and continuous silence still trips at about 1 RTT. This changes the wire and the transport, so it is a foundational choice. Not measured. The one close left in A (`actor_lines=[958]` = tick/expiry) suggests a burst loss longer than one recovery. A + B together would cover it. Raw result lines: `$HOME/.cache/codex-scratch/ping-fifo-fix/evidence/vm-results.txt`. Experiment tree: `.../exp` (liveness.rs only). Pong-bypass patch: `.../evidence/pong-routing.patch`. Also seen: `auth::tests::socket_tests::restart_open_liveness_review::responsive_socket_survives_a_held_terminal_open_with_an_existing_attachment` is flaky on **base** too (1 of 6 base runs fails after 90 s, "socket test stopped before its observed event"; 1 of 5 on branch). It existed before this branch. ### Finish: head c179cdcff (pushed to ping-fifo-fix), scripts/verify fast verbatim ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` Static builds fail on size only. Origin/dev builds to exactly the cap (x86_64 21,300,016 B). This branch: x86_64 21,301,008 B (**+992 B**), aarch64 19,501,672 B (**+1,296 B**). That is real growth from the fix, not the registry-path noise. Caps were not raised. Needs a trim or an owner waiver (D243). No merge, no deploy.

kayg referenced this issue

2026-10-17 13:04:42 +00:00

Machine pill indicator flickers while a full-screen app redraws (production) #227

kayg referenced this issue

2026-10-17 14:06:08 +00:00

A slow access check closes the whole browser connection #240

kayg commented

2026-10-17 16:06:42 +00:00

Author

Owner

Start D300 A+B (adaptive reply deadline + doubt-then-retire). Branch ping-fifo-fix rebased onto origin/release-prod-live 3048580c1 (production fix); local head ab514ecf2 before D300 work. Conflicts resolved keeping production behaviour: remote_control_queue_tests (kept production's Reconnecting-skipping delivery loop, added the diagnostic line), wire.md (both paragraphs), daemon.rs (production's SESSION_ONLY guards combined with the branch's control_ready gate). Plan: failing tests first for (A) deadline from measured srtt/rttvar with floor/ceiling, (B) first miss → doubtful + browser reconnecting without teardown, second consecutive miss → retire, reply clears doubt; then perf-VM 10-min rows base vs branch and dead-host (SIGSTOP) detection time.

Start D300 A+B (adaptive reply deadline + doubt-then-retire). Branch ping-fifo-fix rebased onto origin/release-prod-live 3048580c1 (production fix); local head ab514ecf2 before D300 work. Conflicts resolved keeping production behaviour: remote_control_queue_tests (kept production's Reconnecting-skipping delivery loop, added the diagnostic line), wire.md (both paragraphs), daemon.rs (production's SESSION_ONLY guards combined with the branch's `control_ready` gate). Plan: failing tests first for (A) deadline from measured srtt/rttvar with floor/ceiling, (B) first miss → doubtful + browser reconnecting without teardown, second consecutive miss → retire, reply clears doubt; then perf-VM 10-min rows base vs branch and dead-host (SIGSTOP) detection time.

kayg commented

2026-10-17 16:31:46 +00:00

Author

Owner

D300 A+B implemented; pushed ping-fifo-fix e77f5fb88 (rebased on release-prod-live 3048580c1).

Red first (production behaviour + stub API), scoped server lib run:

test liveness::tests::a_reply_during_doubt_clears_it_and_renews_the_route ... FAILED
test liveness::tests::host_allowance_covers_one_loss_recovery_and_browser_routes_keep_theirs ... FAILED
test liveness::tests::host_allowance_has_a_floor_a_settled_long_haul_value_and_a_ceiling ... FAILED
test liveness::tests::host_link_doubts_on_the_first_miss_and_retires_on_the_second ... FAILED
test remote_runtime::tests::standby::liveness_doubt::one_missed_primary_reply_shows_reconnecting_and_only_a_second_retires_the_link ... FAILED
thread 'liveness::tests::a_reply_during_doubt_clears_it_and_renews_the_route' (1838296) panicked at crates/infraternal-server/src/liveness.rs:503:9:
assertion failed: route.doubt_started()
thread 'liveness::tests::host_allowance_covers_one_loss_recovery_and_browser_routes_keep_theirs' (1838302) panicked at crates/infraternal-server/src/liveness.rs:449:9:
assertion `left == right` failed
  left: 412.5ms
 right: 550ms
thread 'liveness::tests::host_allowance_has_a_floor_a_settled_long_haul_value_and_a_ceiling' (1838303) panicked at crates/infraternal-server/src/liveness.rs:461:9:
assertion failed: long_haul.budget >= Duration::from_millis(375)
thread 'liveness::tests::host_link_doubts_on_the_first_miss_and_retires_on_the_second' (1838304) panicked at crates/infraternal-server/src/liveness.rs:480:9:
thread 'remote_runtime::tests::standby::liveness_doubt::one_missed_primary_reply_shows_reconnecting_and_only_a_second_retires_the_link' (1838317) panicked at crates/infraternal-server/src/remote_liveness_doubt_tests.rs:102:5:
assertion failed: connection.close_reason().is_none()

The actor test fails at its check right after the first miss: the production line closes the QUIC link on the first missed reply. (The socket_tests restart_open_liveness_review failure in the same run is the pre-existing flake already recorded on this issue.)

Green: 100 passed / 0 failed in scoped liveness/control_queue/embedded/websocket; the real mTLS QUIC actor test prints LIVENESS_DOUBT doubt_ms=269 retire_ms=524 (LAN: doubt at the 250 ms floor, retire after the second allowance). clippy -D warnings and fmt clean.

Design: host allowance clamp(2·srtt+4·rttvar+75 ms, 250 ms, 2 s) for primary + terminal routes, browser unchanged. Primary first miss → Reconnecting to settled windows (link kept); reply → new TerminalOutput::Recovered to exactly those windows → browser socket emits terminal.replay_complete at its own forwarded cursor (client already treats it as the readiness fence). Second consecutive miss → retire. Never-answered link retires at 2 s grace. Perf-VM rows and the dead-host timings follow.

D300 A+B implemented; pushed ping-fifo-fix e77f5fb88 (rebased on release-prod-live 3048580c1). **Red first** (production behaviour + stub API), scoped server lib run: ``` test liveness::tests::a_reply_during_doubt_clears_it_and_renews_the_route ... FAILED test liveness::tests::host_allowance_covers_one_loss_recovery_and_browser_routes_keep_theirs ... FAILED test liveness::tests::host_allowance_has_a_floor_a_settled_long_haul_value_and_a_ceiling ... FAILED test liveness::tests::host_link_doubts_on_the_first_miss_and_retires_on_the_second ... FAILED test remote_runtime::tests::standby::liveness_doubt::one_missed_primary_reply_shows_reconnecting_and_only_a_second_retires_the_link ... FAILED thread 'liveness::tests::a_reply_during_doubt_clears_it_and_renews_the_route' (1838296) panicked at crates/infraternal-server/src/liveness.rs:503:9: assertion failed: route.doubt_started() thread 'liveness::tests::host_allowance_covers_one_loss_recovery_and_browser_routes_keep_theirs' (1838302) panicked at crates/infraternal-server/src/liveness.rs:449:9: assertion `left == right` failed left: 412.5ms right: 550ms thread 'liveness::tests::host_allowance_has_a_floor_a_settled_long_haul_value_and_a_ceiling' (1838303) panicked at crates/infraternal-server/src/liveness.rs:461:9: assertion failed: long_haul.budget >= Duration::from_millis(375) thread 'liveness::tests::host_link_doubts_on_the_first_miss_and_retires_on_the_second' (1838304) panicked at crates/infraternal-server/src/liveness.rs:480:9: thread 'remote_runtime::tests::standby::liveness_doubt::one_missed_primary_reply_shows_reconnecting_and_only_a_second_retires_the_link' (1838317) panicked at crates/infraternal-server/src/remote_liveness_doubt_tests.rs:102:5: assertion failed: connection.close_reason().is_none() ``` The actor test fails at its check right after the first miss: the production line closes the QUIC link on the first missed reply. (The socket_tests restart_open_liveness_review failure in the same run is the pre-existing flake already recorded on this issue.) **Green**: 100 passed / 0 failed in scoped liveness/control_queue/embedded/websocket; the real mTLS QUIC actor test prints `LIVENESS_DOUBT doubt_ms=269 retire_ms=524` (LAN: doubt at the 250 ms floor, retire after the second allowance). clippy -D warnings and fmt clean. Design: host allowance clamp(2·srtt+4·rttvar+75 ms, 250 ms, 2 s) for primary + terminal routes, browser unchanged. Primary first miss → Reconnecting to settled windows (link kept); reply → new TerminalOutput::Recovered to exactly those windows → browser socket emits terminal.replay_complete at its own forwarded cursor (client already treats it as the readiness fence). Second consecutive miss → retire. Never-answered link retires at 2 s grace. Perf-VM rows and the dead-host timings follow.

kayg commented

2026-10-17 18:15:46 +00:00

Author

Owner

Break-it review (Opus) started: branch review-ping-fifo-fix from origin/ping-fifo-fix at 4e7a8d2e3 (base release-prod-live 3048580c1). Focus: liveness allowance bounds, alternating misses, doubt scoping/clearing, Recovered ordering, agent FIFO wait deadlock, browser path.

Break-it review (Opus) started: branch `review-ping-fifo-fix` from origin/ping-fifo-fix at 4e7a8d2e3 (base release-prod-live 3048580c1). Focus: liveness allowance bounds, alternating misses, doubt scoping/clearing, Recovered ordering, agent FIFO wait deadlock, browser path.

kayg commented

2026-10-17 19:17:45 +00:00

Author

Owner

D300 perf-VM measurement: base vs branch (head 4e7a8d2e3)

Venue: perf VM infraternal-perf-test (4 vCPU, kernel 6.12.111+deb13-amd64), over netbird ssh, every row under flock /var/lock/perf-bench.lock and attached to its own SSH session (nothing detached). Each row started only when VM load was below 16 and at least 40 GB was free. The highest VM load during any row was 5.43. Matched optimized fixture images (primary_ping_during_terminal_flood, real PTYs, one flood PTY + one echo PTY + one upload PTY, the same shim binary for both):

base = release-prod-live 3048580c1 + fixture: base-release-tests sha256 381a574c…5698591
branch = ping-fifo-fix b01f1e326 (code is unchanged at head 4e7a8d2e3): branch-release-tests sha256 f1a52e30…f3945334
shim sha256 f4b15342…fce2cf2

10-minute rows (flood + upload, one 600 s run each)

Link	Variant	Primary closes	Reconnects	Echo p50	Echo p99	Echo samples	Relay drops
LAN	base	0	0	6.22 ms	19.85 ms	600	0
LAN	branch	0	0	6.19 ms	20.31 ms	600	0
150 ms	base	1	1	153.84 ms	238.43 ms	600	0
150 ms	branch	0	0	153.91 ms	270.26 ms	600	0
150 ms + 1% loss	base	13	13	154.13 ms	787.69 ms	599	6,594
150 ms + 1% loss	branch	0	0	154.16 ms	326.03 ms	600	7,258

Dead host (the host stops answering 15 s in, no flood; 5 trials each)

Time from the host going silent to the window being told ("notice") and to the primary being retired ("retire"):

Link	Variant	Notice per trial (ms)	Notice median	Retire per trial (ms)	Retire median
LAN	base	266.1, 295.1, 269.4, 267.8, 284.0	269.4 ms	266.0, 295.2, 269.3, 267.7, 283.8	269.3 ms
LAN	branch	293.9, 270.2, 266.3, 266.0, 293.1	270.2 ms	544.2, 545.4, 515.8, 538.6, 544.1	544.1 ms
150 ms	base	345.8, 345.5, 346.5, 332.1, 327.2	345.5 ms	345.9, 345.7, 346.6, 332.1, 327.2	345.7 ms
150 ms	branch	454.2, 449.7, 463.8, 450.0, 468.1	454.2 ms	830.1, 823.2, 838.9, 825.1, 843.9	830.1 ms

All 26 runs exited 0, and no window was retired (terminals_retired=0) in any run.

Findings

Primary closes are gone. On a healthy 150 ms link, base closed the primary once in 10 minutes; at 150 ms + 1% loss it closed 13 times (13 reconnects) and pushed echo p99 to 787.69 ms. The branch closed 0 times in both rows, and its p99 under loss was 326.03 ms (−461.67 ms).
The doubt state never fired in the 10-minute rows (0 loss notices, 0 recoveries). The win in those rows comes from the reply deadline set from the measured RTT (part A), not from the doubt-then-retire step (part B). Part B shows only in the dead-host rows.
A truly dead host is detected later. The window is told almost as fast at LAN (median +0.8 ms) but 108.7 ms later at 150 ms, because the deadline now follows the RTT. The primary retires 274.8 ms later at LAN and 484.4 ms later at 150 ms, because it now waits for a second miss. That is the cost of the fix.
Branch echo p99 at 150 ms without loss is 31.82 ms higher (270.26 vs 238.43 ms) with the same p50. Each row is one run, and at 600 samples p99 is set by the 6 slowest echoes, so this may be noise. It is not settled.

Limits

The fixture runs the host agent inside the test process, so kill -STOP would stop the server too. The dead host is simulated by making the relay drop every datagram in both directions (RELAY_BLACKHOLE). From the server's side this matches a stopped agent process: no datagrams arrive.
One 10-minute run per row, so the base close counts (1 and 13) are single runs.

Row logs (VM): /root/infraternal-bench/ping-fifo-fix/d300/; copied to the build host at ~/.cache/codex-scratch/ping-fifo-fix/d300/vm-logs/d300/.

Size ratchet and gate

4e7a8d2e3 build: static caps set to the measured sizes (D243 16 KiB allowance, approved by the orchestrator): x86_64 21,090,448 → 21,093,520 B (+3,072), aarch64 19,316,264 → 19,319,768 B (+3,504); docs/PROCESS.md section 7 now matches.

scripts/verify fast on 4e7a8d2e3 (rc=0):

repository artifact gate: PASS (27 changed path(s), merge base 3048580c1184)
PT_INTERP: absent; size: 21093520 bytes; cap: 21093520 bytes; delta: +0 bytes
PT_INTERP: absent; size: 19319768 bytes; cap: 19319768 bytes; delta: +0 bytes
+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+

**D300 perf-VM measurement: base vs branch** (head `4e7a8d2e3`) Venue: perf VM `infraternal-perf-test` (4 vCPU, kernel 6.12.111+deb13-amd64), over `netbird ssh`, every row under `flock /var/lock/perf-bench.lock` and attached to its own SSH session (nothing detached). Each row started only when VM load was below 16 and at least 40 GB was free. The highest VM load during any row was 5.43. Matched optimized fixture images (`primary_ping_during_terminal_flood`, real PTYs, one flood PTY + one echo PTY + one upload PTY, the same shim binary for both): - base = `release-prod-live` 3048580c1 + fixture: `base-release-tests` sha256 `381a574c…5698591` - branch = `ping-fifo-fix` b01f1e326 (code is unchanged at head 4e7a8d2e3): `branch-release-tests` sha256 `f1a52e30…f3945334` - shim sha256 `f4b15342…fce2cf2` ### 10-minute rows (flood + upload, one 600 s run each) | Link | Variant | Primary closes | Reconnects | Doubt recoveries | Echo p50 | Echo p99 | Echo samples | Relay drops | |---|---|---:|---:|---:|---:|---:|---:|---:| | LAN | base | 0 | 0 | 0 | 6.22 ms | 19.85 ms | 600 | 0 | | LAN | branch | 0 | 0 | 0 | 6.19 ms | 20.31 ms | 600 | 0 | | 150 ms | base | **1** | **1** | 0 | 153.84 ms | 238.43 ms | 600 | 0 | | 150 ms | branch | 0 | 0 | 0 | 153.91 ms | 270.26 ms | 600 | 0 | | 150 ms + 1% loss | base | **13** | **13** | 0 | 154.13 ms | **787.69 ms** | 599 | 6,594 | | 150 ms + 1% loss | branch | 0 | 0 | 0 | 154.16 ms | 326.03 ms | 600 | 7,258 | ### Dead host (the host stops answering 15 s in, no flood; 5 trials each) Time from the host going silent to the window being told ("notice") and to the primary being retired ("retire"): | Link | Variant | Notice per trial (ms) | Notice median | Retire per trial (ms) | Retire median | |---|---|---|---:|---|---:| | LAN | base | 266.1, 295.1, 269.4, 267.8, 284.0 | 269.4 ms | 266.0, 295.2, 269.3, 267.7, 283.8 | 269.3 ms | | LAN | branch | 293.9, 270.2, 266.3, 266.0, 293.1 | 270.2 ms | 544.2, 545.4, 515.8, 538.6, 544.1 | 544.1 ms | | 150 ms | base | 345.8, 345.5, 346.5, 332.1, 327.2 | 345.5 ms | 345.9, 345.7, 346.6, 332.1, 327.2 | 345.7 ms | | 150 ms | branch | 454.2, 449.7, 463.8, 450.0, 468.1 | 454.2 ms | 830.1, 823.2, 838.9, 825.1, 843.9 | 830.1 ms | All 26 runs exited 0, and no window was retired (`terminals_retired=0`) in any run. ### Findings - **Primary closes are gone.** On a healthy 150 ms link, base closed the primary once in 10 minutes; at 150 ms + 1% loss it closed 13 times (13 reconnects) and pushed echo p99 to 787.69 ms. The branch closed 0 times in both rows, and its p99 under loss was 326.03 ms (−461.67 ms). - **The doubt state never fired in the 10-minute rows** (0 loss notices, 0 recoveries). The win in those rows comes from the reply deadline set from the measured RTT (part A), not from the doubt-then-retire step (part B). Part B shows only in the dead-host rows. - **A truly dead host is detected later.** The window is told almost as fast at LAN (median +0.8 ms) but 108.7 ms later at 150 ms, because the deadline now follows the RTT. The primary retires 274.8 ms later at LAN and 484.4 ms later at 150 ms, because it now waits for a second miss. That is the cost of the fix. - **Branch echo p99 at 150 ms without loss is 31.82 ms higher** (270.26 vs 238.43 ms) with the same p50. Each row is one run, and at 600 samples p99 is set by the 6 slowest echoes, so this may be noise. It is not settled. ### Limits - The fixture runs the host agent inside the test process, so `kill -STOP` would stop the server too. The dead host is simulated by making the relay drop every datagram in both directions (`RELAY_BLACKHOLE`). From the server's side this matches a stopped agent process: no datagrams arrive. - One 10-minute run per row, so the base close counts (1 and 13) are single runs. Row logs (VM): `/root/infraternal-bench/ping-fifo-fix/d300/`; copied to the build host at `~/.cache/codex-scratch/ping-fifo-fix/d300/vm-logs/d300/`. ### Size ratchet and gate `4e7a8d2e3` build: static caps set to the measured sizes (D243 16 KiB allowance, approved by the orchestrator): x86_64 21,090,448 → 21,093,520 B (+3,072), aarch64 19,316,264 → 19,319,768 B (+3,504); docs/PROCESS.md section 7 now matches. `scripts/verify fast` on 4e7a8d2e3 (rc=0): ``` repository artifact gate: PASS (27 changed path(s), merge base 3048580c1184) PT_INTERP: absent; size: 21093520 bytes; cap: 21093520 bytes; delta: +0 bytes PT_INTERP: absent; size: 19319768 bytes; cap: 19319768 bytes; delta: +0 bytes +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | PASS | +---------------------------+--------------+ ```

kayg commented

2026-10-17 19:50:22 +00:00

Author

Owner

Break-it review (Opus) of ping-fifo-fix `4e7a8d2e3`: HOLD (P0 0, P1 1, P2 2, P3 3)

Branch review-ping-fifo-fix head a7b648166 = target + failing tests in crates/infraternal-server/src/liveness_doubt_breakit_tests.rs. No fixes. The perf VM was not used.

P1-1: a stale doubt confirms the replacement link mid-replay (failing test, fails 5 runs out of 5)

run_actor's doubted list is drained only by notify_doubt(true). Nothing clears it on connection_lost, on Activate/promotion, or on any liveness = Liveness::host_link() reset. The sequence: doubt, second miss, retire, reattach. The new lease is "not doubtful", so the next tick sends Recovered to the window as soon as the reattach Ack lands. At that point the host is still replaying and its terminal.replay_complete has not been sent. (settled checks only awaiting_ack/proof_rebind_pending.)
Observed: notices so far: ["Ack { in_seq_acked: 0 }", "Recovered"].
websocket.rs turns this into terminal.replay_complete. The browser (terminal.ts:930/1395) treats it as the replay fence and stops suppressing terminal responses for the rest of the replay. Replayed terminal queries then get answered again into the user's PTY, "Catching up…" clears early, and unacked input is resent before the replay ends. This happens on every retire that follows a doubt and has a non-trivial replay, which is the dead-host path that D300 is about.
Test: retired_doubt_never_confirms_a_replacement_link_mid_replay.

P2-1: live output during a doubt is handled as replay in the browser (code read)

Doubt sends terminal.reconnecting, which sets attachmentConfirmed=false and suppressReplayResponses=true (terminal.ts:914-925). The link is kept and live output keeps flowing. Responses to live queries (DA/DSR/OSC from a program that starts during the blip) are dropped, and typing is held until Recovered. The browser path was left untouched, and that is what causes this.

P2-2: retirement timing vs the D300 text (numbers; question for the owner)

Settled path, paused clock (doubt_and_retire_times_per_rtt):

RTT	Reconnecting shown	link retired
1 / 30 ms	251 ms	501 ms
150 ms	376 ms	751 ms
300 ms	676 ms	1351 ms
600 ms	1276 ms	2551 ms
1000 ms	2001 ms	4001 ms

D300 says "a dead host is noticed after about 0.4 s at 150 ms … LAN is unchanged". That holds only for the Reconnecting notice. The link retires, and the reconnect starts, at 0.75 s at 150 ms and at 0.5 s on LAN (it was 0.25 s). Worst-case retirement of a dead host is 4001 ms, two 2 s ceilings (inflated_host_link_still_retires_a_dead_host_within_four_seconds, passes). Inflation limits itself: a reply later than 2× the allowance is rejected.

P3-1: doubt flaps forever at the 2 s ceiling (failing test)

Take a proven link whose RTT climbs to 2–4 s because a deep uplink buffer fills. Every probe raises a doubt and its own reply clears it: 10 doubt episodes in 10 probes, and the link never retires. Each flip brings the P2-1 effects. This does not matter at 150 ms (the allowance is 375 ms once settled). Test: queued_path_at_the_ceiling_does_not_flap_doubt_every_probe.

P3-2: the doubt notice is never retried for a full window

The notify_doubt doc says a full window "is told on a later tick". Only the Recovered side retries. doubt_started fires once per episode, so a window that is full at that tick never gets Reconnecting.

P3-3: Recovered leaves websocket `awaiting_ack = true`

The in_seq_acked > input_sent_end check stays relaxed until the next Ack.

Checked, no finding

A host that answers every other ping: probes are pipelined each tick and a Pong clears all earlier nonces, so no doubt is raised.
The infraternal agent waiting for FIFO capacity: no permanent deadlock. When its lane is full it stops reading, but the server's writes carry a 1 s timeout and host liveness retires the link after 2× the allowance. On shared Unix/TCP/SSH transports, terminal input for all sessions also pauses for that time (bounded).
A server Ping skipped at a full control FIFO cannot hide a dead host: the 1 s control-flight timeout retires a blocked writer, and pipelined probes keep a lease outstanding.
The browser allowance is unchanged. Recovered and Output share one ordered queue, and the fence cursor equals the forwarded bytes.

Gate: `scripts/verify fast` on `a7b648166` (the only failures are the 2 review tests)

| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | FAIL (101)   |
| cargo-deny                | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
test embedded::regressions::liveness_doubt_breakit_tests::queued_path_at_the_ceiling_does_not_flap_doubt_every_probe ... FAILED
test embedded::regressions::liveness_doubt_breakit_tests::retired_doubt_never_confirms_a_replacement_link_mid_replay ... FAILED
test result: FAILED. 514 passed; 2 failed; 49 ignored; 0 measured; 0 filtered out; finished in 411.27s

## Break-it review (Opus) of ping-fifo-fix 4e7a8d2e3: **HOLD** (P0 0, P1 1, P2 2, P3 3) Branch `review-ping-fifo-fix` head **a7b648166** = target + failing tests in `crates/infraternal-server/src/liveness_doubt_breakit_tests.rs`. No fixes. The perf VM was not used. ### P1-1: a stale doubt confirms the replacement link mid-replay (failing test, fails 5 runs out of 5) `run_actor`'s `doubted` list is drained only by `notify_doubt(true)`. Nothing clears it on `connection_lost`, on Activate/promotion, or on any `liveness = Liveness::host_link()` reset. The sequence: doubt, second miss, retire, reattach. The new lease is "not doubtful", so the next tick sends `Recovered` to the window as soon as the reattach Ack lands. At that point the host is still replaying and its `terminal.replay_complete` has not been sent. (`settled` checks only `awaiting_ack`/`proof_rebind_pending`.) Observed: `notices so far: ["Ack { in_seq_acked: 0 }", "Recovered"]`. websocket.rs turns this into `terminal.replay_complete`. The browser (terminal.ts:930/1395) treats it as the replay fence and stops suppressing terminal responses for the rest of the replay. Replayed terminal queries then get answered again into the user's PTY, "Catching up…" clears early, and unacked input is resent before the replay ends. This happens on every retire that follows a doubt and has a non-trivial replay, which is the dead-host path that D300 is about. Test: `retired_doubt_never_confirms_a_replacement_link_mid_replay`. ### P2-1: live output during a doubt is handled as replay in the browser (code read) Doubt sends `terminal.reconnecting`, which sets `attachmentConfirmed=false` and `suppressReplayResponses=true` (terminal.ts:914-925). The link is kept and live output keeps flowing. Responses to *live* queries (DA/DSR/OSC from a program that starts during the blip) are dropped, and typing is held until Recovered. The browser path was left untouched, and that is what causes this. ### P2-2: retirement timing vs the D300 text (numbers; question for the owner) Settled path, paused clock (`doubt_and_retire_times_per_rtt`): | RTT | Reconnecting shown | link retired | |---|---|---| | 1 / 30 ms | 251 ms | 501 ms | | 150 ms | 376 ms | 751 ms | | 300 ms | 676 ms | 1351 ms | | 600 ms | 1276 ms | 2551 ms | | 1000 ms | 2001 ms | 4001 ms | D300 says "a dead host is noticed after about 0.4 s at 150 ms … LAN is unchanged". That holds only for the Reconnecting notice. The link retires, and the reconnect starts, at 0.75 s at 150 ms and at 0.5 s on LAN (it was 0.25 s). Worst-case retirement of a dead host is **4001 ms**, two 2 s ceilings (`inflated_host_link_still_retires_a_dead_host_within_four_seconds`, passes). Inflation limits itself: a reply later than 2× the allowance is rejected. ### P3-1: doubt flaps forever at the 2 s ceiling (failing test) Take a proven link whose RTT climbs to 2–4 s because a deep uplink buffer fills. Every probe raises a doubt and its own reply clears it: 10 doubt episodes in 10 probes, and the link never retires. Each flip brings the P2-1 effects. This does not matter at 150 ms (the allowance is 375 ms once settled). Test: `queued_path_at_the_ceiling_does_not_flap_doubt_every_probe`. ### P3-2: the doubt notice is never retried for a full window The `notify_doubt` doc says a full window "is told on a later tick". Only the Recovered side retries. `doubt_started` fires once per episode, so a window that is full at that tick never gets Reconnecting. ### P3-3: Recovered leaves websocket `awaiting_ack = true` The `in_seq_acked > input_sent_end` check stays relaxed until the next Ack. ### Checked, no finding - A host that answers every other ping: probes are pipelined each tick and a Pong clears all earlier nonces, so no doubt is raised. - The infraternal agent waiting for FIFO capacity: no permanent deadlock. When its lane is full it stops reading, but the server's writes carry a 1 s timeout and host liveness retires the link after 2× the allowance. On shared Unix/TCP/SSH transports, terminal input for all sessions also pauses for that time (bounded). - A server Ping skipped at a full control FIFO cannot hide a dead host: the 1 s control-flight timeout retires a blocked writer, and pipelined probes keep a lease outstanding. - The browser allowance is unchanged. Recovered and Output share one ordered queue, and the fence cursor equals the forwarded bytes. ### Gate: `scripts/verify fast` on a7b648166 (the only failures are the 2 review tests) ``` | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | FAIL (101) | | cargo-deny | PASS | | static builds | PASS | +---------------------------+--------------+ test embedded::regressions::liveness_doubt_breakit_tests::queued_path_at_the_ceiling_does_not_flap_doubt_every_probe ... FAILED test embedded::regressions::liveness_doubt_breakit_tests::retired_doubt_never_confirms_a_replacement_link_mid_replay ... FAILED test result: FAILED. 514 passed; 2 failed; 49 ignored; 0 measured; 0 filtered out; finished in 411.27s ```

kayg commented

2026-10-17 19:51:55 +00:00

Author

Owner

Fix round for the break-it review (Opus) started (Claude). Branch ping-fifo-fix, base 4e7a8d2e3; review tests b814e60e2 + a7b648166 cherry-picked first. Plan: one commit per finding (P1-1, P2-1, P3-1, P3-2, P3-3); P2-2 goes to the owner as a note, timings unchanged.

Fix round for the break-it review (Opus) started (Claude). Branch `ping-fifo-fix`, base 4e7a8d2e3; review tests b814e60e2 + a7b648166 cherry-picked first. Plan: one commit per finding (P1-1, P2-1, P3-1, P3-2, P3-3); P2-2 goes to the owner as a note, timings unchanged.

kayg commented

2026-10-17 20:18:01 +00:00

Author

Owner

Break-it fixes on `ping-fifo-fix` (pushed, head `f06512497`; gate follows)

Review tests cherry-picked first (727255f09, 7cce516c4). One commit per finding:

P1-1 cc657f35e: doubted belongs to its link. Every lease reset (fresh connect, Activate promotion, idle link) goes through fresh_host_lease, which also clears the list, and the actor clears it whenever no link is installed. A replacement link never gets Recovered from an earlier episode; replay completion is left to the replacement's own fence. retired_doubt_never_confirms_a_replacement_link_mid_replay passes.
P3-1 dbc82eb9b: ceiling hysteresis. If a doubt recovers while the allowance is at the 2 s ceiling, the link is pinned: later first misses raise no doubt, and only retirement (still 2 × 2 s) reports them. The first reply that brings the allowance back below the ceiling unpins it. Below the ceiling nothing changes, so 150 ms paths (~375 ms allowance) keep early notice after a single loss. queued_path_at_the_ceiling_does_not_flap_doubt_every_probe passes, plus 2 new liveness tests.
P2-1 (+P3-3) 9c648cc08: a doubt is now its own state. Actor → TerminalOutput::LinkDoubt, socket → terminal.link_doubt, recovery → terminal.link_recovered. The viewer shows "Reconnecting…" in the window status and changes nothing else: it stays confirmed, keeps answering live terminal queries and keeps sending input. link_recovered clears only that exact text and is never a replay fence. Neither notice sets the socket's awaiting_ack, which removes the cause of P3-3. Older viewers treat unknown notices as a protocol failure, so these notices go only to viewers that set the new browser Hello caps bit 1 (LINK_DOUBT_NOTICES). New viewers set bit 1 and ignore notice types they do not know. Tests: websocket link_notice_tests, tracer test/link-doubt.test.ts (now in the ts gate). contracts/wire.md is updated.
P3-2 f06512497: report_doubt offers the doubt notice again on every tick of the episode, so a window whose queue was full is told once it has room. Test: a_window_full_at_doubt_start_is_told_on_a_later_tick.

No timings changed, so the 10-minute perf rows were not re-run. Settled-path doubt and retirement times are the same as in the review table.

Owner note (P2-2, wording only; timings unchanged)

Measured on a settled path with a paused clock (doubt_and_retire_times_per_rtt):

RTT	"Reconnecting" shown	link retired
LAN (1–30 ms)	251 ms	501 ms
150 ms	376 ms	751 ms
300 ms	676 ms	1351 ms
1 s	2001 ms	4001 ms

D300's "a dead host is noticed after about 0.4 s at 150 ms … LAN is unchanged" is true only for the window indicator. The link itself retires, and the reconnect starts, at about twice that: 0.75 s at 150 ms, and 0.5 s on LAN where it was 0.25 s. In the worst case (allowance at the ceiling) a dead host retires after 4 s. Question for kayg: keep these timings and correct the D300 wording, or change the retire rule? Nothing was changed here.

## Break-it fixes on `ping-fifo-fix` (pushed, head f06512497; gate follows) Review tests cherry-picked first (727255f09, 7cce516c4). One commit per finding: - **P1-1** cc657f35e: `doubted` belongs to its link. Every lease reset (fresh connect, Activate promotion, idle link) goes through `fresh_host_lease`, which also clears the list, and the actor clears it whenever no link is installed. A replacement link never gets `Recovered` from an earlier episode; replay completion is left to the replacement's own fence. `retired_doubt_never_confirms_a_replacement_link_mid_replay` passes. - **P3-1** dbc82eb9b: ceiling hysteresis. If a doubt recovers while the allowance is at the 2 s ceiling, the link is *pinned*: later first misses raise no doubt, and only retirement (still 2 × 2 s) reports them. The first reply that brings the allowance back below the ceiling unpins it. Below the ceiling nothing changes, so 150 ms paths (~375 ms allowance) keep early notice after a single loss. `queued_path_at_the_ceiling_does_not_flap_doubt_every_probe` passes, plus 2 new liveness tests. - **P2-1 (+P3-3)** 9c648cc08: a doubt is now its own state. Actor → `TerminalOutput::LinkDoubt`, socket → `terminal.link_doubt`, recovery → `terminal.link_recovered`. The viewer shows "Reconnecting…" in the window status and changes nothing else: it stays confirmed, keeps answering live terminal queries and keeps sending input. `link_recovered` clears only that exact text and is never a replay fence. Neither notice sets the socket's `awaiting_ack`, which removes the cause of P3-3. Older viewers treat unknown notices as a protocol failure, so these notices go only to viewers that set the new browser Hello caps bit 1 (`LINK_DOUBT_NOTICES`). New viewers set bit 1 and ignore notice types they do not know. Tests: websocket `link_notice_tests`, tracer `test/link-doubt.test.ts` (now in the ts gate). contracts/wire.md is updated. - **P3-2** f06512497: `report_doubt` offers the doubt notice again on every tick of the episode, so a window whose queue was full is told once it has room. Test: `a_window_full_at_doubt_start_is_told_on_a_later_tick`. No timings changed, so the 10-minute perf rows were not re-run. Settled-path doubt and retirement times are the same as in the review table. ### Owner note (P2-2, wording only; timings unchanged) Measured on a settled path with a paused clock (`doubt_and_retire_times_per_rtt`): | RTT | "Reconnecting" shown | link retired | |---|---:|---:| | LAN (1–30 ms) | 251 ms | 501 ms | | 150 ms | 376 ms | 751 ms | | 300 ms | 676 ms | 1351 ms | | 1 s | 2001 ms | 4001 ms | D300's "a dead host is noticed after about 0.4 s at 150 ms … LAN is unchanged" is true only for the window indicator. The link itself retires, and the reconnect starts, at about twice that: 0.75 s at 150 ms, and 0.5 s on LAN where it was 0.25 s. In the worst case (allowance at the ceiling) a dead host retires after 4 s. **Question for kayg:** keep these timings and correct the D300 wording, or change the retire rule? Nothing was changed here.

kayg commented

2026-10-17 21:05:50 +00:00

Author

Owner

Break-it fixes done: head `8f3945801` (pushed `ping-fifo-fix`)

Two commits since the earlier note:

b5ed41406 makes the new P3-2 test pass pedantic clippy (assert_is_empty).
8f3945801 ratchets the size caps. Both binaries shrank: x86_64 21,093,520 → 21,091,504 B (−2,016), aarch64 19,319,768 → 19,318,840 B (−928). docs/PROCESS.md §7 matches.

Flake, not from this round: the second gate run had one failure, infraternal-agent daemon::readiness_tests::uninstall_handoff_runs_only_after_a_successful_real_socket_write (daemon_readiness_tests.rs:350, left: true right: false). The fix round did not touch crates/infraternal-agent. The test passes 10 out of 10 runs on its own, and the third full run below is green. A likely cause from reading the code (not proven): Socket::inherited(writer.as_raw_fd()) shares one fd with writer, so the fd is closed twice and can race with other tests in the same process.

scripts/verify fast on 8f3945801 (rc=0):

repository artifact gate: PASS (32 changed path(s), merge base 3048580c1184)
PT_INTERP: absent; size: 21091504 bytes; cap: 21091504 bytes; delta: +0 bytes
PT_INTERP: absent; size: 19318840 bytes; cap: 19318840 bytes; delta: +0 bytes
+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
Tracer: bun x tsc PASS; bun test 242 pass / 0 fail (includes test/link-doubt.test.ts); TS protocol 37 pass / 0 fail

The perf VM was not used: no timing changed. The owner question on P2-2 is still open (see the previous comment).

## Break-it fixes done: head **8f3945801** (pushed `ping-fifo-fix`) Two commits since the earlier note: - b5ed41406 makes the new P3-2 test pass pedantic clippy (`assert_is_empty`). - 8f3945801 ratchets the size caps. Both binaries shrank: x86_64 21,093,520 → **21,091,504 B** (−2,016), aarch64 19,319,768 → **19,318,840 B** (−928). docs/PROCESS.md §7 matches. Flake, not from this round: the second gate run had one failure, `infraternal-agent daemon::readiness_tests::uninstall_handoff_runs_only_after_a_successful_real_socket_write` (daemon_readiness_tests.rs:350, `left: true right: false`). The fix round did not touch `crates/infraternal-agent`. The test passes 10 out of 10 runs on its own, and the third full run below is green. A likely cause from reading the code (not proven): `Socket::inherited(writer.as_raw_fd())` shares one fd with `writer`, so the fd is closed twice and can race with other tests in the same process. `scripts/verify fast` on 8f3945801 (rc=0): ``` repository artifact gate: PASS (32 changed path(s), merge base 3048580c1184) PT_INTERP: absent; size: 21091504 bytes; cap: 21091504 bytes; delta: +0 bytes PT_INTERP: absent; size: 19318840 bytes; cap: 19318840 bytes; delta: +0 bytes +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | PASS | +---------------------------+--------------+ Tracer: bun x tsc PASS; bun test 242 pass / 0 fail (includes test/link-doubt.test.ts); TS protocol 37 pass / 0 fail ``` The perf VM was not used: no timing changed. The owner question on P2-2 is still open (see the previous comment).

kayg commented

2026-10-17 21:14:25 +00:00

Author

Owner

Break-it delta review r2 (Opus) started. Branch review-ping-fifo-fix-r2, base origin/ping-fifo-fix 8f3945801. Scope: prior P1-1/P2-1/P3-1/P3-2/P3-3, caps-bit negotiation, indicator text, retry bounds, ceiling pin vs retirement, agent flake uninstall_handoff_runs_only_after_a_successful_real_socket_write.

Break-it delta review r2 (Opus) started. Branch `review-ping-fifo-fix-r2`, base origin/ping-fifo-fix 8f3945801. Scope: prior P1-1/P2-1/P3-1/P3-2/P3-3, caps-bit negotiation, indicator text, retry bounds, ceiling pin vs retirement, agent flake `uninstall_handoff_runs_only_after_a_successful_real_socket_write`.

kayg commented

2026-10-18 02:47:26 +00:00

Author

Owner

Break-it review r2 (Opus) — #234, origin/ping-fifo-fix `8f3945801`

Review branch: review-ping-fifo-fix-r2 @ fda0c95f7 (tests only; two commits on top of 8f3945801).

Prior findings

P1-1 FIXED. retired_doubt_never_confirms_a_replacement_link_mid_replay passes. Every lease reset goes through fresh_host_lease (fresh connect, Activate in both select arms, idle link). The loop top clears doubted while connection.is_none(). I checked every connection.take() site: each one passes through None before a new link is installed, or through fresh_host_lease.
P2-1 FIXED. A doubt is now LinkDoubt → terminal.link_doubt. It sets no awaiting_ack, the window stays confirmed, and replies and input keep flowing (tracer link-doubt.test.ts passes).
P3-1 FIXED. queued_path_at_the_ceiling_does_not_flap_doubt_every_probe passes. The pin is cleared by any reply that brings the allowance below 2 s.
P3-2 FIXED. A full window is told on a later tick (a_window_full_at_doubt_start_is_told_on_a_later_tick passes).
P3-3 FIXED. Recovery no longer sets or keeps the websocket awaiting_ack (link_notice returns false for both doubt notices).

Checks that passed (new tests in liveness_doubt_breakit_r2_tests.rs)

Caps negotiation.
- Old browser + new server: neither notice is sent (link_notice(.., false) == None). The old viewer would throw on an unknown notice, and it never gets one.
- New browser + old server: dev and 4e7a8d2e3 test only caps & RESUME_NEXT, so bit 1 is ignored. dev has no doubt path at all, so nothing changes for it.
- Leg-B bit 1 (REMOTE_WRITER_BINDING) is a separate agent↔server channel and does not collide with the browser bit.
Reconnect mid-doubt: a_window_that_reconnects_mid_doubt_is_told_and_recovered passes. The new attachment is told once it settles and later gets its recovery. The departed id is dropped and not retried.
Retry bound: doubt_retry_sends_one_notice_per_window_per_episode passes. 8 windows, half of them full at the start, over 80+ ticks: each gets exactly 1 LinkDoubt and 0 Recovered.
Ceiling pin vs dead host: a_pinned_dead_host_retires_within_two_ceilings_without_doubt passes. The link retires ≤ 4 s + 1 tick after the last probe, with no doubt notice. retired() ignores pinned.
Status text: link_recovered clears only the exact text "Reconnecting…". No other code sets that exact string (the connection layer uses "Connection lost. Reconnecting…").

New findings

P3-4: the doubt indicator is invisible on the canvas (failing test)

On the canvas, a terminal's status element is the detached relay element (canvas/status.ts). That relay drops routine states that end in "…" or mention "reconnect", and leaves them to the window's status icon. The status icon follows the terminal hooks (onDisconnected/onAttached), and link_doubt calls none of them. So in the product UI a doubted link shows nothing. "Reconnecting…" is visible only on the non-canvas core screen.

This is safe, because nothing is held or suppressed. But the "browser shows Reconnecting…" claim does not hold where people actually work.

Test: apps/tracer/test/link-doubt-canvas.test.ts (fails: no toast, no hook).

Fix direction for the owner or the fix job: a hook (for example a doubted connection state) that the window status icon reads.

Not a deploy blocker.

Flaky agent test: REFUTED double close; the root cause is a fork race in the test (failing tests)

Socket never closes its fd (Descriptor has no Drop; transport.rs module doc says so). The test drops socket (which deregisters the fd) before writer (which closes it). So there is no double close.

The recorded failure was "left true, right false": the unhealthy write succeeded. The test assumes that drop(reader) closes the peer. But any child that a concurrent test forks holds a copy of every fd until it calls exec (CLOEXEC acts only at exec). While that copy exists the socket stays open and the write lands in its buffer.

Evidence:

the_unhealthy_handoff_case_races_concurrent_spawns: rounds=20000 spawned=428 write_succeeded=62 (gate rerun: spawned=351 write_succeeded=84) (fails).
a_dropped_peer_still_accepts_the_write_while_a_forked_copy_holds_it reproduces it deterministically with a held copy (fails).

Fix: call reader.shutdown(Shutdown::Both) before the drop. Shutdown acts on the socket itself, not on one descriptor. This is a test-only issue, not a product bug.

Verdict

MERGE: P0 0 / P1 0 / P2 0 / P3 1 (P3-4), plus 1 test-only flake with its root cause found.

Gate (`scripts/verify fast` at `fda0c95f7`, verbatim)

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | FAIL (101)   |
| cargo-deny                | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+

The test step fails only on the two intended failing agent tests (test result: FAILED. 138 passed; 2 failed; 20 ignored). Cargo stops at that binary, so the TS step did not run in this gate. The TS failing test is bun test test/link-doubt-canvas.test.ts (1 fail; link-doubt.test.ts 4 pass). An earlier gate run at 52231b0c7 also failed session_exit_events (idle canary: 1 wakeup in 2.2 s, under load). It is unrelated to this delta.

# Break-it review r2 (Opus) — #234, origin/ping-fifo-fix 8f3945801 Review branch: `review-ping-fifo-fix-r2` @ fda0c95f7 (tests only; two commits on top of 8f3945801). ## Prior findings - **P1-1 FIXED.** `retired_doubt_never_confirms_a_replacement_link_mid_replay` passes. Every lease reset goes through `fresh_host_lease` (fresh connect, Activate in both select arms, idle link). The loop top clears `doubted` while `connection.is_none()`. I checked every `connection.take()` site: each one passes through `None` before a new link is installed, or through `fresh_host_lease`. - **P2-1 FIXED.** A doubt is now `LinkDoubt` → `terminal.link_doubt`. It sets no `awaiting_ack`, the window stays confirmed, and replies and input keep flowing (tracer `link-doubt.test.ts` passes). - **P3-1 FIXED.** `queued_path_at_the_ceiling_does_not_flap_doubt_every_probe` passes. The pin is cleared by any reply that brings the allowance below 2 s. - **P3-2 FIXED.** A full window is told on a later tick (`a_window_full_at_doubt_start_is_told_on_a_later_tick` passes). - **P3-3 FIXED.** Recovery no longer sets or keeps the websocket `awaiting_ack` (`link_notice` returns false for both doubt notices). ## Checks that passed (new tests in liveness_doubt_breakit_r2_tests.rs) - Caps negotiation. - Old browser + new server: neither notice is sent (`link_notice(.., false) == None`). The old viewer would throw on an unknown notice, and it never gets one. - New browser + old server: dev and 4e7a8d2e3 test only `caps & RESUME_NEXT`, so bit 1 is ignored. dev has no doubt path at all, so nothing changes for it. - Leg-B bit 1 (`REMOTE_WRITER_BINDING`) is a separate agent↔server channel and does not collide with the browser bit. - Reconnect mid-doubt: `a_window_that_reconnects_mid_doubt_is_told_and_recovered` passes. The new attachment is told once it settles and later gets its recovery. The departed id is dropped and not retried. - Retry bound: `doubt_retry_sends_one_notice_per_window_per_episode` passes. 8 windows, half of them full at the start, over 80+ ticks: each gets exactly 1 LinkDoubt and 0 Recovered. - Ceiling pin vs dead host: `a_pinned_dead_host_retires_within_two_ceilings_without_doubt` passes. The link retires ≤ 4 s + 1 tick after the last probe, with no doubt notice. `retired()` ignores `pinned`. - Status text: `link_recovered` clears only the exact text "Reconnecting…". No other code sets that exact string (the connection layer uses "Connection lost. Reconnecting…"). ## New findings ### P3-4: the doubt indicator is invisible on the canvas (failing test) On the canvas, a terminal's status element is the detached relay element (`canvas/status.ts`). That relay drops routine states that end in "…" or mention "reconnect", and leaves them to the window's status icon. The status icon follows the terminal hooks (`onDisconnected`/`onAttached`), and `link_doubt` calls none of them. So in the product UI a doubted link shows nothing. "Reconnecting…" is visible only on the non-canvas core screen. This is safe, because nothing is held or suppressed. But the "browser shows Reconnecting…" claim does not hold where people actually work. Test: `apps/tracer/test/link-doubt-canvas.test.ts` (fails: no toast, no hook). Fix direction for the owner or the fix job: a hook (for example a doubted connection state) that the window status icon reads. Not a deploy blocker. ### Flaky agent test: REFUTED double close; the root cause is a fork race in the test (failing tests) `Socket` never closes its fd (`Descriptor` has no Drop; transport.rs module doc says so). The test drops `socket` (which deregisters the fd) before `writer` (which closes it). So there is no double close. The recorded failure was "left true, right false": the *unhealthy* write succeeded. The test assumes that `drop(reader)` closes the peer. But any child that a concurrent test forks holds a copy of every fd until it calls exec (CLOEXEC acts only at exec). While that copy exists the socket stays open and the write lands in its buffer. Evidence: - `the_unhealthy_handoff_case_races_concurrent_spawns`: `rounds=20000 spawned=428 write_succeeded=62` (gate rerun: spawned=351 write_succeeded=84) (fails). - `a_dropped_peer_still_accepts_the_write_while_a_forked_copy_holds_it` reproduces it deterministically with a held copy (fails). Fix: call `reader.shutdown(Shutdown::Both)` before the drop. Shutdown acts on the socket itself, not on one descriptor. This is a test-only issue, not a product bug. ## Verdict MERGE: P0 0 / P1 0 / P2 0 / P3 1 (P3-4), plus 1 test-only flake with its root cause found. ## Gate (`scripts/verify fast` at fda0c95f7, verbatim) ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | FAIL (101) | | cargo-deny | PASS | | static builds | PASS | +---------------------------+--------------+ ``` The test step fails only on the two intended failing agent tests (`test result: FAILED. 138 passed; 2 failed; 20 ignored`). Cargo stops at that binary, so the TS step did not run in this gate. The TS failing test is `bun test test/link-doubt-canvas.test.ts` (1 fail; link-doubt.test.ts 4 pass). An earlier gate run at 52231b0c7 also failed `session_exit_events` (idle canary: 1 wakeup in 2.2 s, under load). It is unrelated to this delta.

kayg commented

2026-10-18 02:49:20 +00:00

Author

Owner

release-2 started (production line + #234 D300 ping/FIFO fix + #189 already-added "Open a terminal").
Branch release-2, base origin/release-prod-live d5a4188d3 (Retry + faster add-host + #241).
Merging origin/ping-fifo-fix 8f3945801 with review tests origin/review-ping-fifo-fix-r2 fda0c95f7 (agent flake fixed with reader.shutdown(Shutdown::Both)), then origin/already-added-open de1928b27. Then caps, tests, gate, artifacts, staging proof on the perf VM, Docker sshd add-host. No deploy.

**release-2 started** (production line + #234 D300 ping/FIFO fix + #189 already-added "Open a terminal"). Branch `release-2`, base origin/release-prod-live `d5a4188d3` (Retry + faster add-host + #241). Merging origin/ping-fifo-fix 8f3945801 with review tests origin/review-ping-fifo-fix-r2 fda0c95f7 (agent flake fixed with `reader.shutdown(Shutdown::Both)`), then origin/already-added-open de1928b27. Then caps, tests, gate, artifacts, staging proof on the perf VM, Docker sshd add-host. No deploy.

kayg referenced this issue

2026-10-18 02:49:20 +00:00

Add-host ridiculously fast: under 1.5 s local, under 4 s at 150 ms #189

kayg commented

2026-10-18 06:37:17 +00:00

Author

Owner

release-2 finished. Head 103815e70 (pushed to release-2), base origin/release-prod-live d5a4188d3. No deploy; monitor01 untouched.

Merged (each resolution explained in its merge commit)

5317c3b20: #234 D300 ping-fifo-fix 8f3945801 + r2 review tests fda0c95f7. The flaky agent test now calls reader.shutdown(Shutdown::Both) before the drop. The review's two fork-race tests are now regression tests and pass (a_shut_down_peer_refuses_the_write_while_a_forked_copy_holds_it, the_unhealthy_handoff_case_survives_concurrent_spawns; 3/3 with the original).
c4a814da7: link-doubt-canvas e6387508e (P3-4, web only). link-doubt-canvas.test.ts passes.
4d70ec609: already-added-open de1928b27 (#189). Conflicts were only in the web app (AddHostSink keeps prepareTerminal, enrolled(hosts,status), setupInPill, openHost/hasHost; mock sink; spec watchdog + activeWindow). Rust auto-merged: the duplicate lookup runs before register(), so Retry's lock/refresh and the instant path are untouched.
87676178a: remove-uninstall 27a0a1e65 (#235). Checked against Retry: runtime admission takes the shared parent flock only while admitting a start, so uninstall's bounded exclusive setup_guard cannot deadlock a running infraternal agent.
6c3c5f02f: authority-read-retry a9db5526b (#240). websocket.rs merged cleanly (different functions from D300). The r2 failing P3 test review_authority_cap.rs was left out.
d0422100d + 103815e70: live harness BREAKIT2_READD=1 re-adds the host, checks already_added + host_id, then opens a terminal on that id.
7cdbfc9fd: caps.

Size (same target dir; release-prod-live reproduces its caps exactly)

step	x86_64	aarch64
release-prod-live `d5a4188d3`	21,080,848	19,316,360
+ D300, canvas dot, already-added	21,075,376	19,315,880
+ #235 removal	21,084,144	19,321,064
+ #240 authority retry = caps	21,080,144	19,322,648

Total -704 / +6,288 B (≤ +16 KiB).

Artifacts (`/home/kayg/.cache/codex-scratch/release-2/artifacts`, built at `7cdbfc9fd`; the later commit is test-only)

file	bytes	sha256
infraternal-x86_64-musl	21080144	8608623e55446e9548ee35548254f66b5dccc246dd6f14292018d17c984df6b5
infraternal-aarch64-musl	19322648	fe95a5f415957feef8b00540a8a892434f64a4603fd4e80e79ba3a9ce6489a07
web-dist.tar (265 files)	`3614720`	4d423807f91c69e7d41a30661ed7f0977ec29842459abfae931a9e70a7bf0ff6
bundle/infraternal	21080144	8608623e… (= recipe build)
bundle/infraternal-x86_64-musl	10073536	3fa7c2a37edb9a778c131d7bcf3a8aec83756db90b43bcb0ceed993fd3554f43
bundle/infraternal-aarch64-musl	9473600	cb1367ae9b9c2a94c7b5d9453cc36b5b605b56b8f22224d7f0a0b60cb13ae570
bundle/infraternal-x86_64-session-musl	1869544	72f776f70159c0505b2291ddd4bd7910521c4b5473da40058b12d17f22db2db8
bundle/infraternal-aarch64-session-musl	1632544	b0c4b45d229b1980d8cbac1765bdd103d3b42845fa4ef94332ab0edb4d25f1fc
bundle/SHA256SUMS	456	5094475a12eece6b795b716935f0e78d76069d3b11073ff77a87dbe4d21f0585

Staging

staging-proof (perf VM, lock held): PASS. 12/12 signed handoffs, listener gap p50 4,019 / p99 4,059 ms, probe max 101.6 ms; refusal before shutdown; bad deploy rolled back with backup integrity OK; 128 local sessions; 6 retained browser sockets responsive; teardown clean.
#240 authority stall soak (perf VM, 120 s, 8 terminals typing every 10 ms, stalls 100–800 ms every 1 s): PASS, closes=0 gaps=0 in all three runs: memory/conn (worst echo 829 ms), file/mixed (1,132 ms), file/write (1,034 ms).
Docker sshd add-host with the release bundle payloads:
- instant clean add PASS: prompt 464 ms, fully joined 1,241 ms, same session/writer after handover.
- already-added re-add PASS: refused in 216 ms, already_added=true, host_id = the enrolled host. "Open a terminal" on that id is interactive. The same holds on the full path (full-readd).
- full clean PASS: Done 1,458 ms, installed sha = payload.
- Retry after an injected full-image install failure PASS: Done 1,088 ms.
- remove + re-add rootless PASS: host_cleanup=requested, which is the "…is uninstalling itself." outcome; nothing left; re-add 1,227 ms.
- remove + re-add root with the release host payload: the uninstall worked (requested, left=[]) and the re-add PASSed (3,576 ms). But the harness asserts the seeded foreign mesh objects are named as left, and the host payload is built without the mesh plugin, so it never inspects them. With the mesh-capable server image as payload: PASS (left=["infraternal0","infraternal_mesh"], both kept, everything else removed), and the re-add with the release payload PASS.
- First instant-readd run failed in my new harness step (terminal attached on odd stream 3), after the already-added check had passed. Fixed in 103815e70 and rerun above.

Tests

addhost-wizard Playwright 39/39; tracer bun (src/canvas + link-doubt + link-doubt-canvas) 212/0.
server --features remote lib 539/0; agent lib 65/0; mesh 26/0; session 3/0; infraternal bins 88/0; ssh plugin 214 pass; previous_attempt_shell.py 34 OK.
recovery_support 12/12 with INFRATERNAL_TEST_STATE_ROOT=$HOME/.cache/r2 and the static musl helper. With $HOME/.cache/r2rel, review_live_session_socket_in_newline_home_blocks_cleanup fails, because the socket path exceeds sun_path. That is the environment, not the code; the one ssh-plugin failure was the same test.
cargo test -p infraternal-plugin-remote alone does not compile (tokio rt-multi-thread is only enabled through workspace feature unification). This is pre-existing; the workspace gate runs it.

Gate (`scripts/verify fast` at `103815e70`, RUST_TEST_THREADS=2, INFRATERNAL_TEST_STATE_ROOT=$HOME/.cache/r2, start load 3.9, 73 GB free; exit 0)

| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+

x86_64 21080144 cap 21080144 +0; aarch64 19322648 cap 19322648 +0. TS protocol 37/0.

Evidence: instant-readd.log full-readd.log full-clean.log full-retry.log remove-rootless.log remove-rootless-readd.log remove-root.log remove-root-readd.log remove-root-mesh.log remove-root-mesh-readd.log staging-proof.log authority-stall-soak.log MANIFEST.txt addhost-wizard-playwright.log suites.txt verify-103815e70.log

**release-2 finished.** Head `103815e70` (pushed to `release-2`), base origin/release-prod-live `d5a4188d3`. No deploy; monitor01 untouched. ## Merged (each resolution explained in its merge commit) - 5317c3b20: #234 D300 ping-fifo-fix 8f3945801 + r2 review tests fda0c95f7. The flaky agent test now calls `reader.shutdown(Shutdown::Both)` before the drop. The review's two fork-race tests are now regression tests and pass (`a_shut_down_peer_refuses_the_write_while_a_forked_copy_holds_it`, `the_unhealthy_handoff_case_survives_concurrent_spawns`; 3/3 with the original). - c4a814da7: link-doubt-canvas e6387508e (P3-4, web only). link-doubt-canvas.test.ts passes. - 4d70ec609: already-added-open de1928b27 (#189). Conflicts were only in the web app (AddHostSink keeps prepareTerminal, enrolled(hosts,status), setupInPill, openHost/hasHost; mock sink; spec watchdog + activeWindow). Rust auto-merged: the duplicate lookup runs before register(), so Retry's lock/refresh and the instant path are untouched. - 87676178a: remove-uninstall 27a0a1e65 (#235). Checked against Retry: runtime admission takes the shared parent flock only while admitting a start, so uninstall's bounded exclusive setup_guard cannot deadlock a running infraternal agent. - 6c3c5f02f: authority-read-retry a9db5526b (#240). websocket.rs merged cleanly (different functions from D300). The r2 failing P3 test review_authority_cap.rs was left out. - d0422100d + 103815e70: live harness `BREAKIT2_READD=1` re-adds the host, checks already_added + host_id, then opens a terminal on that id. - 7cdbfc9fd: caps. ## Size (same target dir; release-prod-live reproduces its caps exactly) | step | x86_64 | aarch64 | |---|---:|---:| | release-prod-live d5a4188d3 | 21,080,848 | 19,316,360 | | + D300, canvas dot, already-added | 21,075,376 | 19,315,880 | | + #235 removal | 21,084,144 | 19,321,064 | | + #240 authority retry = **caps** | **21,080,144** | **19,322,648** | Total **-704 / +6,288 B** (≤ +16 KiB). ## Artifacts (`/home/kayg/.cache/codex-scratch/release-2/artifacts`, built at 7cdbfc9fd; the later commit is test-only) | file | bytes | sha256 | |---|---:|---| | infraternal-x86_64-musl | 21080144 | 8608623e55446e9548ee35548254f66b5dccc246dd6f14292018d17c984df6b5 | | infraternal-aarch64-musl | 19322648 | fe95a5f415957feef8b00540a8a892434f64a4603fd4e80e79ba3a9ce6489a07 | | web-dist.tar (265 files) | 3614720 | 4d423807f91c69e7d41a30661ed7f0977ec29842459abfae931a9e70a7bf0ff6 | | bundle/infraternal | 21080144 | 8608623e… (= recipe build) | | bundle/infraternal-x86_64-musl | 10073536 | 3fa7c2a37edb9a778c131d7bcf3a8aec83756db90b43bcb0ceed993fd3554f43 | | bundle/infraternal-aarch64-musl | 9473600 | cb1367ae9b9c2a94c7b5d9453cc36b5b605b56b8f22224d7f0a0b60cb13ae570 | | bundle/infraternal-x86_64-session-musl | 1869544 | 72f776f70159c0505b2291ddd4bd7910521c4b5473da40058b12d17f22db2db8 | | bundle/infraternal-aarch64-session-musl | 1632544 | b0c4b45d229b1980d8cbac1765bdd103d3b42845fa4ef94332ab0edb4d25f1fc | | bundle/SHA256SUMS | 456 | 5094475a12eece6b795b716935f0e78d76069d3b11073ff77a87dbe4d21f0585 | ## Staging - staging-proof (perf VM, lock held): **PASS**. 12/12 signed handoffs, listener gap p50 4,019 / p99 4,059 ms, probe max 101.6 ms; refusal before shutdown; bad deploy rolled back with backup integrity OK; 128 local sessions; 6 retained browser sockets responsive; teardown clean. - #240 authority stall soak (perf VM, 120 s, 8 terminals typing every 10 ms, stalls 100–800 ms every 1 s): **PASS**, closes=0 gaps=0 in all three runs: memory/conn (worst echo 829 ms), file/mixed (1,132 ms), file/write (1,034 ms). - Docker sshd add-host with the release bundle payloads: - instant clean add **PASS**: prompt 464 ms, fully joined 1,241 ms, same session/writer after handover. - already-added re-add **PASS**: refused in 216 ms, already_added=true, host_id = the enrolled host. "Open a terminal" on that id is interactive. The same holds on the full path (full-readd). - full clean **PASS**: Done 1,458 ms, installed sha = payload. - Retry after an injected full-image install failure **PASS**: Done 1,088 ms. - remove + re-add rootless **PASS**: host_cleanup=requested, which is the "…is uninstalling itself." outcome; nothing left; re-add 1,227 ms. - remove + re-add root with the release host payload: the uninstall worked (requested, left=[]) and the re-add **PASS**ed (3,576 ms). But the harness asserts the seeded foreign mesh objects are named as left, and the host payload is built without the mesh plugin, so it never inspects them. With the mesh-capable server image as payload: **PASS** (left=["infraternal0","infraternal_mesh"], both kept, everything else removed), and the re-add with the release payload **PASS**. - First instant-readd run failed in my new harness step (terminal attached on odd stream 3), after the already-added check had passed. Fixed in 103815e70 and rerun above. ## Tests - addhost-wizard Playwright 39/39; tracer bun (src/canvas + link-doubt + link-doubt-canvas) 212/0. - server --features remote lib 539/0; agent lib 65/0; mesh 26/0; session 3/0; infraternal bins 88/0; ssh plugin 214 pass; previous_attempt_shell.py 34 OK. - recovery_support **12/12** with `INFRATERNAL_TEST_STATE_ROOT=$HOME/.cache/r2` and the static musl helper. With `$HOME/.cache/r2rel`, `review_live_session_socket_in_newline_home_blocks_cleanup` fails, because the socket path exceeds sun_path. That is the environment, not the code; the one ssh-plugin failure was the same test. - `cargo test -p infraternal-plugin-remote` alone does not compile (tokio rt-multi-thread is only enabled through workspace feature unification). This is pre-existing; the workspace gate runs it. ## Gate (`scripts/verify fast` at 103815e70, RUST_TEST_THREADS=2, INFRATERNAL_TEST_STATE_ROOT=$HOME/.cache/r2, start load 3.9, 73 GB free; exit 0) ``` | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | PASS | +---------------------------+--------------+ ``` x86_64 21080144 cap 21080144 +0; aarch64 19322648 cap 19322648 +0. TS protocol 37/0. Evidence: [instant-readd.log](https://git.kayg.org/attachments/88279b52-028e-4181-aaf4-4be89b060282) [full-readd.log](https://git.kayg.org/attachments/211e6229-592a-4253-9e5a-5074050ab325) [full-clean.log](https://git.kayg.org/attachments/ab02efaa-a07d-4e52-b463-34f20fcaab58) [full-retry.log](https://git.kayg.org/attachments/1f847ab5-e212-4edf-9705-e12175f5fcd5) [remove-rootless.log](https://git.kayg.org/attachments/ddbbed7c-55dd-48c1-82f8-34a816ea73ba) [remove-rootless-readd.log](https://git.kayg.org/attachments/948314e5-9657-4bbb-adbd-8db242ea875d) [remove-root.log](https://git.kayg.org/attachments/0f5ce31c-12f3-401a-90cb-e657dfb2410a) [remove-root-readd.log](https://git.kayg.org/attachments/03693fd6-2b41-4c13-9e44-d494324b965b) [remove-root-mesh.log](https://git.kayg.org/attachments/c771db84-6e91-401d-afeb-98c68f7ff6a6) [remove-root-mesh-readd.log](https://git.kayg.org/attachments/94064100-acde-4180-ac8c-1fcad8ddbead) [staging-proof.log](https://git.kayg.org/attachments/8318135c-256a-4eb9-84fe-05a86686a377) [authority-stall-soak.log](https://git.kayg.org/attachments/e2c1a814-e67d-4910-baf8-88f6a3ce2fe5) [MANIFEST.txt](https://git.kayg.org/attachments/9ebe880a-feae-468d-b6e2-4b51104fd2f3) [addhost-wizard-playwright.log](https://git.kayg.org/attachments/bd725665-5047-4296-a4e7-13b16e1c8cd0) [suites.txt](https://git.kayg.org/attachments/2036c303-c905-4482-a68a-17fe31e437aa) [verify-103815e70.log](https://git.kayg.org/attachments/f5f73269-9f10-4d37-9c2b-f7e9a86eb5a1)

kayg referenced this issue

2026-10-18 06:37:17 +00:00

Add-host ridiculously fast: under 1.5 s local, under 4 s at 150 ms #189

kayg commented

2026-10-18 06:38:45 +00:00

Author

Owner

Release 2 deployed to production at 14:08 IST: release-prod-live 103815e70, server sha256 8608623e…, MainPID 86026, /healthz ok. Backup /var/backups/infraternal/20261018-083808-103815e70…. CA is locked after the restart until the owner unlocks it.

Release 2 deployed to production at 14:08 IST: release-prod-live 103815e70, server sha256 8608623e…, MainPID 86026, /healthz ok. Backup /var/backups/infraternal/20261018-083808-103815e70…. CA is locked after the restart until the owner unlocks it.

kayg referenced this issue

2026-10-18 12:41:57 +00:00

Phone on LTE: liveness timeouts, then resume refused 33 times a minute (prod) #244

kayg referenced this issue

2026-10-18 18:12:10 +00:00

Busy page retires a healthy socket: 194 drops/min with 65 sessions (prod) #245

kayg referenced this issue

2026-10-20 20:42:58 +00:00

Status board: work in flight (kept current) #24

kayg referenced this issue

2026-10-20 21:49:38 +00:00

Status board: work in flight (kept current) #24

kayg commented

2026-10-28 17:53:22 +00:00

Author

Owner

Closed by the orchestrator during the 2026-10-28 handover (#254), on the owner's request to close finished issues. Evidence: Ping/control-queue fix and link-doubt indicator released in 103815e70 (ancestor of dev and prod, release 2) with measurements. Reopen if anything is still missing.

Closed by the orchestrator during the 2026-10-28 handover (#254), on the owner's request to close finished issues. Evidence: Ping/control-queue fix and link-doubt indicator released in 103815e70 (ancestor of dev and prod, release 2) with measurements. Reopen if anything is still missing.

kayg closed this issue

2026-10-28 17:53:22 +00:00

~~kayg referenced this issue 2026-10-28 17:53:55 +00:00~~

Orchestrator handover 2026-10-28: moving to a new machine #254

kayg referenced this issue

2026-10-28 18:00:45 +00:00

Orchestrator handover 2026-10-28: moving to a new machine #254

Rows
Columns

Scheduled ping closes a healthy host connection when the control queue is full #234

Measurements and root cause (head c179cdcff, perf VM 10.69.69.208, 4 vCPU, under /var/lock/perf-bench.lock)

Fixture

10-minute rows, base origin/dev vs branch (fixture d505e8b35)

Which code closed the primary (cfg(test) caller lines, c179cdcff)

What the branch fixes (each red before, green after, all in the fast gate)

Decision needed (owner; foundational, not decided in a job)

Finish: head c179cdcff (pushed to ping-fifo-fix), scripts/verify fast verbatim

10-minute rows (flood + upload, one 600 s run each)

Dead host (the host stops answering 15 s in, no flood; 5 trials each)

Findings

Limits

Size ratchet and gate

Break-it review (Opus) of ping-fifo-fix 4e7a8d2e3: HOLD (P0 0, P1 1, P2 2, P3 3)

P1-1: a stale doubt confirms the replacement link mid-replay (failing test, fails 5 runs out of 5)

P2-1: live output during a doubt is handled as replay in the browser (code read)

P2-2: retirement timing vs the D300 text (numbers; question for the owner)

P3-1: doubt flaps forever at the 2 s ceiling (failing test)

P3-2: the doubt notice is never retried for a full window

P3-3: Recovered leaves websocket awaiting_ack = true

Checked, no finding

Gate: scripts/verify fast on a7b648166 (the only failures are the 2 review tests)

Break-it fixes on ping-fifo-fix (pushed, head f06512497; gate follows)

Owner note (P2-2, wording only; timings unchanged)

Break-it fixes done: head 8f3945801 (pushed ping-fifo-fix)

Break-it review r2 (Opus) — #234, origin/ping-fifo-fix 8f3945801

Prior findings

Checks that passed (new tests in liveness_doubt_breakit_r2_tests.rs)

New findings

P3-4: the doubt indicator is invisible on the canvas (failing test)

Flaky agent test: REFUTED double close; the root cause is a fork race in the test (failing tests)

Verdict

Gate (scripts/verify fast at fda0c95f7, verbatim)

Merged (each resolution explained in its merge commit)

Size (same target dir; release-prod-live reproduces its caps exactly)

Artifacts (/home/kayg/.cache/codex-scratch/release-2/artifacts, built at 7cdbfc9fd; the later commit is test-only)

Staging

Tests

Gate (scripts/verify fast at 103815e70, RUST_TEST_THREADS=2, INFRATERNAL_TEST_STATE_ROOT=$HOME/.cache/r2, start load 3.9, 73 GB free; exit 0)

Measurements and root cause (head `c179cdcff`, perf VM 10.69.69.208, 4 vCPU, under /var/lock/perf-bench.lock)

10-minute rows, base origin/dev vs branch (fixture `d505e8b35`)

Which code closed the primary (cfg(test) caller lines, `c179cdcff`)

Finish: head `c179cdcff` (pushed to ping-fifo-fix), scripts/verify fast verbatim

Break-it review (Opus) of ping-fifo-fix `4e7a8d2e3`: HOLD (P0 0, P1 1, P2 2, P3 3)

P3-3: Recovered leaves websocket `awaiting_ack = true`

Gate: `scripts/verify fast` on `a7b648166` (the only failures are the 2 review tests)

Break-it fixes on `ping-fifo-fix` (pushed, head `f06512497`; gate follows)

Break-it fixes done: head `8f3945801` (pushed `ping-fifo-fix`)

Break-it review r2 (Opus) — #234, origin/ping-fifo-fix `8f3945801`

Gate (`scripts/verify fast` at `fda0c95f7`, verbatim)

Artifacts (`/home/kayg/.cache/codex-scratch/release-2/artifacts`, built at `7cdbfc9fd`; the later commit is test-only)

Gate (`scripts/verify fast` at `103815e70`, RUST_TEST_THREADS=2, INFRATERNAL_TEST_STATE_ROOT=$HOME/.cache/r2, start load 3.9, 73 GB free; exit 0)