J444: same-session flood stops a stream's output and freezes typing (aarch64), echo p99 +41–148 ms on x86

kayg commented

2026-11-09 15:07:13 +00:00

Owner

J444. J407 found (#447 comment 88884) that on aarch64 (dev-02), with yes & cat in the PTY you type into, the infraternal agent stops sending that stream's output after a 64 KiB warm-up: yes/cat blocked, relay idle, key echo never arrives (typing freeze). origin/dev's own scripts/verify-latency.py (c442500c2) hangs in the same-session phase (exit 124). On x86 the phase finishes but echo p99 grows 41–148 ms, far above D375's +8.3 ms flood row.

Scope:

Root cause (credit/flow-control deadlock in agent/shim, fixture credit drift, or aarch64 ordering), and whether a real browser client can hit it.
Fix where it lives. Keys and echo must never starve under a flood; bounded latency.
Red test on today's dev, green on aarch64 + x86, same-session flood echo p99 vs D375, flood MB/s before/after.
verify quick then verify fast on compute01.

Branch fix-flood-echo-stall from origin/dev be54e662d.

J444. J407 found (#447 comment 88884) that on aarch64 (dev-02), with `yes & cat` in the PTY you type into, the infraternal agent stops sending that stream's output after a 64 KiB warm-up: `yes`/`cat` blocked, relay idle, key echo never arrives (typing freeze). origin/dev's own `scripts/verify-latency.py` (c442500c2) hangs in the same-session phase (exit 124). On x86 the phase finishes but echo p99 grows 41–148 ms, far above D375's +8.3 ms flood row. Scope: 1. Root cause (credit/flow-control deadlock in agent/shim, fixture credit drift, or aarch64 ordering), and whether a real browser client can hit it. 2. Fix where it lives. Keys and echo must never starve under a flood; bounded latency. 3. Red test on today's dev, green on aarch64 + x86, same-session flood echo p99 vs D375, flood MB/s before/after. 4. `verify quick` then `verify fast` on compute01. Branch `fix-flood-echo-stall` from origin/dev be54e662d.

kayg commented

2026-11-09 15:10:45 +00:00

Author

Owner

Start: branch fix-flood-echo-stall from origin/dev be54e662d. Worktrees: code ~/Developer/Worktrees/infraternal-j444, dev-02 ~/worktrees/j444, target ~/.cache/codex-scratch/j444/target.

Plan (est. ~4 h):

Reproduce on dev-02 (aarch64 release musl) with an instrumented copy of the latency fixture that tracks credit granted vs Output+Snapshot bytes received (scratch only, not committed). (~1 h incl. queue)
Root cause from that state. Leads from reading the code:
- scripts/verify-latency.py never returns credit for Snapshot (0x15) bytes, while the agent charges repaints against credit (relay::Output::repaint) and the browser returns it (terminal.ts returnCredit(snapshot.bytes.byteLength)). Each starvation→repaint cycle under yes leaks a repaint's worth of credit from the fixture, and the fixture also holds back <64 KiB. That would be fixture drift.
- Separately, the agent's relay stops reading the shim link while a credit-starved frame is pending (daemon.rs if pending.is_none()), so the shim's InputAck waits behind bulk output, and the shim stops reading the PTY (can_read), which matches J407's dump (yes/cat blocked). Bounded by the 250 ms starvation detach, but that is latency for echo and ack.
Fix where it lives, red test first, measure on aarch64 + x86 (echo p99 growth vs D375 +8.3 ms, flood MB/s before/after). (~2 h)
verify quick, verify fast on compute01. (~1 h + queue)

Start: branch `fix-flood-echo-stall` from origin/dev `be54e662d`. Worktrees: code `~/Developer/Worktrees/infraternal-j444`, dev-02 `~/worktrees/j444`, target `~/.cache/codex-scratch/j444/target`. Plan (est. ~4 h): 1. Reproduce on dev-02 (aarch64 release musl) with an instrumented copy of the latency fixture that tracks credit granted vs Output+Snapshot bytes received (scratch only, not committed). (~1 h incl. queue) 2. Root cause from that state. Leads from reading the code: - `scripts/verify-latency.py` never returns credit for Snapshot (0x15) bytes, while the agent charges repaints against credit (`relay::Output::repaint`) and the browser returns it (`terminal.ts` `returnCredit(snapshot.bytes.byteLength)`). Each starvation→repaint cycle under `yes` leaks a repaint's worth of credit from the fixture, and the fixture also holds back <64 KiB. That would be fixture drift. - Separately, the agent's relay stops reading the shim link while a credit-starved frame is pending (`daemon.rs` `if pending.is_none()`), so the shim's `InputAck` waits behind bulk output, and the shim stops reading the PTY (`can_read`), which matches J407's dump (`yes`/`cat` blocked). Bounded by the 250 ms starvation detach, but that is latency for echo and ack. 3. Fix where it lives, red test first, measure on aarch64 + x86 (echo p99 growth vs D375 +8.3 ms, flood MB/s before/after). (~2 h) 4. `verify quick`, `verify fast` on compute01. (~1 h + queue)

kayg referenced this issue

2026-11-09 16:09:07 +00:00

Latency gate: D375 120 Hz target table in PROCESS §6 and verify latency (J407) #447

kayg referenced this issue

2026-11-09 17:05:32 +00:00

Latency gate: D375 120 Hz target table in PROCESS §6 and verify latency (J407) #447

kayg commented

2026-11-09 17:21:59 +00:00

Author

Owner

Finding: root cause of the aarch64 "freeze". It is the fixture, not an infraternal agent deadlock.

Reproduced on dev-02 (aarch64, release musl from origin/dev be54e662d) with an instrumented copy of the fixture (scratch only). Evidence on dev-02 in ~/.cache/codex-scratch/j444/evidence/.

At the stall the agent still held ~990 KB of unspent credit on the flooding stream (diag-dev-norefund-aarch64.log: outstanding_credit=989772). The fixture's missing Snapshot refunds came to only 1.4 KB. So the credit-leak lead is real but tiny.
A root-side look while the stall was held (/tmp/j444/rootinspect.out, copied in the evidence dir) showed the flooding shim with no data link, draining the PTY normally, and the agent idle in epoll_pwait. J407's dump (yes/cat blocked) is the 250 ms parked window of the cycle below.
A temporary relay trace (relay-trace-aarch64.log, relay-trace2-aarch64.log; not committed) shows the agent never stops. It cycles about every 300 ms: forward the 1 MiB window → credit 0 → 250 ms starvation deadline → detach → relink when credit returns. yes (~150 MB/s) has by then overrun the 8 MiB ring, so recovery is a 365-byte repaint whose cursor jumps ~8.4 MB (e.g. 21.91M → 30.33M).
In 3 of 3 stalls the missing key had 4–7 repaints after it and 35–65 MB of raw output skipped (diag-skip-aarch64-{1,2,3}.log). The fixture counts an echo only as a raw x byte in Output. yes scrolls that x off screen at once, so neither the raw stream nor the repaint can contain it, and the dev fixture waits for ever (exit 124). x86 rarely hits a 250 ms starvation, so it mostly passes.

Can real browsers hit it? Not as a freeze. The browser returns credit for every Output and Snapshot byte once painted (apps/tracer/src/terminal.ts returnCredit, incl. snapshot.bytes.byteLength), the server passes Credit through unchanged (websocket.rs), and the browser renders the repaint, which is the true screen (yes has scrolled the x away there too). Keys keep being acked and written.

Fix (fixture), commit dbc37453a: the same-session phase types into a loop that draws a numbered token on row 1 per key while yes runs in a scroll region below it. The token is read from raw output or from a repaint, the phase has its own quiet baseline, repaint credit is returned, and every key wait is bounded (30 s) with a flow-state message.

What remains is real product latency. On aarch64 with the fixed fixture: quiet p99 325.91 ms, flood p99 546.91 ms, growth +221.00 ms (p50 302.46 → 302.46). The flood runs at 3.41 MB/s (window-limited: 1 MiB per ~300 ms RTT). A key's visible effect in a flooding session waits behind whatever bulk is already in flight or buffered (credit window, shim→agent socket buffer, parked frame). x86 numbers next.

**Finding: root cause of the aarch64 "freeze". It is the fixture, not an infraternal agent deadlock.** Reproduced on dev-02 (aarch64, release musl from origin/dev be54e662d) with an instrumented copy of the fixture (scratch only). Evidence on dev-02 in `~/.cache/codex-scratch/j444/evidence/`. 1. At the stall the agent still held ~990 KB of unspent credit on the flooding stream (`diag-dev-norefund-aarch64.log`: `outstanding_credit=989772`). The fixture's missing Snapshot refunds came to only 1.4 KB. So the credit-leak lead is real but tiny. 2. A root-side look while the stall was held (`/tmp/j444/rootinspect.out`, copied in the evidence dir) showed the flooding shim with **no data link**, draining the PTY normally, and the agent idle in `epoll_pwait`. J407's dump (`yes`/`cat` blocked) is the 250 ms parked window of the cycle below. 3. A temporary relay trace (`relay-trace-aarch64.log`, `relay-trace2-aarch64.log`; not committed) shows the agent never stops. It cycles about every 300 ms: forward the 1 MiB window → credit 0 → 250 ms starvation deadline → detach → relink when credit returns. `yes` (~150 MB/s) has by then overrun the 8 MiB ring, so recovery is a 365-byte repaint whose cursor jumps ~8.4 MB (e.g. 21.91M → 30.33M). 4. In 3 of 3 stalls the missing key had 4–7 repaints after it and 35–65 MB of raw output skipped (`diag-skip-aarch64-{1,2,3}.log`). The fixture counts an echo only as a raw `x` byte in Output. `yes` scrolls that `x` off screen at once, so neither the raw stream nor the repaint can contain it, and the dev fixture waits for ever (exit 124). x86 rarely hits a 250 ms starvation, so it mostly passes. **Can real browsers hit it?** Not as a freeze. The browser returns credit for every Output and Snapshot byte once painted (`apps/tracer/src/terminal.ts` `returnCredit`, incl. `snapshot.bytes.byteLength`), the server passes Credit through unchanged (`websocket.rs`), and the browser renders the repaint, which is the true screen (`yes` has scrolled the `x` away there too). Keys keep being acked and written. **Fix (fixture), commit dbc37453a:** the same-session phase types into a loop that draws a numbered token on row 1 per key while `yes` runs in a scroll region below it. The token is read from raw output or from a repaint, the phase has its own quiet baseline, repaint credit is returned, and every key wait is bounded (30 s) with a flow-state message. **What remains is real product latency.** On aarch64 with the fixed fixture: quiet p99 325.91 ms, flood p99 546.91 ms, **growth +221.00 ms** (p50 302.46 → 302.46). The flood runs at 3.41 MB/s (window-limited: 1 MiB per ~300 ms RTT). A key's visible effect in a flooding session waits behind whatever bulk is already in flight or buffered (credit window, shim→agent socket buffer, parked frame). x86 numbers next.

kayg commented

2026-11-09 18:06:31 +00:00

Author

Owner

Before/after numbers (same release binaries built from origin/dev be54e662d; the product is unchanged on this branch). Logs: dev-02 ~/.cache/codex-scratch/j444/evidence/{dev,new}-fixture-aarch64-unknown-linux-musl-*.log, perf VM ~/j444-out/{dev-fixture-x86-perf.log,new-fixture-x86-perf-{1,2,3}.log} (perf-run receipts in ~/j444-out/receipts/).

run	dev fixture (`c442500c2` / `be54e662d`)	fixed fixture: same-session growth (quiet p99 → flood p99)	flood MB/s
x86 perf VM #1	hang, exit 124	+2.23 ms (324.49 → 326.73)	3.56
x86 perf VM #2		+125.84 ms (325.49 → 451.33)	3.40
x86 perf VM #3		−1.56 ms (328.80 → 327.24)	3.54
aarch64 dev-02 #1–5	hang, exit 124 (3 of 3 runs)	+221.00, +199.97, +9.17, +218.13, +204.73, +35.78 ms (try run + 5)	3.41–3.66

The freeze is not aarch64-specific: dev's fixture also hangs on x86 on the quiet perf VM. J407's x86 pass on compute01 was load-dependent luck.
With the fixed fixture, every key's effect shows on both architectures (no hang, 100/100 keys each run).
Other-session isolation is unchanged and passes (e.g. perf VM −11.48 / −2.87 ms).
Flood throughput: 3.40–3.66 MB/s on both architectures, which is the 1 MiB credit window per ~300 ms RTT. There is no "before" MB/s from dev's fixture because it hangs. The binary is identical, so the flood itself is unchanged.

Why the same-session tail still misses D375's +8.3 ms (product, explained, not fixed here). None of these runs had a repaint (repaints=0), so the tail is plain credit-window queueing. Bulk output spends the whole 1 MiB window in a burst, then the relay sits at zero credit for one credit round trip (~150–300 ms, under the 250 ms starvation deadline). A key whose echo is produced in that gap waits behind the queued bulk until refunds return, up to ~1 RTT. That's why p50 never moves (302 → 302 ms) while a few percent of keys add 100–220 ms. On top of the window, the queue ahead of the echo includes the PTY buffer, the shim→agent socket buffer and one parked frame.

Design question for kayg (owner call: it trades raw output for latency), with my recommendation:

A (recommended): input-triggered catch-up. When a key is written while that stream's output is credit-starved, the relay skips the queued bulk and asks the shim for a repaint now, paid from a small credit reserve that bulk may not spend (e.g. 64 KiB of the 1 MiB window). Expected: echo ≈ RTT + about one frame, inside +8.3 ms. Cost: up to ~6 % less bulk throughput on window-limited links, only while flooding, and the raw bytes produced before the key are replaced by the screen. That's exactly what the existing 250 ms starvation path already does, just triggered by typing instead of waiting.
B: a bigger browser window (INITIAL_CREDIT 1 MiB → 4 MiB). Fewer starvation gaps on high-BDP links but a longer queue ahead of the echo when bandwidth-bound. Not recommended.
C: keep it, and scope D375's flood row to other sessions; report same-session as REPORT only.

Until decided, the same-session rows in scripts/verify latency report this tail as FAIL (instead of hanging). That row is not part of verify fast.

**Before/after numbers** (same release binaries built from origin/dev be54e662d; the product is unchanged on this branch). Logs: dev-02 `~/.cache/codex-scratch/j444/evidence/{dev,new}-fixture-aarch64-unknown-linux-musl-*.log`, perf VM `~/j444-out/{dev-fixture-x86-perf.log,new-fixture-x86-perf-{1,2,3}.log}` (perf-run receipts in `~/j444-out/receipts/`). | run | dev fixture (c442500c2 / be54e662d) | fixed fixture: same-session growth (quiet p99 → flood p99) | flood MB/s | |---|---|---|---| | x86 perf VM #1 | **hang, exit 124** | +2.23 ms (324.49 → 326.73) | 3.56 | | x86 perf VM #2 | | +125.84 ms (325.49 → 451.33) | 3.40 | | x86 perf VM #3 | | −1.56 ms (328.80 → 327.24) | 3.54 | | aarch64 dev-02 #1–5 | **hang, exit 124** (3 of 3 runs) | +221.00, +199.97, +9.17, +218.13, +204.73, +35.78 ms (try run + 5) | 3.41–3.66 | - The freeze is **not aarch64-specific**: dev's fixture also hangs on x86 on the quiet perf VM. J407's x86 pass on compute01 was load-dependent luck. - With the fixed fixture, every key's effect shows on both architectures (no hang, 100/100 keys each run). - Other-session isolation is unchanged and passes (e.g. perf VM −11.48 / −2.87 ms). - Flood throughput: 3.40–3.66 MB/s on both architectures, which is the 1 MiB credit window per ~300 ms RTT. There is no "before" MB/s from dev's fixture because it hangs. The binary is identical, so the flood itself is unchanged. **Why the same-session tail still misses D375's +8.3 ms (product, explained, not fixed here).** None of these runs had a repaint (`repaints=0`), so the tail is plain credit-window queueing. Bulk output spends the whole 1 MiB window in a burst, then the relay sits at zero credit for one credit round trip (~150–300 ms, under the 250 ms starvation deadline). A key whose echo is produced in that gap waits behind the queued bulk until refunds return, up to ~1 RTT. That's why p50 never moves (302 → 302 ms) while a few percent of keys add 100–220 ms. On top of the window, the queue ahead of the echo includes the PTY buffer, the shim→agent socket buffer and one parked frame. **Design question for kayg** (owner call: it trades raw output for latency), with my recommendation: - **A (recommended): input-triggered catch-up.** When a key is written while that stream's output is credit-starved, the relay skips the queued bulk and asks the shim for a repaint now, paid from a small credit reserve that bulk may not spend (e.g. 64 KiB of the 1 MiB window). Expected: echo ≈ RTT + about one frame, inside +8.3 ms. Cost: up to ~6 % less bulk throughput on window-limited links, only while flooding, and the raw bytes produced before the key are replaced by the screen. That's exactly what the existing 250 ms starvation path already does, just triggered by typing instead of waiting. - B: a bigger browser window (INITIAL_CREDIT 1 MiB → 4 MiB). Fewer starvation gaps on high-BDP links but a longer queue ahead of the echo when bandwidth-bound. Not recommended. - C: keep it, and scope D375's flood row to other sessions; report same-session as REPORT only. Until decided, the same-session rows in `scripts/verify latency` report this tail as FAIL (instead of hanging). That row is not part of `verify fast`.

kayg commented

2026-11-09 18:25:30 +00:00

Author

Owner

Progress: head c9bbda442 (origin/dev 547a48885 merged; the branch changes only scripts/verify-latency.py and the new apps/tracer/real/same-session-flood.spec.ts). Queued: verify quick + verify fast in one compute01 slot (log ~/.cache/codex-scratch/j444/gate.log, target ~/.cache/codex-scratch/j444/target). Real-browser spec on dev-02 against the aarch64 build (log ~/.cache/codex-scratch/j444/browser.log, target ~/.cache/codex-scratch/j444/target). Both queues are long.

Progress: head `c9bbda442` (origin/dev 547a48885 merged; the branch changes only `scripts/verify-latency.py` and the new `apps/tracer/real/same-session-flood.spec.ts`). Queued: `verify quick` + `verify fast` in one compute01 slot (log `~/.cache/codex-scratch/j444/gate.log`, target `~/.cache/codex-scratch/j444/target`). Real-browser spec on dev-02 against the aarch64 build (log `~/.cache/codex-scratch/j444/browser.log`, target `~/.cache/codex-scratch/j444/target`). Both queues are long.

kayg referenced this issue

2026-11-09 18:52:22 +00:00

Latency gate: D375 120 Hz target table in PROCESS §6 and verify latency (J407) #447

kayg referenced this issue

2026-11-09 19:40:54 +00:00

Latency gate: D375 120 Hz target table in PROCESS §6 and verify latency (J407) #447

kayg referenced this issue

2026-11-10 04:42:26 +00:00

Latency gate: D375 120 Hz target table in PROCESS §6 and verify latency (J407) #447

kayg commented

2026-11-10 05:03:34 +00:00

Author

Owner

Resumed (2026-11-10). The x86/aarch64 flood tail is the product: bulk output gets priority over keys.

Fixture credit batching (fixed, d405440f5). The fixture returned credit in 64 KiB batches; the tracer returns it after each painted write. Batching inflated the tail (dev-02 variants, evidence/var-*.log: 64 KiB +112.32 ms / per frame +27.76 ms / 4 MiB window +27.06 ms / RTT 0 +6.42 ms). The fixture now returns each frame's credit as the browser does.
What remains is the product. Per-frame fixture, same binaries:
- x86 perf VM (perf-run, ~/j444-out/after-x86-{1,2,3}.log): growth +179.36 / +166.54 / +153.29 ms, and the Ack p99 also fails.
- aarch64 dev-02 (evidence/after-arm-{1,2,3}.log): +228.34 / +234.49 / +55.19 ms, Ack failing too.
- Flood 3.50–3.67 MB/s; other-session isolation passes (−4.76..+1.04 ms).
- Per key (x86, ~/j444-out/var-x86.log): slow keys sit ~+200 ms above quiet (echo 500–511 ms vs ~300). Some have a late Ack too (k79 ack 499, k69 ack 493).
Mechanism. Bulk spends the whole 1 MiB window in a burst. The relay then parks on a credit-starved frame, so it neither sends output nor reads the shim link. The key is written to the PTY at once, but its InputAck (queued behind output on the shim link) and its echo (behind bulk in the PTY/socket) both wait for refunds, about one credit round trip.
Real browser (aarch64, production tracer + real server, 150 ms each way), evidence/browser3.log, screenshot attached: no freeze, all 40 keys shown and acked. But the flood moves the median: quiet p50 155.6 ms → flood p50 332.5 ms, max 343.6 (bridge-observed). A real person's keys aren't synchronised with refunds the way the fixture's are, so most keys land in the gap. The fixture's p99 understates what people feel.

Why I stopped short of changing the relay. The only fix that reaches +8.3 ms is skipping queued bulk when a key arrives:

A credit reserve alone would have to cover the whole queue ahead of the echo (PTY buffer + shim→agent socket ≈ 200+ KB, i.e. ~25 % of the window, and that much less flood throughput).
So the fix is: on Input while output is credit-starved, discard the parked bulk, wait for the shim to write the key, then send a repaint paid from a small reserve (e.g. 64 KiB). The InputAck travels with it.

That changes what clients receive (raw bytes produced before the key become a repaint, as the 250 ms starvation path already does today), and it lives in the relay/shim flow-control path, which gets the strictest review. Estimate: ~1 day including shim+relay unit tests, the fixture and browser proofs, and review. Needs a go from kayg/the orchestrator: recommend A (input-triggered catch-up + reserve). Alternatives: B, a bigger browser window (fewer gaps, longer queue when bandwidth-bound), or C, scope D375's flood row to other sessions.

**Resumed (2026-11-10). The x86/aarch64 flood tail is the product: bulk output gets priority over keys.** 1. **Fixture credit batching (fixed, d405440f5).** The fixture returned credit in 64 KiB batches; the tracer returns it after each painted write. Batching inflated the tail (dev-02 variants, `evidence/var-*.log`: 64 KiB +112.32 ms / per frame +27.76 ms / 4 MiB window +27.06 ms / RTT 0 +6.42 ms). The fixture now returns each frame's credit as the browser does. 2. **What remains is the product.** Per-frame fixture, same binaries: - x86 perf VM (`perf-run`, `~/j444-out/after-x86-{1,2,3}.log`): growth **+179.36 / +166.54 / +153.29 ms**, and the **Ack** p99 also fails. - aarch64 dev-02 (`evidence/after-arm-{1,2,3}.log`): **+228.34 / +234.49 / +55.19 ms**, Ack failing too. - Flood 3.50–3.67 MB/s; other-session isolation passes (−4.76..+1.04 ms). - Per key (x86, `~/j444-out/var-x86.log`): slow keys sit ~+200 ms above quiet (echo 500–511 ms vs ~300). Some have a late Ack too (k79 ack 499, k69 ack 493). 3. **Mechanism.** Bulk spends the whole 1 MiB window in a burst. The relay then parks on a credit-starved frame, so it neither sends output nor reads the shim link. The key is written to the PTY at once, but its InputAck (queued behind output on the shim link) and its echo (behind bulk in the PTY/socket) both wait for refunds, about one credit round trip. 4. **Real browser (aarch64, production tracer + real server, 150 ms each way), `evidence/browser3.log`, screenshot attached:** no freeze, all 40 keys shown and acked. But the flood moves the **median**: quiet p50 155.6 ms → flood p50 332.5 ms, max 343.6 (bridge-observed). A real person's keys aren't synchronised with refunds the way the fixture's are, so most keys land in the gap. The fixture's p99 understates what people feel. **Why I stopped short of changing the relay.** The only fix that reaches +8.3 ms is skipping queued bulk when a key arrives: - A credit reserve alone would have to cover the whole queue ahead of the echo (PTY buffer + shim→agent socket ≈ 200+ KB, i.e. ~25 % of the window, and that much less flood throughput). - So the fix is: on Input while output is credit-starved, discard the parked bulk, wait for the shim to write the key, then send a repaint paid from a small reserve (e.g. 64 KiB). The InputAck travels with it. That changes what clients receive (raw bytes produced before the key become a repaint, as the 250 ms starvation path already does today), and it lives in the relay/shim flow-control path, which gets the strictest review. Estimate: ~1 day including shim+relay unit tests, the fixture and browser proofs, and review. **Needs a go from kayg/the orchestrator:** recommend A (input-triggered catch-up + reserve). Alternatives: B, a bigger browser window (fewer gaps, longer queue when bandwidth-bound), or C, scope D375's flood row to other sessions.

same-session-flood.png

58 KiB

kayg referenced this issue

2026-11-10 09:41:54 +00:00

Latency gate: D375 120 Hz target table in PROCESS §6 and verify latency (J407) #447

kayg commented

2026-11-10 09:53:37 +00:00

Author

Owner

Finish (J444). Branch fix-flood-echo-stall, head d405440f5 (git ls-remote: d405440f591663d83ae738c6b9ecae060d8bd374 refs/heads/fix-flood-echo-stall), base origin/dev 97928be8a. Not merged.

Root cause of the freeze: the fixture. Under yes the agent skips raw output for a repaint after 250 ms of credit starvation; yes has scrolled the echoed x away, so it is in neither form, and the fixture waited for ever. dev's fixture hangs on both aarch64 and x86 (perf VM).
Fixed: the D375 flood row now gives a number on both architectures (row-1 token echo, repaint credit returned, per-frame credit like the tracer, bounded waits).
Real browser: no freeze (spec passes, screenshot attached above). But the flood moves its p50 by +177 ms.
Not fixed: the product's same-session tail (+153..+234 ms; Ack too). It needs the relay change proposed in comment 89401, which is waiting on a go.

verify quick (compute01, HEAD d405440f5):

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| scope guard               | PASS         |
| migration names           | PASS         |
| Nix Cargo vendoring       | PASS         |
| VT source recipe          | PASS         |
| fmt                       | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| ui literals               | PASS         |
| tracer typecheck          | PASS         |
QUICK_EXIT=0

verify fast (compute01, same run):

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | FAIL (101)   |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| privileged integration    | SKIP         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| nightly browser suite reporter | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
VERIFY_EXIT=1

The test row's only failure was crates/plugins/ssh/tests/lease_progress.rs::remote_payload_progress_renews_the_lease_but_idle_does_not (panicked at :169). This branch changes no Rust (diff vs merge base: scripts/verify-latency.py, apps/tracer/real/same-session-flood.spec.ts), so that is origin/dev's code. It then passed 5/5 alone on compute01. The previous run's shim process-per-user failure (exhausted_pool_drains_live_ptys_during_random_session_churn) passed in this run and 3/3 alone. Log: compute01 ~/.cache/codex-scratch/j444/evidence/gate2.log. Targets on compute01 and dev-02 deleted.

kayg commented

2026-11-10 15:15:37 +00:00

Author

Owner

Start (J444b): branch j444b-d381-key-wins-flood from fix-flood-echo-stall at base d405440f591663d83ae738c6b9ecae060d8bd374. Implementing D381 with reserved repaint credit, red-first relay invariants, x86/aarch64 measurements, and the real-browser 150 ms each-way check. No merge or deploy.

Start (J444b): branch `j444b-d381-key-wins-flood` from `fix-flood-echo-stall` at base `d405440f591663d83ae738c6b9ecae060d8bd374`. Implementing D381 with reserved repaint credit, red-first relay invariants, x86/aarch64 measurements, and the real-browser 150 ms each-way check. No merge or deploy.

kayg commented

2026-11-10 15:35:56 +00:00

Author

Owner

J444b plan and implementation choice: use a 64 KiB repaint reserve in addition to the existing 1 MiB bulk allowance, so a flood can still spend the full bulk window. The browser grants both at attach; the relay never creates credit, and only actual client-returned credit replenishes the window. On a key during a credit-starved Output, discard only that session’s pending bulk frame, wait for the shim InputAck, then request one repaint; coalesce keys while a repaint is in flight. Keep the existing 250 ms recovery timer and path unchanged. Tests will cover credit, input/repaint order, sibling-session output, and spam bounded by unspent credit. Then measure the fixed fixture on perf-run x86 and dev-02 aarch64, run the real browser check at 150 ms each way, and run verify fast on compute01. Estimate: one working day including shared-VM queues.

kayg commented

2026-11-10 16:32:53 +00:00

Author

Owner

D381 invariant tests are red against d405440f5:

reserved_credit_preserves_the_bulk_window_and_cannot_be_overspent sees raw output use 1,114,112 bytes from a 1 MiB + 64 KiB grant; the 1 MiB bulk limit should stop it at 1,048,576. Evidence: /home/kayg/.cache/codex-scratch/j444b/evidence/red-reserve.log (dev-02).
key_repaint_waits_for_its_input_ack_and_contains_the_echo, key_repaint_does_not_interrupt_sibling_session_output, and key_spam_cannot_mint_repaint_credit fail because no key-triggered repaint request is released through the current credit-stalled read path. Evidence: /home/kayg/.cache/codex-scratch/j444b/evidence/red-daemon-invariants.log (dev-02).

I’m implementing the 64 KiB repaint floor and an InputAck-gated, per-stream catch-up path. The existing 250 ms recovery path remains the fallback.

D381 invariant tests are red against d405440f5: - `reserved_credit_preserves_the_bulk_window_and_cannot_be_overspent` sees raw output use 1,114,112 bytes from a 1 MiB + 64 KiB grant; the 1 MiB bulk limit should stop it at 1,048,576. Evidence: `/home/kayg/.cache/codex-scratch/j444b/evidence/red-reserve.log` (dev-02). - `key_repaint_waits_for_its_input_ack_and_contains_the_echo`, `key_repaint_does_not_interrupt_sibling_session_output`, and `key_spam_cannot_mint_repaint_credit` fail because no key-triggered repaint request is released through the current credit-stalled read path. Evidence: `/home/kayg/.cache/codex-scratch/j444b/evidence/red-daemon-invariants.log` (dev-02). I’m implementing the 64 KiB repaint floor and an InputAck-gated, per-stream catch-up path. The existing 250 ms recovery path remains the fallback.

kayg commented

2026-11-10 16:56:24 +00:00

Author

Owner

Finding: the first J444b verify quick run reached Clippy and reported seven issues: two collapsible conditionals, two lossless u32-to-u64 casts, one doc-markdown warning, and two identical test match arms. Evidence: /home/kayg/.cache/codex-scratch/j444b/evidence/quick.log. I applied the lint-only changes and will rerun quick on dev-02. The red invariant evidence against d405440f5 remains in red-reserve.log and red-daemon-invariants.log.

Finding: the first J444b verify quick run reached Clippy and reported seven issues: two collapsible conditionals, two lossless u32-to-u64 casts, one doc-markdown warning, and two identical test match arms. Evidence: /home/kayg/.cache/codex-scratch/j444b/evidence/quick.log. I applied the lint-only changes and will rerun quick on dev-02. The red invariant evidence against d405440f5 remains in red-reserve.log and red-daemon-invariants.log.

kayg commented

2026-11-10 17:00:18 +00:00

Author

Owner

Progress: merged current origin/dev (7d4f1a4b69) into the J444b work branch so the final gate uses the current baseline. Pushed only j444b-d381-key-wins-flood; git ls-remote confirms d4a1a9173c. The D381 changes remain uncommitted; next I am rerunning quick with the lint fixes, then the architecture, browser, and fast-gate proofs.

Progress: merged current origin/dev (7d4f1a4b69ad9d4e68a9e848b8c0634bb39f1d0b) into the J444b work branch so the final gate uses the current baseline. Pushed only j444b-d381-key-wins-flood; git ls-remote confirms d4a1a9173c808218b9f8c9940c9c8217ed884e3e. The D381 changes remain uncommitted; next I am rerunning quick with the lint fixes, then the architecture, browser, and fast-gate proofs.

Rows
Columns

J444: same-session flood stops a stream's output and freezes typing (aarch64), echo p99 +41–148 ms on x86 #477