kayg/infraternal
Watch
1
0
Fork
You've already forked infraternal
 0
Code Issues 356 Pull requests Projects Releases Packages Wiki Activity Actions

Machine pill indicator flickers while a full-screen app redraws (production) #227

New issue
Open
opened 2026-10-16 05:35:17 +00:00 by kayg · 150 comments
kayg commented 2026-10-16 05:35:17 +00:00
Owner
Copy link

Owner report 2026-10-16 (video while signing in to agy on production, 492e1505a binary + f210a4f25 web): the indicator dot in a terminal window's machine pill ("kayg@10.69.69.125 · shell ●") keeps flickering.

Measured from the owner's 120 fps recording (12×12 px at the dot, luma per frame): the dot sits at Y≈80 (grey) and jumps to Y≈133 for 1–2 frames about every 14 frames, ≈8.5 flashes/s, steadily over 6.5 s while agy's sign-in TUI is on screen (it redraws continuously). So something toggles the dot's state on every output burst / redraw.

Expected: the dot shows a stable state; it never blinks with output. (TASTE: results never animate; nothing flickers.)

Owner report 2026-10-16 (video while signing in to agy on production, 492e1505a binary + f210a4f25 web): the indicator dot in a terminal window's machine pill ("kayg@10.69.69.125 · shell ●") keeps flickering. Measured from the owner's 120 fps recording (12×12 px at the dot, luma per frame): the dot sits at Y≈80 (grey) and jumps to Y≈133 for 1–2 frames about every 14 frames, ≈8.5 flashes/s, steadily over 6.5 s while agy's sign-in TUI is on screen (it redraws continuously). So something toggles the dot's state on every output burst / redraw. Expected: the dot shows a stable state; it never blinks with output. (TASTE: results never animate; nothing flickers.)
mock-reproduction.log
16 KiB
real-reproduction.log
16 KiB
agy-control-summary.txt
811 B
real-redraw-repro-summary.txt
1.4 KiB
real-synthetic-baseline-summary.txt
1.2 KiB
real-synthetic-baseline.log
43 KiB
agy-control-replay-summary.txt
1.4 KiB
agy-control-replay.log
110 KiB
close-origin-summary.txt
761 B
agy-control-replay-close-origin.log
103 KiB
web-current-size.json
613 B
connection-liveness-before.log
873 B
connection-liveness-after.log
1.1 KiB
diag-agy-492e-summary.json
2.7 KiB
diag-agy-492e-real-server.log
106 KiB
proof-detail-492e-summary.txt
1.2 KiB
mock-baseline-vs-branch.txt
979 B
real-demo-summary.txt
661 B
pill-flicker-demo.mp4
416 KiB
pill-flicker-frame.png
337 KiB
finding-dist-build.txt
420 B
perf-frame-count-finding.txt
367 B
perf-cost-summary.txt
1.1 KiB
perf-cost-trials.csv
850 B
size-summary.txt
978 B
transfer-base-fixed.txt
6.5 KiB
transfer-branch-fixed.txt
6.5 KiB
real-query-summary.txt
1.3 KiB
agy-plain-run-summary.txt
948 B
agy-interactive-summary.txt
1 KiB
pill-flicker-demo.mp4
831 KiB
pill-flicker-demo.png
214 KiB
production-redraw-demo-summary.txt
1 KiB
mock-browser-final-summary.txt
631 B
iterate-final.log
301 KiB
browser-mock-final.log
15 KiB
browser-real-final.log
56 KiB
verify-fast-final.log
266 KiB
baseline-liveness-test.log
2.8 KiB
verify-ts-final.log
29 KiB
size-comparison-normalized.txt
577 B
pill-flicker-demo-final.mp4
736 KiB
pill-flicker-demo-final.png
332 KiB
real-demo-final.log
40 KiB
base-offline.png
37 KiB
head-offline.png
43 KiB
head-live.png
44 KiB
base-10s-network.log
1.3 KiB
head-10s-network.log
1.3 KiB
base-redraw-1.log
69 KiB
head-redraw-1.log
70 KiB
fast-gate.log
295 KiB
ts-fast.log
30 KiB
rust-single-retry-features.log
4.1 KiB
playwright.log
16 KiB
fast.log
288 KiB
dist-comparison.log
15 KiB
pill-flicker-real-server-final.mp4
567 KiB
real-server-redraw.log
57 KiB
network-debug-base.log
11 KiB
network-debug-perf-vm.log
12 KiB
network-debug-perf-vm-isolated.log
3.1 KiB
control-replay-base.log
57 KiB
network-recovery-before-fix.log
8.6 KiB
pill-flicker-real-server-after-fix.mp4
394 KiB
mock-flicker-final-summary.log
366 B
pill-flicker-real-server-final.mp4
376 KiB
network-recovery-final.log
150 KiB
network-recovery-prod492e-20.log
296 KiB
mock-final-browser.log
15 KiB
network-recovery-prod492e.log
150 KiB
network-recovery-prod492e-final2.log
148 KiB
pill-flicker-real-server-prod492e-final.mp4
389 KiB
pill-flicker-final-frame.png
341 KiB
real-final-browser.log
56 KiB
real-server-close-diagnostic.log
126 KiB
perf-cost-summary.log
2.3 KiB
perf-cost-prod492e-final2.log
1.5 MiB
verify-fast-final.log
288 KiB
verify-fast-final-clean.log
208 KiB
connection-restart.log
783 B
typescript-gate.log
31 KiB
agy-reproduction.log
14 KiB
pill-flicker-frame.png pill-flicker-demo.png pill-flicker-demo-final.png base-offline.png head-offline.png head-live.png pill-flicker-final-frame.png
kayg commented 2026-10-16 05:36:44 +00:00
Author
Owner
Copy link

Start: branch pill-flicker, base f210a4f25055a1c6ee4e624e36f99f3f20ead640 (origin/dev). Read all #227 comments (0); issue metadata exposes no parent issue link. Reproducing the reported state change before editing.

Start: branch `pill-flicker`, base `f210a4f25055a1c6ee4e624e36f99f3f20ead640` (origin/dev). Read all #227 comments (0); issue metadata exposes no parent issue link. Reproducing the reported state change before editing.
kayg commented 2026-10-16 05:41:11 +00:00
Author
Owner
Copy link

Mock baseline finding: the product canvas over the mock PTY received 19 alternate-screen, cursor-addressed redraw frames at 111 ms intervals (9.0 frames/s) over 2.1 s. The machine pill stayed data-status=live; the dot stayed rgb(62, 158, 41) for all 151 animation-frame samples, with zero status or dot-class transitions. This workload does not reproduce the production flicker in the test double. Receipt: mock-reproduction.log.

Mock baseline finding: the product canvas over the mock PTY received 19 alternate-screen, cursor-addressed redraw frames at 111 ms intervals (9.0 frames/s) over 2.1 s. The machine pill stayed `data-status=live`; the dot stayed `rgb(62, 158, 41)` for all 151 animation-frame samples, with zero status or dot-class transitions. This workload does not reproduce the production flicker in the test double. Receipt: [mock-reproduction.log](https://git.kayg.org/attachments/abf70cf4-8de6-4a75-8180-7cfe9a928ad1).
kayg commented 2026-10-16 06:07:26 +00:00
Author
Owner
Copy link

Real-server redraw finding: the 19-frame ANSI workload (~9 Hz by the scripted 111 ms cadence) produced one brief live → connecting → live interval (~67 ms in this host trace), not the reported ≥10 flips in 2 s. The final bridge record shows two protocol sockets, each with one AttachTerm and one replay. body.ts maps the pill directly from the binding's connected flag; I’m tracing the connection transition source next. Real trace

Real-server redraw finding: the 19-frame ANSI workload (~9 Hz by the scripted 111 ms cadence) produced one brief live → connecting → live interval (~67 ms in this host trace), not the reported ≥10 flips in 2 s. The final bridge record shows two protocol sockets, each with one AttachTerm and one replay. `body.ts` maps the pill directly from the binding's connected flag; I’m tracing the connection transition source next. [Real trace](https://git.kayg.org/attachments/8a572f8e-0424-485b-a43a-f8cba424ab15)
kayg commented 2026-10-16 06:14:45 +00:00
Author
Owner
Copy link

I ran the actual agy TUI for 2 s inside the real-server PTY with a fresh empty HOME. The 2.5 s status/frame trace recorded no pill mutations; the repeated socket handoff had already settled before tracing. The reported ≥10 flips/2 s still does not reproduce here. Sanitized agy trace

I ran the actual `agy` TUI for 2 s inside the real-server PTY with a fresh empty HOME. The 2.5 s status/frame trace recorded no pill mutations; the repeated socket handoff had already settled before tracing. The reported ≥10 flips/2 s still does not reproduce here. [Sanitized agy trace](https://git.kayg.org/attachments/72833bc0-023f-4d74-ab99-ae8c5fe82054)
kayg commented 2026-10-16 06:20:59 +00:00
Author
Owner
Copy link

Correction to the prior finding: the ~67 ms interval in that attached raw trace came from the shared build host and is diagnostic only, not performance evidence. I will use the perf VM for all reported frame-rate and timing measurements.

Correction to the prior finding: the ~67 ms interval in that attached raw trace came from the shared build host and is diagnostic only, not performance evidence. I will use the perf VM for all reported frame-rate and timing measurements.
kayg commented 2026-10-16 06:41:31 +00:00
Author
Owner
Copy link

Connection finding (correction to the prior sanitized run): a later real-server trace recorded one live → connecting → live pair. Its bridge records show a server-closed socket followed by Hello/Resume/AttachTerm; the pill is driven by the binding's connected state, which changes on disconnect/reattach. The isolated agy invocation emitted one screen render (2,158 bytes) and no periodic redraw, so it still does not reproduce the reported rate. Control and connection summary

Connection finding (correction to the prior sanitized run): a later real-server trace recorded one live → connecting → live pair. Its bridge records show a server-closed socket followed by Hello/Resume/AttachTerm; the pill is driven by the binding's connected state, which changes on disconnect/reattach. The isolated `agy` invocation emitted one screen render (2,158 bytes) and no periodic redraw, so it still does not reproduce the reported rate. [Control and connection summary](https://git.kayg.org/attachments/5a31b622-4a00-4f9b-94e0-73d7870ba2b0)
kayg commented 2026-10-16 06:44:58 +00:00
Author
Owner
Copy link

Trace step completed at f210a4f25055a1c6ee4e624e36f99f3f20ead640: the mock and real harnesses now use 24×80 cursor-addressed redraws; the real trace records pill mutations, per-frame samples, output-frame times and socket closes. No product code changed.

bun run check (exit 0):

$ bun x tsc --project tsconfig.json
Trace step completed at `f210a4f25055a1c6ee4e624e36f99f3f20ead640`: the mock and real harnesses now use 24×80 cursor-addressed redraws; the real trace records pill mutations, per-frame samples, output-frame times and socket closes. No product code changed. `bun run check` (exit 0): ``` $ bun x tsc --project tsconfig.json ```
kayg commented 2026-10-16 06:53:47 +00:00
Author
Owner
Copy link

Measurement correction: all frame rates, RAF sample counts and elapsed intervals in the earlier mock/real host traces are diagnostic only; the shared build host is not the source of performance evidence. I’ll use the perf VM for the required timing and frame-rate results.

Measurement correction: all frame rates, RAF sample counts and elapsed intervals in the earlier mock/real host traces are diagnostic only; the shared build host is not the source of performance evidence. I’ll use the perf VM for the required timing and frame-rate results.
kayg commented 2026-10-16 07:04:01 +00:00
Author
Owner
Copy link

Trace harness update at f210a4f250: the real-server spec launches a generated PTY script by short path, records pill mutations and frame samples for a configurable window, and timestamps output frames/socket closes. bun run check passed ($ bun x tsc --project tsconfig.json). No product code changed. The next run will measure the actual TUI and synthetic redraw on the perf VM.

Trace harness update at f210a4f25055a1c6ee4e624e36f99f3f20ead640: the real-server spec launches a generated PTY script by short path, records pill mutations and frame samples for a configurable window, and timestamps output frames/socket closes. `bun run check` passed (`$ bun x tsc --project tsconfig.json`). No product code changed. The next run will measure the actual TUI and synthetic redraw on the perf VM.
kayg commented 2026-10-16 07:10:26 +00:00
Author
Owner
Copy link

Real trace instrumentation now timestamps terminal recovery notices, stream closes, and AttachTerm requests alongside pill mutations and output frames. bun run check passed ($ bun x tsc --project tsconfig.json). The isolated TUI executable and its runtime libraries are staged under the perf VM job directory; the next capture runs there with no inherited credentials.

Real trace instrumentation now timestamps terminal recovery notices, stream closes, and AttachTerm requests alongside pill mutations and output frames. `bun run check` passed (`$ bun x tsc --project tsconfig.json`). The isolated TUI executable and its runtime libraries are staged under the perf VM job directory; the next capture runs there with no inherited credentials.
kayg commented 2026-10-16 07:32:32 +00:00
Author
Owner
Copy link

Production-equivalent server artifact built at f210a4f250. Server crates are unchanged from production commit 492e1505a. Static x86_64 binary: 21,422,096 bytes; SHA-256 beb50f0804f14f6ac85c712b613362f2047a95d4f874418b88b037cc8fb0d603. Build output: Finished release profile [optimized] target(s) in 24m 01s.

Production-equivalent server artifact built at f210a4f25055a1c6ee4e624e36f99f3f20ead640. Server crates are unchanged from production commit 492e1505a. Static x86_64 binary: 21,422,096 bytes; SHA-256 `beb50f0804f14f6ac85c712b613362f2047a95d4f874418b88b037cc8fb0d603`. Build output: `Finished `release` profile [optimized] target(s) in 24m 01s`.
kayg commented 2026-10-16 08:18:52 +00:00
Author
Owner
Copy link

Perf VM capture has waited 45 minutes for /var/lock/perf-bench.lock; it remains queued and I am continuing to wait. Current /var/lock/perf-bench.owner contents:

process-per-user ppu-soak-r8c-shared 2026-10-16T08:06:03.157433+00:00

Perf VM capture has waited 45 minutes for `/var/lock/perf-bench.lock`; it remains queued and I am continuing to wait. Current `/var/lock/perf-bench.owner` contents: `process-per-user ppu-soak-r8c-shared 2026-10-16T08:06:03.157433+00:00`
kayg commented 2026-10-16 08:48:41 +00:00
Author
Owner
Copy link

Perf VM runner acquired the lock at 08:48:20Z, then exited before tests with error: Module not found "test/mock-server.ts" from the staged tracer source. No measurements were produced. I’m staging the missing test sources and requeueing.

Perf VM runner acquired the lock at 08:48:20Z, then exited before tests with `error: Module not found "test/mock-server.ts"` from the staged tracer source. No measurements were produced. I’m staging the missing test sources and requeueing.
kayg commented 2026-10-16 09:34:06 +00:00
Author
Owner
Copy link

The corrected Perf VM runner has now waited 45 minutes for /var/lock/perf-bench.lock; it remains queued and I am continuing to wait. Current /var/lock/perf-bench.owner contents:

mesh-autoswitch candidate-natural31-floor-r7 venue=150 sample=4 started=2026-10-16T09:32:29Z sample_started=2026-10-16T09:33:59Z

The corrected Perf VM runner has now waited 45 minutes for `/var/lock/perf-bench.lock`; it remains queued and I am continuing to wait. Current `/var/lock/perf-bench.owner` contents: `mesh-autoswitch candidate-natural31-floor-r7 venue=150 sample=4 started=2026-10-16T09:32:29Z sample_started=2026-10-16T09:33:59Z`
kayg commented 2026-10-16 09:44:20 +00:00
Author
Owner
Copy link

The corrected Perf VM retry acquired the lock at 09:44:08Z, then exited before tests because the runner referenced src/apps/tracer/playwright.pill-flicker.config.ts, which is absent from the staged app tree. No tests or measurements ran. I’m correcting the runner to use the staged config and requeueing.

The corrected Perf VM retry acquired the lock at 09:44:08Z, then exited before tests because the runner referenced `src/apps/tracer/playwright.pill-flicker.config.ts`, which is absent from the staged app tree. No tests or measurements ran. I’m correcting the runner to use the staged config and requeueing.
kayg commented 2026-10-16 09:47:37 +00:00
Author
Owner
Copy link

Resume: branch pill-flicker, base f210a4f25055a1c6ee4e624e36f99f3f20ead640 (origin/dev). Read all #227 comments; issue metadata has no parent link. Continuing the interrupted real-server investigation by capturing each server Close frame and matching server log line against the agy byte stream and replay/control workloads.

Resume: branch `pill-flicker`, base `f210a4f25055a1c6ee4e624e36f99f3f20ead640` (origin/dev). Read all #227 comments; issue metadata has no parent link. Continuing the interrupted real-server investigation by capturing each server Close frame and matching server log line against the agy byte stream and replay/control workloads.
kayg commented 2026-10-16 09:56:57 +00:00
Author
Owner
Copy link

Client finding: apps/tracer/src/connection.ts renews fastRecoveryUntil after each quick close while a stream is attached, so repeated successful-but-immediately-closed resumes never reach exponential backoff. Added a failing regression in apps/tracer/src/connection-restart.test.ts: the second quick server close still schedules 5 ms (expected >=50 ms). Test output: Expected: >= 50; Received: 5. No product code changed.

Client finding: `apps/tracer/src/connection.ts` renews `fastRecoveryUntil` after each quick close while a stream is attached, so repeated successful-but-immediately-closed resumes never reach exponential backoff. Added a failing regression in `apps/tracer/src/connection-restart.test.ts`: the second quick server close still schedules 5 ms (expected >=50 ms). Test output: `Expected: >= 50; Received: 5`. No product code changed.
kayg commented 2026-10-16 10:07:42 +00:00
Author
Owner
Copy link

Status finding: apps/tracer/src/connection.ts reports the server close reason before forceClose(), but disconnect handling immediately replaces it with the generic reconnect status; apps/tracer/src/canvas/status.ts then dismisses the reason toast on that routine status. Added a failing regression in apps/tracer/src/canvas/status.test.ts: a close reason is dismissed once while the terminal is reconnecting, before it attaches. No product code changed.

Status finding: `apps/tracer/src/connection.ts` reports the server close reason before `forceClose()`, but disconnect handling immediately replaces it with the generic reconnect status; `apps/tracer/src/canvas/status.ts` then dismisses the reason toast on that routine status. Added a failing regression in `apps/tracer/src/canvas/status.test.ts`: a close reason is dismissed once while the terminal is reconnecting, before it attaches. No product code changed.
kayg commented 2026-10-16 10:20:05 +00:00
Author
Owner
Copy link

Client retry/status step is in the worktree at f210a4f250. A previously attached stream now gets one prompt retry; repeated short-lived retries back off. Server close reasons remain visible through reconnect status changes, and resumed attachments only mark the pill live after a 300 ms settle. The retry regression failed before the change (5 ms repeat) and now passes. bun test src/connection-restart.test.ts src/canvas/status.test.ts: 6 pass, 0 fail. bun run check: pass.

Client retry/status step is in the worktree at f210a4f25055a1c6ee4e624e36f99f3f20ead640. A previously attached stream now gets one prompt retry; repeated short-lived retries back off. Server close reasons remain visible through reconnect status changes, and resumed attachments only mark the pill live after a 300 ms settle. The retry regression failed before the change (5 ms repeat) and now passes. `bun test src/connection-restart.test.ts src/canvas/status.test.ts`: 6 pass, 0 fail. `bun run check`: pass.
kayg commented 2026-10-16 10:26:27 +00:00
Author
Owner
Copy link

Design note for the orchestrator: TASTE/DECISIONS had no settle interval for a resumed connection. I used 300 ms of uninterrupted attachment before the dot turns live; the 110–120 ms close loop then stays steadily grey, with no animation. The exact interval can be made a named design choice if this behavior is retained.

Design note for the orchestrator: TASTE/DECISIONS had no settle interval for a resumed connection. I used 300 ms of uninterrupted attachment before the dot turns live; the 110–120 ms close loop then stays steadily grey, with no animation. The exact interval can be made a named design choice if this behavior is retained.
kayg commented 2026-10-16 10:35:31 +00:00
Author
Owner
Copy link

Build-host repro status: the single wrapped real-server Playwright baseline has waited 45 minutes in the shared build-slot queue. No test child has started and real-synthetic-baseline.log remains 0 bytes. The next dependency is a free host build slot; I have not bypassed the wrapper or moved the repro to the perf VM.

Build-host repro status: the single wrapped real-server Playwright baseline has waited 45 minutes in the shared build-slot queue. No test child has started and `real-synthetic-baseline.log` remains 0 bytes. The next dependency is a free host build slot; I have not bypassed the wrapper or moved the repro to the perf VM.
kayg commented 2026-10-16 10:44:00 +00:00
Author
Owner
Copy link

The earlier real-server traces recorded an upstream WebSocket close with code 1005 and no application Close frame. MuxConnection discarded the browser CloseEvent, leaving only the generic reconnecting status. The new clean-close regression failed before the change (Received: "Connection lost. Reconnecting…") and passes now: the fallback reason is retained through the retry; internal connection rotation stays quiet. bun test src/connection-restart.test.ts: 4 pass, 0 fail. bun test src/canvas/status.test.ts: 3 pass, 0 fail. bun run check: pass. This stabilizes the client reason; the server-side close cause is still under investigation.

The earlier real-server traces recorded an upstream WebSocket close with code 1005 and no application Close frame. `MuxConnection` discarded the browser CloseEvent, leaving only the generic reconnecting status. The new clean-close regression failed before the change (`Received: "Connection lost. Reconnecting…"`) and passes now: the fallback reason is retained through the retry; internal connection rotation stays quiet. `bun test src/connection-restart.test.ts`: 4 pass, 0 fail. `bun test src/canvas/status.test.ts`: 3 pass, 0 fail. `bun run check`: pass. This stabilizes the client reason; the server-side close cause is still under investigation.
kayg commented 2026-10-16 10:50:35 +00:00
Author
Owner
Copy link

Build-host repro update: the wrapped real-server Playwright baseline has remained queued for 60 minutes; it has not started and real-synthetic-baseline.log is still 0 bytes. I am continuing to wait for the shared host build slot. The current client unit/type checks pass, but the server-close cause remains unverified until the real-server run begins.

Build-host repro update: the wrapped real-server Playwright baseline has remained queued for 60 minutes; it has not started and `real-synthetic-baseline.log` is still 0 bytes. I am continuing to wait for the shared host build slot. The current client unit/type checks pass, but the server-close cause remains unverified until the real-server run begins.
kayg commented 2026-10-16 11:02:19 +00:00
Author
Owner
Copy link

The real-server baseline has waited 71 minutes for the host build slot. Its test process has not started and its log remains 0 bytes. Current owners:
slot 1: multiplexer-skip — scripts/verify fast
slot 2: app-windows-ship — browser-and-size-gate.sh
slot 3: link-tooltip — scripts/verify fast
slot 4: mesh-unreachable-review — scripts/verify fast
priority slot: addhost-retry-review — build-review.sh
I am keeping the queued run alive and continuing source review while it waits.

The real-server baseline has waited 71 minutes for the host build slot. Its test process has not started and its log remains 0 bytes. Current owners: slot 1: multiplexer-skip — scripts/verify fast slot 2: app-windows-ship — browser-and-size-gate.sh slot 3: link-tooltip — scripts/verify fast slot 4: mesh-unreachable-review — scripts/verify fast priority slot: addhost-retry-review — build-review.sh I am keeping the queued run alive and continuing source review while it waits.
kayg commented 2026-10-16 11:20:20 +00:00
Author
Owner
Copy link

The real-server baseline has now waited 90 minutes. The test process has not started; its log is still 0 bytes. Current build-slot owners:
slot 1: mesh-unreachable-review — cargo clippy --locked -p infraternal-server --tests -- -D warnings
slot 2: app-windows-ship — browser-and-size-gate.sh
slot 3: mesh-unreachable-review — run-scale.sh
slot 4: mesh-unreachable-review — scripts/verify fast
priority slot: review-addhost-retry — build-review.sh
The queued Playwright command remains alive and will run through the wrapper when a slot opens.

The real-server baseline has now waited 90 minutes. The test process has not started; its log is still 0 bytes. Current build-slot owners: slot 1: mesh-unreachable-review — cargo clippy --locked -p infraternal-server --tests -- -D warnings slot 2: app-windows-ship — browser-and-size-gate.sh slot 3: mesh-unreachable-review — run-scale.sh slot 4: mesh-unreachable-review — scripts/verify fast priority slot: review-addhost-retry — build-review.sh The queued Playwright command remains alive and will run through the wrapper when a slot opens.
kayg commented 2026-10-16 11:20:28 +00:00
Author
Owner
Copy link

The real-server baseline has now waited 90 minutes. The test process has not started; its log is still 0 bytes. Current build-slot owner records:
2026-10-16T13:10:05+02:00 slot 1: mesh-unreachable-review pid 300900: cargo clippy --locked -p infraternal-server --tests -- -D warnings
2026-10-16T12:50:13+02:00 slot 2: ? pid 4017255: bash /home/kayg/.cache/codex-scratch/app-windows-ship/evidence/browser-and-size-gate.sh
2026-10-16T13:09:50+02:00 slot 3: mesh-unreachable-review pid 321087: netbird ssh --no-browser root@10.69.69.208 bash /root/infraternal-bench/mesh-unreachable-review/run-scale.sh
2026-10-16T11:55:47+02:00 slot 4: mesh-unreachable-review pid 3737231: scripts/verify fast
2026-10-16T13:00:52+02:00 slot prio: review-addhost-retry pid 258601: bash /home/kayg/.cache/codex-scratch/addhost-retry-review/build-review.sh
The queued Playwright command remains alive and will run through the wrapper when a slot opens.

The real-server baseline has now waited 90 minutes. The test process has not started; its log is still 0 bytes. Current build-slot owner records: 2026-10-16T13:10:05+02:00 slot 1: mesh-unreachable-review pid 300900: cargo clippy --locked -p infraternal-server --tests -- -D warnings 2026-10-16T12:50:13+02:00 slot 2: ? pid 4017255: bash /home/kayg/.cache/codex-scratch/app-windows-ship/evidence/browser-and-size-gate.sh 2026-10-16T13:09:50+02:00 slot 3: mesh-unreachable-review pid 321087: netbird ssh --no-browser root@10.69.69.208 bash /root/infraternal-bench/mesh-unreachable-review/run-scale.sh 2026-10-16T11:55:47+02:00 slot 4: mesh-unreachable-review pid 3737231: scripts/verify fast 2026-10-16T13:00:52+02:00 slot prio: review-addhost-retry pid 258601: bash /home/kayg/.cache/codex-scratch/addhost-retry-review/build-review.sh The queued Playwright command remains alive and will run through the wrapper when a slot opens.
kayg commented 2026-10-16 12:06:47 +00:00
Author
Owner
Copy link

Finding: the real agy trace records a server WebSocket close at 483 ms with code 1005 and an empty reason, and no app-level mux Close frame (agy-timed-output.log). crates/infraternal-server/src/websocket.rs sends Message::Close(None) for every completed socket loop, including normal exit and loop_result errors, so this close event does not identify the server branch. I will add focused diagnostic branch logging for the real-server replay before attributing the closure to terminal output.

Finding: the real agy trace records a server WebSocket close at 483 ms with code 1005 and an empty reason, and no app-level mux Close frame (`agy-timed-output.log`). `crates/infraternal-server/src/websocket.rs` sends `Message::Close(None)` for every completed socket loop, including normal exit and `loop_result` errors, so this close event does not identify the server branch. I will add focused diagnostic branch logging for the real-server replay before attributing the closure to terminal output.
kayg commented 2026-10-16 12:08:05 +00:00
Author
Owner
Copy link

The real-server baseline has waited 2h15m through the host build-slot wrapper and has not started; real-synthetic-baseline.log remains 0 bytes. Current wrapper owner records:
2026-10-16T13:51:34+02:00 slot 1: multiplexer-skip pid 333163: scripts/verify fast
2026-10-16T12:50:13+02:00 slot 2: ? pid 4017255: bash /home/kayg/.cache/codex-scratch/app-windows-ship/evidence/browser-and-size-gate.sh
2026-10-16T13:09:50+02:00 slot 3: mesh-unreachable-review pid 321087: netbird ssh --no-browser root@10.69.69.208 bash /root/infraternal-bench/mesh-unreachable-review/run-scale.sh
2026-10-16T13:44:59+02:00 slot 4: ? pid 4013097: bash /home/kayg/.cache/codex-scratch/process-per-user/r8/verify-aged.sh
2026-10-16T13:29:41+02:00 slot prio: mesh-unreachable-review pid 256386: scripts/verify iterate
2026-10-16T14:01:22+02:00 slot prio2: mesh-unreachable-review pid 742606: netbird ssh --no-browser root@10.69.69.208 bash /root/infraternal-bench/mesh-unreachable-review/run-browser.sh
The original queued test remains alive under the build-slot wrapper and will continue waiting.

The real-server baseline has waited 2h15m through the host build-slot wrapper and has not started; `real-synthetic-baseline.log` remains 0 bytes. Current wrapper owner records: 2026-10-16T13:51:34+02:00 slot 1: multiplexer-skip pid 333163: scripts/verify fast 2026-10-16T12:50:13+02:00 slot 2: ? pid 4017255: bash /home/kayg/.cache/codex-scratch/app-windows-ship/evidence/browser-and-size-gate.sh 2026-10-16T13:09:50+02:00 slot 3: mesh-unreachable-review pid 321087: netbird ssh --no-browser root@10.69.69.208 bash /root/infraternal-bench/mesh-unreachable-review/run-scale.sh 2026-10-16T13:44:59+02:00 slot 4: ? pid 4013097: bash /home/kayg/.cache/codex-scratch/process-per-user/r8/verify-aged.sh 2026-10-16T13:29:41+02:00 slot prio: mesh-unreachable-review pid 256386: scripts/verify iterate 2026-10-16T14:01:22+02:00 slot prio2: mesh-unreachable-review pid 742606: netbird ssh --no-browser root@10.69.69.208 bash /root/infraternal-bench/mesh-unreachable-review/run-browser.sh The original queued test remains alive under the build-slot wrapper and will continue waiting.
kayg commented 2026-10-16 12:09:55 +00:00
Author
Owner
Copy link

Harness setup finding: the queued real-server baseline environment had TRACER_TEST_PORT=4787 and scratch state roots, but no INFRATERAL_BIN; the checkout default target/x86_64-unknown-linux-musl/release/infraternal is absent. The existing scratch binary is /home/kayg/.cache/codex-scratch/pill-flicker/target/x86_64-unknown-linux-musl/release/infraternal (21,422,096 bytes, mtime 2026-10-16 09:32 +0200). I am restarting only my queued wrapper with that binary path set explicitly.

Harness setup finding: the queued real-server baseline environment had `TRACER_TEST_PORT=4787` and scratch state roots, but no `INFRATERAL_BIN`; the checkout default `target/x86_64-unknown-linux-musl/release/infraternal` is absent. The existing scratch binary is `/home/kayg/.cache/codex-scratch/pill-flicker/target/x86_64-unknown-linux-musl/release/infraternal` (21,422,096 bytes, mtime 2026-10-16 09:32 +0200). I am restarting only my queued wrapper with that binary path set explicitly.
kayg commented 2026-10-16 12:18:07 +00:00
Author
Owner
Copy link

Harness setup correction: the corrected queue acquired a slot but Playwright exited before launching tests because its playwright.real.config.ts path is relative to apps/tracer; the command had been started from the repository root. I am rerunning the same production-base server/web baseline with working directory apps/tracer.

Harness setup correction: the corrected queue acquired a slot but Playwright exited before launching tests because its `playwright.real.config.ts` path is relative to `apps/tracer`; the command had been started from the repository root. I am rerunning the same production-base server/web baseline with working directory `apps/tracer`.
kayg commented 2026-10-16 12:22:13 +00:00
Author
Owner
Copy link

Harness setup correction: the test now ran but failed before server setup because I passed INFRATERAL_BIN; the fixture reads INFRATERNAL_BIN (with the n after r) in apps/tracer/real/server.ts:112. The baseline server binary path was therefore not overridden. I am rerunning with the fixture's exact variable name and a fresh test port.

Harness setup correction: the test now ran but failed before server setup because I passed `INFRATERAL_BIN`; the fixture reads `INFRATERNAL_BIN` (with the `n` after `r`) in `apps/tracer/real/server.ts:112`. The baseline server binary path was therefore not overridden. I am rerunning with the fixture's exact variable name and a fresh test port.
kayg commented 2026-10-16 12:22:46 +00:00
Author
Owner
Copy link

Harness setup correction: with the binary override fixed, the fixture stopped before launching because the socket path under INFRATERNA_TEST_STATE_ROOT=/home/kayg/.cache/codex-scratch/pill-flicker/runtime-real exceeded the 108-byte Unix socket limit (apps/tracer/real/server.ts:120-122). I will use the shorter job-local scratch root /home/kayg/.cache/codex-scratch/pill-flicker/r and a fresh test port.

Harness setup correction: with the binary override fixed, the fixture stopped before launching because the socket path under `INFRATERNA_TEST_STATE_ROOT=/home/kayg/.cache/codex-scratch/pill-flicker/runtime-real` exceeded the 108-byte Unix socket limit (`apps/tracer/real/server.ts:120-122`). I will use the shorter job-local scratch root `/home/kayg/.cache/codex-scratch/pill-flicker/r` and a fresh test port.
kayg commented 2026-10-16 12:23:18 +00:00
Author
Owner
Copy link

Runtime-path correction: the proposed /home/kayg/.cache/codex-scratch/pill-flicker/r root still yields a 113-byte session socket path, above the 108-byte limit. I will keep build artifacts, captured logs and evidence under the job scratch directory, while using the fixture's short /tmp runtime root only for its ephemeral Unix socket directory (as its tmpdir() default does).

Runtime-path correction: the proposed `/home/kayg/.cache/codex-scratch/pill-flicker/r` root still yields a 113-byte session socket path, above the 108-byte limit. I will keep build artifacts, captured logs and evidence under the job scratch directory, while using the fixture's short `/tmp` runtime root only for its ephemeral Unix socket directory (as its `tmpdir()` default does).
kayg commented 2026-10-16 12:26:45 +00:00
Author
Owner
Copy link

Real-server production-base baseline completed (binary 21,422,096 bytes; web 327,181 bytes / 83,577 Brotli). Synthetic 9 Hz redraw sent 40,795 output bytes with 19 title changes, 39 synchronized-output toggles and 475 cursor-addressed writes. The pill changed live→connecting at +478 ms and back live at +619 ms: one reconnect, not ≥10 flips/2 s. The upstream socket ended first with WS 1006/empty reason and no mux Close; resume attached at output cursor 183 and replayed once. The resumed socket carried 40,542 bytes without another close. The server emitted no close-cause line, so source-level diagnostics are still needed before attributing the single disconnect. summary, raw receipt.

Real-server production-base baseline completed (binary 21,422,096 bytes; web 327,181 bytes / 83,577 Brotli). Synthetic 9 Hz redraw sent 40,795 output bytes with 19 title changes, 39 synchronized-output toggles and 475 cursor-addressed writes. The pill changed live→connecting at +478 ms and back live at +619 ms: one reconnect, not ≥10 flips/2 s. The upstream socket ended first with WS 1006/empty reason and no mux Close; resume attached at output cursor 183 and replayed once. The resumed socket carried 40,542 bytes without another close. The server emitted no close-cause line, so source-level diagnostics are still needed before attributing the single disconnect. [summary](https://git.kayg.org/attachments/7d95ef81-84c5-4402-adb9-54a3b36e0d94), [raw receipt](https://git.kayg.org/attachments/a6b0e295-f0ef-48ae-8704-36b6398b6885).
kayg commented 2026-10-16 12:31:11 +00:00
Author
Owner
Copy link

The first staged agy replay did not reach the control corpus: Playwright’s Ctrl+C went to the terminal foreground process group and stopped the runner along with agy, so the completion marker timed out after 20 s. I changed the scratch workload to time-limit only the agy child and let the script continue; no cause is inferred from that failed harness attempt. bun run check passes after the test harness adjustment, and no agy process remains running.

The first staged agy replay did not reach the control corpus: Playwright’s Ctrl+C went to the terminal foreground process group and stopped the runner along with agy, so the completion marker timed out after 20 s. I changed the scratch workload to time-limit only the agy child and let the script continue; no cause is inferred from that failed harness attempt. `bun run check` passes after the test harness adjustment, and no agy process remains running.
kayg commented 2026-10-16 12:37:30 +00:00
Author
Owner
Copy link

Real-server staged replay completed. The only close was before agy started: upstream WS Close 1005/empty reason at +431 ms; the browser close event was 4000/empty, with no mux Close. Browser Ping/Pong continued through +426 ms. One resume attached at output cursor 182 and replayed once. agy began at +619 ms; its sign-in screen and the full OSC 8/title/synchronized-output/mode/query/65 KiB line corpus completed at +10.4 s, sending 134,324 output bytes with no further close or replay. The pill changed live→connecting→live once; this does not reproduce 8.5 Hz. The staged input did not trigger this close. I am instrumenting the browser close call to identify the source of the pre-workload 4000. summary, receipt.

Real-server staged replay completed. The only close was before agy started: upstream WS Close 1005/empty reason at +431 ms; the browser close event was 4000/empty, with no mux Close. Browser Ping/Pong continued through +426 ms. One resume attached at output cursor 182 and replayed once. agy began at +619 ms; its sign-in screen and the full OSC 8/title/synchronized-output/mode/query/65 KiB line corpus completed at +10.4 s, sending 134,324 output bytes with no further close or replay. The pill changed live→connecting→live once; this does not reproduce 8.5 Hz. The staged input did not trigger this close. I am instrumenting the browser close call to identify the source of the pre-workload `4000`. [summary](https://git.kayg.org/attachments/e8022355-84db-4413-90c0-9c259be9cf87), [receipt](https://git.kayg.org/attachments/a9d68bff-05e8-4dba-80ee-4eaab64ff4e6).
kayg commented 2026-10-16 12:48:20 +00:00
Author
Owner
Copy link

Close-origin evidence: the browser called WebSocket.close(4000, "connection unresponsive") at +356 ms; the upstream protocol bridge reported WS 1005/empty reason at +318 ms, and the pill changed live→connecting at +370 ms then back live at +441 ms. Ping/Pong frames were logged through +308 ms; maximum RAF gap during the first 700 ms was 19.3 ms. The hook did not capture the target socket path, so this does not yet prove which socket the local close call targeted. I’m adding the safe path/readyState fields before assigning cause. summary, receipt.

Close-origin evidence: the browser called `WebSocket.close(4000, "connection unresponsive")` at +356 ms; the upstream protocol bridge reported WS 1005/empty reason at +318 ms, and the pill changed live→connecting at +370 ms then back live at +441 ms. Ping/Pong frames were logged through +308 ms; maximum RAF gap during the first 700 ms was 19.3 ms. The hook did not capture the target socket path, so this does not yet prove which socket the local close call targeted. I’m adding the safe path/readyState fields before assigning cause. [summary](https://git.kayg.org/attachments/ea97454a-51de-4ab1-bd91-4ffe884fa9e4), [receipt](https://git.kayg.org/attachments/d741366a-0515-4628-b8d1-89b41cb62032).
kayg commented 2026-10-16 13:01:01 +00:00
Author
Owner
Copy link

Finding: the staged agy sign-in screen plus the requested control corpus did not reproduce an input-triggered close. The real-server capture recorded 134,324 terminal-output bytes and completed the OSC 8, repeated OSC 0/2 + synchronized-output frames, mode toggles, device queries and 65 KiB line. One upstream EOF occurred at +431 ms before the agy-start marker; Resume used output cursor 182 and replayed once, then the socket stayed open through the remaining 10.4 s. Evidence: agy-control-replay-summary.json and agy-control-replay.log are attached above. An earlier run captured a client close call 4000 / connection unresponsive at +356 ms, but the subsequent path-instrumented run had no page-side WebSocket close call. I am capturing the server loop/reader exit reason now because the close is intermittent and precedes the workload.

Finding: the staged agy sign-in screen plus the requested control corpus did not reproduce an input-triggered close. The real-server capture recorded 134,324 terminal-output bytes and completed the OSC 8, repeated OSC 0/2 + synchronized-output frames, mode toggles, device queries and 65 KiB line. One upstream EOF occurred at +431 ms before the `agy-start` marker; Resume used output cursor 182 and replayed once, then the socket stayed open through the remaining 10.4 s. Evidence: `agy-control-replay-summary.json` and `agy-control-replay.log` are attached above. An earlier run captured a client close call `4000 / connection unresponsive` at +356 ms, but the subsequent path-instrumented run had no page-side WebSocket close call. I am capturing the server loop/reader exit reason now because the close is intermittent and precedes the workload.
kayg commented 2026-10-16 13:06:46 +00:00
Author
Owner
Copy link

Correction: the preceding real-server traces used a binary built from the worktree's f210 source, not the deployed server commit 492e1505a. Their web assets were f210a4f25. I canceled the queued run before it started and am rebuilding the diagnostic server from 492e1505a while keeping the f210 web assets, then replaying the same staged agy screen and control corpus. Do not treat the earlier server-side results as reproduction against the deployed server.

Correction: the preceding real-server traces used a binary built from the worktree's f210 source, not the deployed server commit 492e1505a. Their web assets were f210a4f25. I canceled the queued run before it started and am rebuilding the diagnostic server from 492e1505a while keeping the f210 web assets, then replaying the same staged agy screen and control corpus. Do not treat the earlier server-side results as reproduction against the deployed server.
kayg commented 2026-10-16 14:19:05 +00:00
Author
Owner
Copy link

Current production tracer assets match the f210a4f25 base at 327,181 raw bytes; Brotli is 83,525 bytes versus 83,577 on base (−52 bytes). Receipt: web-current-size.json.

Current production tracer assets match the f210a4f25 base at 327,181 raw bytes; Brotli is 83,525 bytes versus 83,577 on base (−52 bytes). Receipt: [web-current-size.json](https://git.kayg.org/attachments/e6327643-e589-470d-b86f-3a7d0b097d92).
kayg commented 2026-10-16 14:21:00 +00:00
Author
Owner
Copy link

The tracer's local liveness path retires a socket with close code 4000 and reason connection unresponsive after a missed Pong. It previously called disconnect() before WebSocket.close(), clearing the current socket so onclose ignored that reason and status fell back to generic reconnect text. I now record and map the reason before retirement; the new assertion fails on base and passes with the fix (10 tests, 0 failures). This explains the lost reason; the instrumented 492e replay is still needed to confirm what delays or misses the Pong under agy's exact output. Receipts: old-path failure, fixed tests.

The tracer's local liveness path retires a socket with close code 4000 and reason `connection unresponsive` after a missed Pong. It previously called `disconnect()` before `WebSocket.close()`, clearing the current socket so `onclose` ignored that reason and status fell back to generic reconnect text. I now record and map the reason before retirement; the new assertion fails on base and passes with the fix (10 tests, 0 failures). This explains the lost reason; the instrumented 492e replay is still needed to confirm what delays or misses the Pong under agy's exact output. Receipts: [old-path failure](https://git.kayg.org/attachments/db745d0f-f84e-4994-8bd4-9850865fad0b), [fixed tests](https://git.kayg.org/attachments/998c2f61-7927-4cf7-819f-b63a7d364e6b).
kayg commented 2026-10-16 14:25:00 +00:00
Author
Owner
Copy link

Exact 492e instrumented real-server trace: the server socket ended at +522 ms with WebSocket 1005/empty reason; its log said proof authority or transmit failed and terminal output handling failed. The browser recorded its own close(4000, "connection unresponsive") at +549 ms. This preceded the staged agy-start marker at +823 ms. The resumed socket attached at +792 ms, replayed once, and remained open. The staged agy step ended at +5,827 ms but emitted only 26 terminal-output bytes, so it did not exercise the sign-in redraw. Later control replay delivered 133,942 bytes with no further close. The current server diagnostics still collapse the proof/send error; I am instrumenting that path and replacing the ineffective agy stage before concluding on the trigger. Receipts: safe trace summary, real-server log.

Exact 492e instrumented real-server trace: the server socket ended at +522 ms with WebSocket 1005/empty reason; its log said `proof authority or transmit failed` and `terminal output handling failed`. The browser recorded its own `close(4000, "connection unresponsive")` at +549 ms. This preceded the staged `agy-start` marker at +823 ms. The resumed socket attached at +792 ms, replayed once, and remained open. The staged agy step ended at +5,827 ms but emitted only 26 terminal-output bytes, so it did not exercise the sign-in redraw. Later control replay delivered 133,942 bytes with no further close. The current server diagnostics still collapse the proof/send error; I am instrumenting that path and replacing the ineffective agy stage before concluding on the trigger. Receipts: [safe trace summary](https://git.kayg.org/attachments/3ffb51d3-e3af-4559-a877-6647646ff2f9), [real-server log](https://git.kayg.org/attachments/bf404265-752f-49c8-87c6-f15ab1c37211).
kayg commented 2026-10-16 14:33:58 +00:00
Author
Owner
Copy link

Proof-detail replay of binary 492e1505: over 11.1 s, the pill had 0 status mutations and all 641 sampled frames were live/rgb(62, 158, 41). The server accepted 494 Pong replies; no lease expiry or WebSocket write failure occurred before harness teardown. One resume replay attached once and remained open. The attached summary records the exact counts.

The staged agy command ran 5.0 s but emitted only the 26-byte start marker, no TUI output. The 134,261-byte OSC/CSI control corpus also caused no close. The earlier close was before the agy stage, so its trigger is still unproven. I will replace that ineffective stage with a measured 9 Hz cursor-addressed redraw and keep the production server close cause open until reproduced.

Proof-detail replay of binary 492e1505: over 11.1 s, the pill had 0 status mutations and all 641 sampled frames were live/rgb(62, 158, 41). The server accepted 494 Pong replies; no lease expiry or WebSocket write failure occurred before harness teardown. One resume replay attached once and remained open. The attached summary records the exact counts. The staged agy command ran 5.0 s but emitted only the 26-byte start marker, no TUI output. The 134,261-byte OSC/CSI control corpus also caused no close. The earlier close was before the agy stage, so its trigger is still unproven. I will replace that ineffective stage with a measured 9 Hz cursor-addressed redraw and keep the production server close cause open until reproduced.
kayg commented 2026-10-16 14:39:14 +00:00
Author
Owner
Copy link

The exact 492e server close is now isolated. The first close was caused by my real-server harness typing a long absolute launcher path as 92 separate one-byte terminal Input frames over ~261 ms: server probe Pongs stopped at 14:35:31.219Z, the 250 ms lease expired at 14:35:31.493Z, and the server closed with 1005/empty reason. This was before the redraw began, so it was a harness artifact, not evidence against the screen output.

After shortening the command, the exact production server handled the 10.628 s 9 Hz cursor-addressed redraw (191,025 bytes, 90 title updates, 2,250 addressed rows) with 0 pill status mutations, 632/632 live green frame samples, and no socket closes during the workload. The attached summary has both runs. The 134 KB control corpus also remained open. The staged agy invocation still emits no UI bytes; its output is only the fixed marker. I have not found evidence for a server close triggered by the sign-in output yet, so I’m continuing with an accurate agy invocation/capture and the required mock reconnect regression rather than making an unsupported server change.

The exact 492e server close is now isolated. The first close was caused by my real-server harness typing a long absolute launcher path as 92 separate one-byte terminal Input frames over ~261 ms: server probe Pongs stopped at 14:35:31.219Z, the 250 ms lease expired at 14:35:31.493Z, and the server closed with 1005/empty reason. This was before the redraw began, so it was a harness artifact, not evidence against the screen output. After shortening the command, the exact production server handled the 10.628 s 9 Hz cursor-addressed redraw (191,025 bytes, 90 title updates, 2,250 addressed rows) with 0 pill status mutations, 632/632 live green frame samples, and no socket closes during the workload. The attached summary has both runs. The 134 KB control corpus also remained open. The staged agy invocation still emits no UI bytes; its output is only the fixed marker. I have not found evidence for a server close triggered by the sign-in output yet, so I’m continuing with an accurate agy invocation/capture and the required mock reconnect regression rather than making an unsupported server change.
kayg commented 2026-10-16 14:49:34 +00:00
Author
Owner
Copy link

The mock browser regression now reproduces the requested rapid state churn. With INFRATERNAL_TEST_RTT=0 and a 110 ms close after each attach, the exact f210a4f source produced 32 dot flips in the first 2 s (40 over the 2.5 s trace: 20 live and 20 connecting transitions) and failed the regression. The branch produced 1 connecting transition, 0 live promotions during retries and 0 live frames after the first disconnect; both browser specs passed. The default 300 ms proxy RTT masks this cadence, so the local repro explicitly sets RTT to zero. Attached comparison includes the counts.

The mock browser regression now reproduces the requested rapid state churn. With INFRATERNAL_TEST_RTT=0 and a 110 ms close after each attach, the exact f210a4f source produced 32 dot flips in the first 2 s (40 over the 2.5 s trace: 20 live and 20 connecting transitions) and failed the regression. The branch produced 1 connecting transition, 0 live promotions during retries and 0 live frames after the first disconnect; both browser specs passed. The default 300 ms proxy RTT masks this cadence, so the local repro explicitly sets RTT to zero. Attached comparison includes the counts.
kayg commented 2026-10-16 14:59:45 +00:00
Author
Owner
Copy link

The clean production server build (static x86_64 musl, 21,421,840 bytes) passed the real-server redraw spec. Across 10.408 s and 191,025 output bytes (90 titles, 90 clears, 2,250 cursor-addressed rows), the dot had 0 status mutations, all 580 sampled frames stayed live/rgb(62, 158, 41), and no socket closed during the workload. The 10.000 s H.264 demo and a representative frame are attached.

The recorded workload is the synthetic sign-in screen; the staged agy executable still produced no screen output under its isolated home. I have not presented the synthetic demo as an agy capture.

The clean production server build (static x86_64 musl, 21,421,840 bytes) passed the real-server redraw spec. Across 10.408 s and 191,025 output bytes (90 titles, 90 clears, 2,250 cursor-addressed rows), the dot had 0 status mutations, all 580 sampled frames stayed live/rgb(62, 158, 41), and no socket closed during the workload. The 10.000 s H.264 demo and a representative frame are attached. The recorded workload is the synthetic sign-in screen; the staged agy executable still produced no screen output under its isolated home. I have not presented the synthetic demo as an agy capture.
kayg commented 2026-10-16 15:04:42 +00:00
Author
Owner
Copy link

The pre-existing branch dist was a readable diagnostic build, not the production bundle: transfer-budget measured critical JavaScript at 1,071,509 identity / 339,959 gzip / 285,088 Brotli bytes (Brotli budget 256,000), so I excluded it. A fresh minified production build will be used for the before/after measurement. The clean f210 base production build measures 961,716 identity / 297,686 gzip / 247,538 Brotli bytes.

The pre-existing branch dist was a readable diagnostic build, not the production bundle: transfer-budget measured critical JavaScript at 1,071,509 identity / 339,959 gzip / 285,088 Brotli bytes (Brotli budget 256,000), so I excluded it. A fresh minified production build will be used for the before/after measurement. The clean f210 base production build measures 961,716 identity / 297,686 gzip / 247,538 Brotli bytes.
kayg commented 2026-10-16 15:17:37 +00:00
Author
Owner
Copy link

Perf VM measurement harness finding: a 10.4 s production-server redraw produced 191 browser WebSocket Output frames. The initial 1,000-frame per-run guard rejected that sample. I changed the focused harness to require at least 100 frames per run and will aggregate six alternating trials per build, targeting at least 1,000 Output frames for each before/after total.

Perf VM measurement harness finding: a 10.4 s production-server redraw produced 191 browser WebSocket Output frames. The initial 1,000-frame per-run guard rejected that sample. I changed the focused harness to require at least 100 frames per run and will aggregate six alternating trials per build, targeting at least 1,000 Output frames for each before/after total.
kayg commented 2026-10-16 15:25:15 +00:00
Author
Owner
Copy link

Perf comparison complete on infraternal-perf-test (4 vCPU / 7 GiB), each Playwright run under CPUWeight=10000. Load average: start 1.27/0.70/0.42; end 0.36/0.62/0.46. Both builds used the same production-equivalent server binary (21,422,096 bytes; SHA-256 beb50f0804f14f6ac85c712b613362f2047a95d4f874418b88b037cc8fb0d603).

Six alternating trials per frontend: base 1,003 Output frames / 6,915.073 ms main-thread TaskDuration = 6,894.390 ms per 1,000 frames; branch 1,025 / 7,032.175 ms = 6,860.659 ms per 1,000 frames (-0.489%). 12/12 real-server Playwright runs passed. All six trials per build had zero data-status mutations and zero client/server socket closes during redraw; sampled dot stayed rgb(62, 158, 41).

Summary, per-trial CSV.

Perf comparison complete on infraternal-perf-test (4 vCPU / 7 GiB), each Playwright run under CPUWeight=10000. Load average: start 1.27/0.70/0.42; end 0.36/0.62/0.46. Both builds used the same production-equivalent server binary (21,422,096 bytes; SHA-256 beb50f0804f14f6ac85c712b613362f2047a95d4f874418b88b037cc8fb0d603). Six alternating trials per frontend: base 1,003 Output frames / 6,915.073 ms main-thread TaskDuration = 6,894.390 ms per 1,000 frames; branch 1,025 / 7,032.175 ms = 6,860.659 ms per 1,000 frames (-0.489%). 12/12 real-server Playwright runs passed. All six trials per build had zero data-status mutations and zero client/server socket closes during redraw; sampled dot stayed rgb(62, 158, 41). [Summary](https://git.kayg.org/attachments/8d39085d-f437-4966-bae1-f726fcf4ce48), [per-trial CSV](https://git.kayg.org/attachments/f7025ff9-7c9b-4008-92dd-f2ffc8e4b44d).
kayg commented 2026-10-16 15:27:15 +00:00
Author
Owner
Copy link

Size comparison complete. Both minified production tracer builds passed bun transfer-budget.ts; the compressed critical JavaScript path is 247,589 B Brotli on f210 and 247,544 B on branch (-45 B). Identity is +174 B and gzip +56 B; production Brotli is smaller. Stripped static x86_64-musl server binary is 21,422,096 B on base and 21,421,840 B on branch (-256 B), below the 16 KiB limit.

Summary, base transfer receipt, branch transfer receipt.

Size comparison complete. Both minified production tracer builds passed `bun transfer-budget.ts`; the compressed critical JavaScript path is 247,589 B Brotli on f210 and 247,544 B on branch (-45 B). Identity is +174 B and gzip +56 B; production Brotli is smaller. Stripped static x86_64-musl server binary is 21,422,096 B on base and 21,421,840 B on branch (-256 B), below the 16 KiB limit. [Summary](https://git.kayg.org/attachments/3a57c74c-2c27-4d38-bd1a-432f21bfe685), [base transfer receipt](https://git.kayg.org/attachments/0ced7269-ce78-41a5-ac69-6bf7d7e3251c), [branch transfer receipt](https://git.kayg.org/attachments/6d9e5be4-81d6-4d58-854e-f18e73e8abac).
kayg commented 2026-10-16 15:31:27 +00:00
Author
Owner
Copy link

Real-server replay finding: both the f210a4f25 baseline and branch completed the 90-frame control/query workload (829,393 output bytes per run) without any WebSocket close or pill mutation; all 613/614 sampled frames remained live green. The workload included OSC 8 links, OSC title changes, mode toggles, device/terminal/keyboard/graphics queries, cursor redraws and repeated 8 KiB lines. No server Close frame occurred, so this workload yielded no server close reason. I still cannot claim the exact agy reconnect trigger: the staged agy sign-in output has not been captured reliably as a repeating byte stream. Sanitized evidence: real-query-summary.txt.

Real-server replay finding: both the f210a4f25 baseline and branch completed the 90-frame control/query workload (829,393 output bytes per run) without any WebSocket close or pill mutation; all 613/614 sampled frames remained live green. The workload included OSC 8 links, OSC title changes, mode toggles, device/terminal/keyboard/graphics queries, cursor redraws and repeated 8 KiB lines. No server Close frame occurred, so this workload yielded no server close reason. I still cannot claim the exact agy reconnect trigger: the staged agy sign-in output has not been captured reliably as a repeating byte stream. Sanitized evidence: [real-query-summary.txt](https://git.kayg.org/attachments/7d4f601c-2810-4ae8-891d-10055c236012).
kayg commented 2026-10-16 15:36:35 +00:00
Author
Owner
Copy link

The current real-server attempt invoked the host agy executable with a fresh isolated HOME and no inherited environment or user input. Its default invocation emitted 531 bytes including wrapper/setup output but no alternate screen, cursor redraw, clear-screen or title controls. Across 14.0 seconds the pill had 0 mutations and there were no socket closes. This did not exercise the reported full-screen sign-in TUI, so it does not identify the production close trigger. Summary: agy-plain-run-summary.txt. I’m checking the executable’s interactive invocation next.

The current real-server attempt invoked the host `agy` executable with a fresh isolated HOME and no inherited environment or user input. Its default invocation emitted 531 bytes including wrapper/setup output but no alternate screen, cursor redraw, clear-screen or title controls. Across 14.0 seconds the pill had 0 mutations and there were no socket closes. This did not exercise the reported full-screen sign-in TUI, so it does not identify the production close trigger. Summary: [agy-plain-run-summary.txt](https://git.kayg.org/attachments/867d45cc-2a1d-42b4-b52d-16a47d8c095e). I’m checking the executable’s interactive invocation next.
kayg commented 2026-10-16 15:37:46 +00:00
Author
Owner
Copy link

Follow-up agy finding: agy --prompt-interactive in another fresh isolated HOME exited with status 2 after 0.28 seconds and emitted no full-screen terminal controls. The pill remained live green during its 16 sampled frames; no socket closed. This does not reproduce the reported TUI and cannot identify the production close trigger. Sanitized summary: agy-interactive-summary.txt. I’m retaining the real-server 9 Hz redraw fixture as the reproducible visual case and will record that separately.

Follow-up agy finding: `agy --prompt-interactive` in another fresh isolated HOME exited with status 2 after 0.28 seconds and emitted no full-screen terminal controls. The pill remained live green during its 16 sampled frames; no socket closed. This does not reproduce the reported TUI and cannot identify the production close trigger. Sanitized summary: [agy-interactive-summary.txt](https://git.kayg.org/attachments/40a76422-a371-420c-98f0-ccf916ad7b03). I’m retaining the real-server 9 Hz redraw fixture as the reproducible visual case and will record that separately.
kayg commented 2026-10-16 15:41:27 +00:00
Author
Owner
Copy link

Correction to my prior redraw update: that run used a misspelled INFRATERNAL_BIN variable and therefore selected the worktree default binary instead of the pinned static scratch build. With the correct binary path, the 9 Hz real-PTY redraw completed (191,025 bytes, 90 title updates, 2,250 row writes) with 0 socket closes and 0 pill mutations. The pinned run's exact result and 10-second recording are attached above. The earlier 1005/empty close was observed before the staged agy workload and remains unexplained; no close frame was available in that trace. Thus the exact production close trigger is still unisolated.

Correction to my prior redraw update: that run used a misspelled `INFRATERNAL_BIN` variable and therefore selected the worktree default binary instead of the pinned static scratch build. With the correct binary path, the 9 Hz real-PTY redraw completed (191,025 bytes, 90 title updates, 2,250 row writes) with 0 socket closes and 0 pill mutations. The pinned run's exact result and 10-second recording are attached above. The earlier 1005/empty close was observed before the staged agy workload and remains unexplained; no close frame was available in that trace. Thus the exact production close trigger is still unisolated.
kayg commented 2026-10-16 15:41:36 +00:00
Author
Owner
Copy link

The final mock browser regression passed with the preview-enabled test bundle: redraw output caused zero status mutations; the 110 ms close-after-attach storm made 6 sockets/6 attaches over 2.5 seconds but the pill only changed once from live to connecting, never flashed live again, and kept the reason toast. Summary: mock-browser-final-summary.txt.

The final mock browser regression passed with the preview-enabled test bundle: redraw output caused zero status mutations; the 110 ms close-after-attach storm made 6 sockets/6 attaches over 2.5 seconds but the pill only changed once from live to connecting, never flashed live again, and kept the reason toast. Summary: [mock-browser-final-summary.txt](https://git.kayg.org/attachments/57b2f2f3-1457-4474-ac9a-22eba121f52e).
kayg commented 2026-10-16 15:43:56 +00:00
Author
Owner
Copy link

Step complete at 1fd00a3c8: resumed streams now get one fast retry, then bounded exponential backoff; close reasons survive route retirement and retries. Targeted unit gate: 10 tests, 35 assertions passed. bun run check passed. This change addresses the 8 Hz reconnect presentation loop; the real-server agy close trigger remains unisolated.

Step complete at 1fd00a3c8: resumed streams now get one fast retry, then bounded exponential backoff; close reasons survive route retirement and retries. Targeted unit gate: 10 tests, 35 assertions passed. `bun run check` passed. This change addresses the 8 Hz reconnect presentation loop; the real-server agy close trigger remains unisolated.
kayg commented 2026-10-16 16:07:44 +00:00
Author
Owner
Copy link

The first scripts/verify iterate run failed only in nine mesh node_state tests: Directory::new fell back to the package test process's current directory and tried to create crates/plugins/mesh/target/..., which does not exist when CARGO_TARGET_DIR is unset. The same test passes with the job scratch target explicitly set (CARGO_TARGET_DIR=/home/kayg/.cache/codex-scratch/pill-flicker/target). I’m rerunning iterate with that required per-job target path; no Rust source changed.

The first `scripts/verify iterate` run failed only in nine mesh `node_state` tests: `Directory::new` fell back to the package test process's current directory and tried to create `crates/plugins/mesh/target/...`, which does not exist when `CARGO_TARGET_DIR` is unset. The same test passes with the job scratch target explicitly set (`CARGO_TARGET_DIR=/home/kayg/.cache/codex-scratch/pill-flicker/target`). I’m rerunning iterate with that required per-job target path; no Rust source changed.
kayg commented 2026-10-16 16:37:18 +00:00
Author
Owner
Copy link

Commit 1 1fd00a3c8 passes the full iterate gate on the scratch target. Rust rows: fmt, VT native fmt, CLI plugin compile-out, clippy, test, and cargo-deny all PASS. Workspace test gate completed in 665 s; TypeScript suite reports 193 + 37 passing tests and 0 failures. Captured transcript: iterate-final.log.

Commit 1 `1fd00a3c8` passes the full iterate gate on the scratch target. Rust rows: fmt, VT native fmt, CLI plugin compile-out, clippy, test, and cargo-deny all PASS. Workspace test gate completed in 665 s; TypeScript suite reports 193 + 37 passing tests and 0 failures. Captured transcript: [iterate-final.log](https://git.kayg.org/attachments/5f3fc8a5-1e2c-4c92-8343-32fa8ac08451).
kayg commented 2026-10-16 16:39:35 +00:00
Author
Owner
Copy link

Commit ba01d1266 is pushed to pill-flicker. The resumed pill stays connecting until a 300 ms stable attach; connection failure toast persists through retry statuses and clears only after 300 ms stable empty status. bun run check passed; bun test src/canvas/status.test.ts: 4 pass, 0 fail, 7 assertions.

Commit `ba01d1266` is pushed to `pill-flicker`. The resumed pill stays connecting until a 300 ms stable attach; connection failure toast persists through retry statuses and clears only after 300 ms stable empty status. `bun run check` passed; `bun test src/canvas/status.test.ts`: 4 pass, 0 fail, 7 assertions.
kayg commented 2026-10-16 16:42:58 +00:00
Author
Owner
Copy link

Production-version finding: server and agent source trees at 492e1505a match base f210a4f25. In the 492e real-server trace, bridge b0 ended at +522 ms with WebSocket close 1005 and an empty reason; no mux Close frame was recorded. The page called close(4000, "connection unresponsive") at +549 ms. The resumed AttachTerm happened at +792 ms; the agy stage began at +823 ms. The 10.6 s agy/control replay then carried 134,319 output bytes with no further close or pill mutation. The actual agy invocation produced one screen render, not the reported repeated TUI redraw, so that close predates the workload and the exact production trigger remains unisolated. The synthetic ~9 Hz redraw and immediate-close regression coverage is attached: browser-mock-final.log.

Production-version finding: server and agent source trees at `492e1505a` match base `f210a4f25`. In the 492e real-server trace, bridge b0 ended at +522 ms with WebSocket close 1005 and an empty reason; no mux Close frame was recorded. The page called `close(4000, "connection unresponsive")` at +549 ms. The resumed AttachTerm happened at +792 ms; the agy stage began at +823 ms. The 10.6 s agy/control replay then carried 134,319 output bytes with no further close or pill mutation. The actual agy invocation produced one screen render, not the reported repeated TUI redraw, so that close predates the workload and the exact production trigger remains unisolated. The synthetic ~9 Hz redraw and immediate-close regression coverage is attached: [browser-mock-final.log](https://git.kayg.org/attachments/0ad6f16c-5096-4b55-b871-698a274f74f2).
kayg commented 2026-10-16 17:03:26 +00:00
Author
Owner
Copy link

The real-server browser regression passes against the pinned static server build and production tracer bundle. From workload marker +341 ms to +11,107 ms, 600 RAF samples stayed rgb(62, 158, 41), with zero status mutations and zero client/server socket closes. The PTY emitted 191,025 bytes, including 90 OSC title updates, 90 clear-screen frames, and 2,250 cursor-addressed writes. The only logged WebSocket closes were normal browser teardown after the workload (1001/1006). Transcript: browser-real-final.log.

The real-server browser regression passes against the pinned static server build and production tracer bundle. From workload marker +341 ms to +11,107 ms, 600 RAF samples stayed `rgb(62, 158, 41)`, with zero status mutations and zero client/server socket closes. The PTY emitted 191,025 bytes, including 90 OSC title updates, 90 clear-screen frames, and 2,250 cursor-addressed writes. The only logged WebSocket closes were normal browser teardown after the workload (1001/1006). Transcript: [browser-real-final.log](https://git.kayg.org/attachments/ac5706ed-de6f-4c3f-b7bb-f2b9c5fbdaad).
kayg commented 2026-10-16 17:03:42 +00:00
Author
Owner
Copy link

Commit 427c67a7a is pushed to pill-flicker. bun run check passed. Browser regressions: mock 2 passed (19 redraw frames at ~9 Hz with zero mutations; immediate-close storm made 6 sockets, 1 live→connecting transition, 0 connecting→live flashes); real server 1 passed (600 frames, one color, zero workload mutations/closes). Receipts: mock trace, real-server trace.

Commit `427c67a7a` is pushed to `pill-flicker`. `bun run check` passed. Browser regressions: mock 2 passed (19 redraw frames at ~9 Hz with zero mutations; immediate-close storm made 6 sockets, 1 live→connecting transition, 0 connecting→live flashes); real server 1 passed (600 frames, one color, zero workload mutations/closes). Receipts: [mock trace](https://git.kayg.org/attachments/0ad6f16c-5096-4b55-b871-698a274f74f2), [real-server trace](https://git.kayg.org/attachments/ac5706ed-de6f-4c3f-b7bb-f2b9c5fbdaad).
kayg commented 2026-10-16 17:04:58 +00:00
Author
Owner
Copy link

Commit 9e01a0331 is pushed to pill-flicker. The stripped static x86_64 artifact is 21,421,840 bytes versus the prior 21,422,096-byte ceiling (−256 bytes). The gate now uses that exact D243 ceiling; final fast will verify the reproducible build and both shipping targets.

Commit `9e01a0331` is pushed to `pill-flicker`. The stripped static x86_64 artifact is 21,421,840 bytes versus the prior 21,422,096-byte ceiling (−256 bytes). The gate now uses that exact D243 ceiling; final fast will verify the reproducible build and both shipping targets.
kayg commented 2026-10-16 17:13:23 +00:00
Author
Owner
Copy link

Final fast has one failure so far: auth::tests::socket_tests::restart_open_liveness_review::responsive_socket_survives_a_held_terminal_open_with_an_existing_attachment; the neighboring socket tests pass. Host load average at the failure was 7.66 11.00 14.85. This is an auth socket liveness timeout test from the allowed host-load comparison class. I will run it on a detached dev baseline checkout in the same load window after this gate exits; the full transcript is being retained for comparison.

Final fast has one failure so far: `auth::tests::socket_tests::restart_open_liveness_review::responsive_socket_survives_a_held_terminal_open_with_an_existing_attachment`; the neighboring socket tests pass. Host load average at the failure was `7.66 11.00 14.85`. This is an auth socket liveness timeout test from the allowed host-load comparison class. I will run it on a detached `dev` baseline checkout in the same load window after this gate exits; the full transcript is being retained for comparison.
kayg commented 2026-10-16 17:31:45 +00:00
Author
Owner
Copy link

Final scripts/verify fast result: fmt, VT native fmt, CLI plugin compile-out, clippy, cargo-deny, and static builds PASS; workspace tests report 448 passed, 1 failed, 48 ignored. The lone failure was auth::tests::socket_tests::restart_open_liveness_review::responsive_socket_survives_a_held_terminal_open_with_an_existing_attachment, panicking at crates/infraternal-server/src/auth/socket_clock.rs:81 with socket test stopped before its observed event (test stage 568 s; server lib test result 356.91 s).

The exact test also failed on detached origin/dev 650472b98 with the same panic after 90.00 s. Baseline host load was 11.29/15.09/17.16 before and 20.21/17.37/17.54 after; branch fast failure load was 7.66/11.00/14.85. This is the documented host-load exception; the failure is unchanged on baseline. Receipts: branch fast transcript, baseline isolated test.

Final `scripts/verify fast` result: fmt, VT native fmt, CLI plugin compile-out, clippy, cargo-deny, and static builds PASS; workspace tests report 448 passed, 1 failed, 48 ignored. The lone failure was `auth::tests::socket_tests::restart_open_liveness_review::responsive_socket_survives_a_held_terminal_open_with_an_existing_attachment`, panicking at `crates/infraternal-server/src/auth/socket_clock.rs:81` with `socket test stopped before its observed event` (test stage 568 s; server lib test result 356.91 s). The exact test also failed on detached `origin/dev` `650472b98` with the same panic after 90.00 s. Baseline host load was 11.29/15.09/17.16 before and 20.21/17.37/17.54 after; branch fast failure load was 7.66/11.00/14.85. This is the documented host-load exception; the failure is unchanged on baseline. Receipts: [branch fast transcript](https://git.kayg.org/attachments/232bca3c-ec6f-40f7-a617-97e271513607), [baseline isolated test](https://git.kayg.org/attachments/4b69a4e2-7ad6-4ed4-9152-7808d9e410e6).
kayg commented 2026-10-16 17:43:43 +00:00
Author
Owner
Copy link

TypeScript verification completed on branch 9e01a03316: compiler, tracer build/precompression/transfer budget, Rust stream differential and TypeScript suites passed; 37 tests and 34,463 assertions, 0 failures. Log: verify-ts-final.log.

Normalized critical transfer against exact base f210a4f250 (same injected build SHA/date): Brotli 247,589 → 247,544 bytes (−45); gzip 297,669 → 297,725 (+56); identity 961,716 → 961,890 (+174). Receipt: size-comparison-normalized.txt.

The final fast transcript and the exact same socket timing failure on the detached dev baseline are attached in my prior comment; all other fast rows passed. The failed test is the known host load exception.

TypeScript verification completed on branch 9e01a0331614ec6777900c34bc6be9d72486b0a7: compiler, tracer build/precompression/transfer budget, Rust stream differential and TypeScript suites passed; 37 tests and 34,463 assertions, 0 failures. Log: [verify-ts-final.log](https://git.kayg.org/attachments/249fb39b-2a1f-495d-84ea-d8662eda162f). Normalized critical transfer against exact base f210a4f25055a1c6ee4e624e36f99f3f20ead640 (same injected build SHA/date): Brotli 247,589 → 247,544 bytes (−45); gzip 297,669 → 297,725 (+56); identity 961,716 → 961,890 (+174). Receipt: [size-comparison-normalized.txt](https://git.kayg.org/attachments/9460a9be-3bb7-4487-9260-1f084a2d22bc). The final fast transcript and the exact same socket timing failure on the detached dev baseline are attached in my prior comment; all other fast rows passed. The failed test is the known host load exception.
kayg commented 2026-10-16 17:48:06 +00:00
Author
Owner
Copy link

Branch result at 9e01a03316, pushed to pill-flicker; not merged or deployed.

The client now backs off after repeated short reconnects and keeps the machine pill steady while a resumed stream proves stable. The mock close-storm regression passed with 6 attaches in 2.5 s, one live→connecting transition, zero connecting→live flashes, and one total state flip. Main-thread cost per 1,000 output frames on the perf VM was 6,894.390 ms base vs 6,860.659 ms branch (−0.489%). With normalized build identifiers, critical transfer was identity 961,716→961,890 (+174 B), gzip 297,669→297,725 (+56 B), Brotli 247,589→247,544 (−45 B). The size receipt is attached above.

Current production-build real-server demo: video, frame, trace. It is 10.000 s; the 10.688 s workload produced 191,025 PTY bytes, 90 title updates, 90 clears, and 2,250 cursor-addressed writes. Across 416 sampled animation frames, the dot stayed rgb(62, 158, 41), with zero pill mutations and zero socket closes.

The server-side close trigger remains unisolated. On the exact production server build, the agy/control replay carried 134,324 bytes without another close; the only close preceded the workload and was traced to the harness typing a long launcher path as 92 one-byte terminal inputs over ~261 ms, exceeding the 250 ms proof lease. With that harness issue removed, the 9 Hz redraw completed without a close. The installed agy CLI in a fresh isolated HOME did not render the reported sign-in TUI under its documented interactive invocation. The video therefore uses the deterministic 9 Hz sign-in-style redraw fixture, not an authentic agy screen. No server change was made without a workload-caused close to fix; this root-cause portion remains blocked on a safe reproducible production close.

TypeScript verification passed: 37 tests, 34,463 assertions, 0 failures (log attached above). Final scripts/verify fast transcript: log. The sole test failure also reproduced on the detached dev baseline (baseline log: https://git.kayg.org/attachments/4b69a4e2-7ad6-4ed4-9152-7808d9e410e6):

test result: FAILED. 448 passed; 1 failed; 48 ignored; 0 measured; 0 filtered out; finished in 356.91s
error: test failed, to rerun `-p infraternal-server --lib`
(test: 568s)
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | FAIL (101)   |
| cargo-deny                | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+

The failing test is auth::tests::socket_tests::restart_open_liveness_review::responsive_socket_survives_a_held_terminal_open_with_an_existing_attachment; it panicked at crates/infraternal-server/src/auth/socket_clock.rs:81:13 with socket test stopped before its observed event, identically on baseline.

Branch result at 9e01a0331614ec6777900c34bc6be9d72486b0a7, pushed to pill-flicker; not merged or deployed. The client now backs off after repeated short reconnects and keeps the machine pill steady while a resumed stream proves stable. The mock close-storm regression passed with 6 attaches in 2.5 s, one live→connecting transition, zero connecting→live flashes, and one total state flip. Main-thread cost per 1,000 output frames on the perf VM was 6,894.390 ms base vs 6,860.659 ms branch (−0.489%). With normalized build identifiers, critical transfer was identity 961,716→961,890 (+174 B), gzip 297,669→297,725 (+56 B), Brotli 247,589→247,544 (−45 B). The size receipt is attached above. Current production-build real-server demo: [video](https://git.kayg.org/attachments/4c705fb7-c769-424b-a372-57a3ac6b967d), [frame](https://git.kayg.org/attachments/baa70ead-9bac-44dd-8eef-2d87bd6d6318), [trace](https://git.kayg.org/attachments/bee28e61-18bf-4e05-a5c1-f225e66ded30). It is 10.000 s; the 10.688 s workload produced 191,025 PTY bytes, 90 title updates, 90 clears, and 2,250 cursor-addressed writes. Across 416 sampled animation frames, the dot stayed `rgb(62, 158, 41)`, with zero pill mutations and zero socket closes. The server-side close trigger remains unisolated. On the exact production server build, the agy/control replay carried 134,324 bytes without another close; the only close preceded the workload and was traced to the harness typing a long launcher path as 92 one-byte terminal inputs over ~261 ms, exceeding the 250 ms proof lease. With that harness issue removed, the 9 Hz redraw completed without a close. The installed agy CLI in a fresh isolated HOME did not render the reported sign-in TUI under its documented interactive invocation. The video therefore uses the deterministic 9 Hz sign-in-style redraw fixture, not an authentic agy screen. No server change was made without a workload-caused close to fix; this root-cause portion remains blocked on a safe reproducible production close. TypeScript verification passed: 37 tests, 34,463 assertions, 0 failures (log attached above). Final `scripts/verify fast` transcript: [log](https://git.kayg.org/attachments/232bca3c-ec6f-40f7-a617-97e271513607). The sole test failure also reproduced on the detached `dev` baseline (baseline log: https://git.kayg.org/attachments/4b69a4e2-7ad6-4ed4-9152-7808d9e410e6): ```text test result: FAILED. 448 passed; 1 failed; 48 ignored; 0 measured; 0 filtered out; finished in 356.91s error: test failed, to rerun `-p infraternal-server --lib` (test: 568s) +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | FAIL (101) | | cargo-deny | PASS | | static builds | PASS | +---------------------------+--------------+ ``` The failing test is `auth::tests::socket_tests::restart_open_liveness_review::responsive_socket_survives_a_held_terminal_open_with_an_existing_attachment`; it panicked at `crates/infraternal-server/src/auth/socket_clock.rs:81:13` with `socket test stopped before its observed event`, identically on baseline.
kayg commented 2026-10-16 17:50:34 +00:00
Author
Owner
Copy link

Start: branch review-pill-flicker, based on review target 9e01a0331614ec6777900c34bc6be9d72486b0a7; target diff f210a4f25055a1c6ee4e624e36f99f3f20ead640..9e01a0331614ec6777900c34bc6be9d72486b0a7. Read all 67 #227 comments; issue metadata has no parent issue link. Beginning independent verification of the client retry, status, and x86 size-cap changes.

Start: branch `review-pill-flicker`, based on review target `9e01a0331614ec6777900c34bc6be9d72486b0a7`; target diff `f210a4f25055a1c6ee4e624e36f99f3f20ead640..9e01a0331614ec6777900c34bc6be9d72486b0a7`. Read all 67 #227 comments; issue metadata has no parent issue link. Beginning independent verification of the client retry, status, and x86 size-cap changes.
kayg commented 2026-10-16 17:54:48 +00:00
Author
Owner
Copy link

P1 finding (verification in progress): a committed fake-clock regression on review-pill-flicker (073d05ec2) fails on target 9e01a0331. After a simulated 10 s route outage, the next successful Welcome is delayed 1,305 ms after the route returns; the test expects at most 50 ms for the local path under D265. This is a scheduler-level reproduction, not a perf-VM measurement yet; I’m checking the real restart and network-cut behavior before the final verdict.

P1 finding (verification in progress): a committed fake-clock regression on `review-pill-flicker` (`073d05ec2`) fails on target `9e01a0331`. After a simulated 10 s route outage, the next successful Welcome is delayed 1,305 ms after the route returns; the test expects at most 50 ms for the local path under D265. This is a scheduler-level reproduction, not a perf-VM measurement yet; I’m checking the real restart and network-cut behavior before the final verdict.
kayg commented 2026-10-16 18:02:48 +00:00
Author
Owner
Copy link

RF-1 base comparison: the same fake-clock test passes on base f210a4f25055a1c6ee4e624e36f99f3f20ead640 (1 pass, 0 fail), while target 9e01a0331 recovers 1,305 ms after the simulated route return against a 50 ms assertion. This isolates the scheduling regression to the target change; I’m still verifying it with the production browser on the perf VM.

RF-1 base comparison: the same fake-clock test passes on base `f210a4f25055a1c6ee4e624e36f99f3f20ead640` (1 pass, 0 fail), while target `9e01a0331` recovers 1,305 ms after the simulated route return against a 50 ms assertion. This isolates the scheduling regression to the target change; I’m still verifying it with the production browser on the perf VM.
kayg commented 2026-10-16 18:05:40 +00:00
Author
Owner
Copy link

P2 finding: a committed toast-ownership regression (bbb1c2e36) fails on target 9e01a0331. When the shared toast is replaced by a newer message, the next retry shows the same connection reason again and replaces the newer message. The controller's documented onClose callback runs when another toast replaces a message; the test models that behavior.

P2 finding: a committed toast-ownership regression (`bbb1c2e36`) fails on target `9e01a0331`. When the shared toast is replaced by a newer message, the next retry shows the same connection reason again and replaces the newer message. The controller's documented `onClose` callback runs when another toast replaces a message; the test models that behavior.
kayg commented 2026-10-16 18:14:02 +00:00
Author
Owner
Copy link

P1 confirmed on real production browser builds. On the 4 vCPU Intel i9-9900K perf VM (8.26 GB RAM; per-trial load averages 0.16–1.39), three alternating runs after a 10 s browser offline interval measured return-to-attach at base 1,863/188/572 ms (median 572 ms) and head 4,519/2,921/4,939 ms (median 4,519 ms). Return-to-live was base 1,871/203/580 ms (median 580 ms) and head 4,827/3,230/5,251 ms (median 4,827 ms). Head's median attach delay is +3,947 ms; median live delay is +4,247 ms. A separate fake-clock case also fails at 1,305 ms against the 50 ms reconnect allowance; the same test passes on base. Logs and all six trial screenshots are in /home/kayg/.cache/codex-scratch/pill-flicker-review/evidence/real-browser/.

base offline
head offline
head live

P1 confirmed on real production browser builds. On the 4 vCPU Intel i9-9900K perf VM (8.26 GB RAM; per-trial load averages 0.16–1.39), three alternating runs after a 10 s browser offline interval measured return-to-attach at base 1,863/188/572 ms (median 572 ms) and head 4,519/2,921/4,939 ms (median 4,519 ms). Return-to-live was base 1,871/203/580 ms (median 580 ms) and head 4,827/3,230/5,251 ms (median 4,827 ms). Head's median attach delay is +3,947 ms; median live delay is +4,247 ms. A separate fake-clock case also fails at 1,305 ms against the 50 ms reconnect allowance; the same test passes on base. Logs and all six trial screenshots are in `/home/kayg/.cache/codex-scratch/pill-flicker-review/evidence/real-browser/`. ![base offline](https://git.kayg.org/attachments/4f989026-3166-4060-babb-5039a497d41b) ![head offline](https://git.kayg.org/attachments/5b4d1361-024a-494a-8b48-a2e3649b9709) ![head live](https://git.kayg.org/attachments/5514abb3-7bc4-4f37-81eb-84a2e3649b9709)
kayg commented 2026-10-16 18:29:04 +00:00
Author
Owner
Copy link

Review evidence update (review branch head bbb1c2e36): the real PTY server-restart typing test passed on base and head. A real-browser 10 s offline test queued echo CUT_<variant> while disconnected and passed on both builds with exactly two server-captured marker occurrences (command echo and result), with replay complete before the pill returned live.

Five real-PTY redraw trials per build on the 4 vCPU Intel i9-9900K perf VM (8.26 GB RAM) yielded 1,122 base output frames / 7,262.447 ms task time (6,472.769 ms per 1,000 frames) and 1,112 head frames / 6,826.133 ms (6,138.609 ms per 1,000). All ten trials recorded zero status mutations during redraw, 607–609 frame samples, and zero client/server socket closes. Loads before trials were 0.20–2.08 (base) and 0.21–1.61 (head). Raw logs: /home/kayg/.cache/codex-scratch/pill-flicker-review/evidence/redraw/.

Review evidence update (review branch head `bbb1c2e36`): the real PTY server-restart typing test passed on base and head. A real-browser 10 s offline test queued `echo CUT_<variant>` while disconnected and passed on both builds with exactly two server-captured marker occurrences (command echo and result), with replay complete before the pill returned live. Five real-PTY redraw trials per build on the 4 vCPU Intel i9-9900K perf VM (8.26 GB RAM) yielded 1,122 base output frames / 7,262.447 ms task time (6,472.769 ms per 1,000 frames) and 1,112 head frames / 6,826.133 ms (6,138.609 ms per 1,000). All ten trials recorded zero status mutations during redraw, 607–609 frame samples, and zero client/server socket closes. Loads before trials were 0.20–2.08 (base) and 0.21–1.61 (head). Raw logs: `/home/kayg/.cache/codex-scratch/pill-flicker-review/evidence/redraw/`.
kayg commented 2026-10-16 18:29:09 +00:00
Author
Owner
Copy link

Correction to the previous screenshot comment: the attached head-live screenshot link is https://git.kayg.org/attachments/5514abb3-7bc4-4f37-81eb-84cd288eb9f7.

Correction to the previous screenshot comment: the attached head-live screenshot link is https://git.kayg.org/attachments/5514abb3-7bc4-4f37-81eb-84cd288eb9f7.
kayg commented 2026-10-16 18:34:55 +00:00
Author
Owner
Copy link

Correction to the synthetic regression-test note: the first version advanced Date.now but not performance.now, so it did not advance base's ten-second fast-recovery deadline. I corrected the test to advance both clocks and return at 9.5 s, inside base's deadline. The corrected test passes on f210a4f25055a1c6ee4e624e36f99f3f20ead640 (1 pass) and fails on 9e01a0331 at 1,805 ms against the 50 ms allowance. The correction is committed on review-pill-flicker as eed9e64b4. The separate three-pair real-browser comparison after an exact 10 s cut is unchanged: median return-to-live 580 ms base vs 4,827 ms head.

Correction to the synthetic regression-test note: the first version advanced `Date.now` but not `performance.now`, so it did not advance base's ten-second fast-recovery deadline. I corrected the test to advance both clocks and return at 9.5 s, inside base's deadline. The corrected test passes on `f210a4f25055a1c6ee4e624e36f99f3f20ead640` (1 pass) and fails on `9e01a0331` at 1,805 ms against the 50 ms allowance. The correction is committed on `review-pill-flicker` as `eed9e64b4`. The separate three-pair real-browser comparison after an exact 10 s cut is unchanged: median return-to-live 580 ms base vs 4,827 ms head.
kayg commented 2026-10-16 18:40:44 +00:00
Author
Owner
Copy link

Representative raw perf-VM logs attached: base 10 s recovery, head 10 s recovery, base redraw, head redraw. Full per-trial logs remain under /home/kayg/.cache/codex-scratch/pill-flicker-review/evidence/.

Representative raw perf-VM logs attached: [base 10 s recovery](https://git.kayg.org/attachments/5e629e6d-6166-4819-899b-6641839aa3d0), [head 10 s recovery](https://git.kayg.org/attachments/04f58e4f-b7eb-4600-934a-24a7217fb509), [base redraw](https://git.kayg.org/attachments/5e8d9060-9def-46a4-b92e-122f14b96a1a), [head redraw](https://git.kayg.org/attachments/457dd767-37c9-44fb-a483-be1b4819a406). Full per-trial logs remain under `/home/kayg/.cache/codex-scratch/pill-flicker-review/evidence/`.
kayg commented 2026-10-16 18:41:15 +00:00
Author
Owner
Copy link

Single-restart recovery update: three real production-browser runs measured ready-to-live at base 23/406/402 ms (median 402 ms) and head 324/326/335 ms (median 326 ms). Head replay completed 17–25 ms after server readiness (median 22 ms); live followed after the 300 ms stability interval. The one-restart path did not regress. Raw trial logs are in the attached perf-vm-evidence.tar contents and local evidence directory.

Single-restart recovery update: three real production-browser runs measured ready-to-live at base 23/406/402 ms (median 402 ms) and head 324/326/335 ms (median 326 ms). Head replay completed 17–25 ms after server readiness (median 22 ms); live followed after the 300 ms stability interval. The one-restart path did not regress. Raw trial logs are in the attached `perf-vm-evidence.tar` contents and local evidence directory.
kayg commented 2026-10-16 18:55:16 +00:00
Author
Owner
Copy link

P2 confirmed for D243: exact-head scripts/verify fast builds the pinned Rust 1.99.0 x86_64 static artifact at 21,422,096 bytes, while commit 9e01a0331 sets the exact cap to 21,421,840 bytes. The gate output is:

PT_INTERP: absent; size: 21422096 bytes; cap: 21421840 bytes; delta: +256 bytes
binary exceeds the 21421840-byte size budget: 21422096 bytes

This artifact is statically linked and stripped. The changed cap does not match the artifact produced by the gate. Full output is being preserved under /home/kayg/.cache/codex-scratch/pill-flicker-review/fast-gate.log.

P2 confirmed for D243: exact-head `scripts/verify fast` builds the pinned Rust 1.99.0 x86_64 static artifact at 21,422,096 bytes, while commit `9e01a0331` sets the exact cap to 21,421,840 bytes. The gate output is: ``` PT_INTERP: absent; size: 21422096 bytes; cap: 21421840 bytes; delta: +256 bytes binary exceeds the 21421840-byte size budget: 21422096 bytes ``` This artifact is statically linked and stripped. The changed cap does not match the artifact produced by the gate. Full output is being preserved under `/home/kayg/.cache/codex-scratch/pill-flicker-review/fast-gate.log`.
kayg commented 2026-10-16 19:20:33 +00:00
Author
Owner
Copy link

Review complete. Verdict: do not merge. P0: 0, P1: 1, P2: 2, P3: 0.

Reviewed target 9e01a0331614ec6777900c34bc6be9d72486b0a7; review-only branch review-pill-flicker is at eed9e64b4. Three failing regression tests are committed there; no implementation files were changed.

RF-1 is P1: after a 10 s network cut, three alternating production-build runs measured median return-to-attach of 572 ms on base vs 4,519 ms on head, and return-to-live of 580 ms vs 4,827 ms. The corrected clock test also fails on head at 1,805 ms against a 50 ms local-path bound. A single legitimate server restart remained fast: median ready-to-live 402 ms base vs 326 ms head. Redraw verification recorded zero status mutations and zero socket closes across five trials per build.

RF-2 is P2: a repeated close reason reclaims the shared toast after a newer message replaces it. RF-3 is P2: the exact x86_64 artifact is 21,422,096 bytes against the new 21,421,840-byte cap (+256 bytes); aarch64 is exactly at its cap.

TypeScript fast row: 37 tests across 10 files passed, 34,463 assertions, 0 failures. The Rust workspace fast row had 448 passed, 1 failed, 48 ignored; the failing queue test passed when run alone (1 passed, 496 filtered), so that failure did not reproduce in isolation. Full fast gate remains red.

scripts/verify fast output on the target head (verbatim summary):

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | FAIL (101)   |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Static size failure output:

PT_INTERP: absent; size: 21422096 bytes; cap: 21421840 bytes; delta: +256 bytes
binary exceeds the 21421840-byte size budget: 21422096 bytes

Full logs: fast-gate.log, ts-fast.log, isolated Rust retry. Recovery screenshots and benchmark logs were attached in earlier comments.

Review complete. Verdict: **do not merge**. P0: 0, P1: 1, P2: 2, P3: 0. Reviewed target `9e01a0331614ec6777900c34bc6be9d72486b0a7`; review-only branch `review-pill-flicker` is at `eed9e64b4`. Three failing regression tests are committed there; no implementation files were changed. RF-1 is P1: after a 10 s network cut, three alternating production-build runs measured median return-to-attach of 572 ms on base vs 4,519 ms on head, and return-to-live of 580 ms vs 4,827 ms. The corrected clock test also fails on head at 1,805 ms against a 50 ms local-path bound. A single legitimate server restart remained fast: median ready-to-live 402 ms base vs 326 ms head. Redraw verification recorded zero status mutations and zero socket closes across five trials per build. RF-2 is P2: a repeated close reason reclaims the shared toast after a newer message replaces it. RF-3 is P2: the exact x86_64 artifact is 21,422,096 bytes against the new 21,421,840-byte cap (+256 bytes); aarch64 is exactly at its cap. TypeScript fast row: 37 tests across 10 files passed, 34,463 assertions, 0 failures. The Rust workspace fast row had 448 passed, 1 failed, 48 ignored; the failing queue test passed when run alone (1 passed, 496 filtered), so that failure did not reproduce in isolation. Full fast gate remains red. `scripts/verify fast` output on the target head (verbatim summary): ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | FAIL (101) | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` Static size failure output: ``` PT_INTERP: absent; size: 21422096 bytes; cap: 21421840 bytes; delta: +256 bytes binary exceeds the 21421840-byte size budget: 21422096 bytes ``` Full logs: [fast-gate.log](https://git.kayg.org/attachments/f5b59d94-7ea5-48b7-b6a2-8b5286a37af2), [ts-fast.log](https://git.kayg.org/attachments/2f38a521-b385-4947-b13d-98e351c9210c), [isolated Rust retry](https://git.kayg.org/attachments/89080af0-c71a-4c66-a5a7-f849abd6d5db). Recovery screenshots and benchmark logs were attached in earlier comments.
kayg commented 2026-10-16 19:23:26 +00:00
Author
Owner
Copy link

Resume start: branch pill-flicker at 9e01a0331614ec6777900c34bc6be9d72486b0a7; current origin/dev base is 650472b987dbd1fe7192248dd1367d0766c5fb79. Read all 79 #227 comments; issue metadata has no parent link. origin/dev is at the mesh-autoswitch night-release merge, but its log does not yet contain the mesh-unreachable merge or #229 R5. I’ll first cherry-pick and verify the three review regressions, then rebase and keep the size cap at the new base value until R5 lands.

Resume start: branch `pill-flicker` at `9e01a0331614ec6777900c34bc6be9d72486b0a7`; current `origin/dev` base is `650472b987dbd1fe7192248dd1367d0766c5fb79`. Read all 79 #227 comments; issue metadata has no parent link. `origin/dev` is at the mesh-autoswitch night-release merge, but its log does not yet contain the mesh-unreachable merge or #229 R5. I’ll first cherry-pick and verify the three review regressions, then rebase and keep the size cap at the new base value until R5 lands.
kayg commented 2026-10-16 19:23:56 +00:00
Author
Owner
Copy link

Review regression reproduction: cherry-picked 073d05ec2, bbb1c2e36 and eed9e64b4 as three test-only commits (53786be21, d636aa0eb, fdc4be474). On the prior branch implementation, bun test src/connection-restart.test.ts src/canvas/status.test.ts produced 8 pass / 2 fail: after a simulated 9.5 s outage the first resumed Welcome was 1,805 ms late (expected <=50 ms); after a newer toast replaced the close reason, the next retry reclaimed the toast. This confirms both review findings before fixes. The branch also retains the prior 300 ms stable-attach behavior for the close-storm regression.

Review regression reproduction: cherry-picked `073d05ec2`, `bbb1c2e36` and `eed9e64b4` as three test-only commits (`53786be21`, `d636aa0eb`, `fdc4be474`). On the prior branch implementation, `bun test src/connection-restart.test.ts src/canvas/status.test.ts` produced 8 pass / 2 fail: after a simulated 9.5 s outage the first resumed Welcome was 1,805 ms late (expected <=50 ms); after a newer toast replaced the close reason, the next retry reclaimed the toast. This confirms both review findings before fixes. The branch also retains the prior 300 ms stable-attach behavior for the close-storm regression.
kayg commented 2026-10-16 19:30:46 +00:00
Author
Owner
Copy link

Client fixes completed on rebased pill-flicker: 96bee808f restores a 5–10 ms retry cadence after unwelcomed path failures and reconnects synchronously on online/visible events; ea5cf6639 preserves shared-toast identity so the same close reason cannot replace a newer message. The review regressions passed after the fixes: bun test src/connection-restart.test.ts src/canvas/status.test.ts — 11 pass, 0 fail, 33 assertions; bun run check passed. Current origin/dev lacks mesh-unreachable and #229 R5, so the static cap remains exactly the base value, 21,422,096 bytes.

Client fixes completed on rebased `pill-flicker`: `96bee808f` restores a 5–10 ms retry cadence after unwelcomed path failures and reconnects synchronously on online/visible events; `ea5cf6639` preserves shared-toast identity so the same close reason cannot replace a newer message. The review regressions passed after the fixes: `bun test src/connection-restart.test.ts src/canvas/status.test.ts` — 11 pass, 0 fail, 33 assertions; `bun run check` passed. Current `origin/dev` lacks mesh-unreachable and #229 R5, so the static cap remains exactly the base value, 21,422,096 bytes.
kayg commented 2026-10-16 19:32:19 +00:00
Author
Owner
Copy link

Rebased browser regressions passed on the current production tracer build (TRACER_TEST_PORT=4427, origin/dev 650472b98 plus the pill fix): 2 Playwright specs passed. The 9 Hz mock redraw produced zero status/dot mutations across 151 frame samples. The 110 ms immediate-close storm opened 4 sockets and 4 attachments; the pill changed once from live to connecting, with zero live promotions/flashes during retries. Receipt: playwright log.

Rebased browser regressions passed on the current production tracer build (`TRACER_TEST_PORT=4427`, `origin/dev` 650472b98 plus the pill fix): 2 Playwright specs passed. The 9 Hz mock redraw produced zero status/dot mutations across 151 frame samples. The 110 ms immediate-close storm opened 4 sockets and 4 attachments; the pill changed once from live to connecting, with zero live promotions/flashes during retries. Receipt: [playwright log](https://git.kayg.org/attachments/471e1f7a-c6d0-430b-a9a2-d068346b2236).
kayg commented 2026-10-16 19:56:23 +00:00
Author
Owner
Copy link

Iterate gate finding: infraternal-server passed 452 tests and failed 2 after the full parallel suite ran 416.32 s. auth::tests::socket_tests::restart_open_liveness_review::responsive_socket_survives_a_held_terminal_open_with_an_existing_attachment panicked at crates/infraternal-server/src/auth/socket_clock.rs:81 (“socket test stopped before its observed event”); remote_runtime::tests::standby::control_queue::saturated_native_control_queue_does_not_hide_primary_terminal_output failed live_connection.close_reason().is_none() at crates/infraternal-server/src/remote_control_queue_tests.rs:216. The remaining 450 server tests passed (48 ignored). Isolated reruns are next.

Iterate gate finding: `infraternal-server` passed 452 tests and failed 2 after the full parallel suite ran 416.32 s. `auth::tests::socket_tests::restart_open_liveness_review::responsive_socket_survives_a_held_terminal_open_with_an_existing_attachment` panicked at `crates/infraternal-server/src/auth/socket_clock.rs:81` (“socket test stopped before its observed event”); `remote_runtime::tests::standby::control_queue::saturated_native_control_queue_does_not_hide_primary_terminal_output` failed `live_connection.close_reason().is_none()` at `crates/infraternal-server/src/remote_control_queue_tests.rs:216`. The remaining 450 server tests passed (48 ignored). Isolated reruns are next.
kayg commented 2026-10-16 20:01:29 +00:00
Author
Owner
Copy link

Iterate follow-up: both gate failures pass when isolated on this branch. responsive_socket_survives_a_held_terminal_open_with_an_existing_attachment: 1 passed in 1.20 s. saturated_native_control_queue_does_not_hide_primary_terminal_output (with --features remote): 1 passed in 0.11 s (CONTROL_SATURATION active=true queued=1 native_primary_live=true shadow_output=0). The broad run had 452 passed, 2 failures, 48 ignored in 416.32 s; the parallel auth suite emitted multiple over-60-second warnings. No server code changed in this UI lane.

Iterate follow-up: both gate failures pass when isolated on this branch. `responsive_socket_survives_a_held_terminal_open_with_an_existing_attachment`: 1 passed in 1.20 s. `saturated_native_control_queue_does_not_hide_primary_terminal_output` (with `--features remote`): 1 passed in 0.11 s (`CONTROL_SATURATION active=true queued=1 native_primary_live=true shadow_output=0`). The broad run had 452 passed, 2 failures, 48 ignored in 416.32 s; the parallel auth suite emitted multiple over-60-second warnings. No server code changed in this UI lane.
kayg commented 2026-10-16 20:31:21 +00:00
Author
Owner
Copy link

Final scripts/verify fast finding, head a33f7d093: the Rust gate reported test FAIL (101) (server tests: 453 passed, 1 failed, 48 ignored; the failing row was remote_runtime::tests::standby::control_queue::saturated_native_control_queue_does_not_hide_primary_terminal_output) and static builds FAIL (1). x86_64 measured 21,320,400 bytes at its unchanged cap (delta +0); aarch64 measured 19,517,536 against the unchanged 19,513,440 cap (delta +4,096). origin/dev is 650472b98 and its history has the mesh-autoswitch merge but not #229/R5; I did not change either cap. Raw transcript: fast.log.

Verbatim final Rust gate table:
| Rust gate | Result |
+---------------------------+--------------+
| fmt | PASS |
| VT native fmt | PASS |
| CLI plugin compile-out | PASS |
| clippy | PASS |
| test | FAIL (101) |
| cargo-deny | PASS |
| static builds | FAIL (1) |
+---------------------------+--------------+

Final `scripts/verify fast` finding, head `a33f7d093`: the Rust gate reported `test FAIL (101)` (server tests: 453 passed, 1 failed, 48 ignored; the failing row was `remote_runtime::tests::standby::control_queue::saturated_native_control_queue_does_not_hide_primary_terminal_output`) and `static builds FAIL (1)`. x86_64 measured 21,320,400 bytes at its unchanged cap (delta +0); aarch64 measured 19,517,536 against the unchanged 19,513,440 cap (delta +4,096). `origin/dev` is `650472b98` and its history has the mesh-autoswitch merge but not #229/R5; I did not change either cap. Raw transcript: [fast.log](https://git.kayg.org/attachments/ae18277d-1728-4587-a7a5-e6993b210223). Verbatim final Rust gate table: | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | FAIL (101) | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+
kayg commented 2026-10-16 20:34:18 +00:00
Author
Owner
Copy link

Fast-gate rerun of the server row: cargo test -p infraternal-server --features remote --lib saturated_native_control_queue_does_not_hide_primary_terminal_output -- --nocapture passed 1/1 in 0.10 s (CONTROL_SATURATION active=true queued=1 native_primary_live=true shadow_output=0). The parallel workspace run is the only test failure; no Rust sources or dependency features changed in this branch.

Fast-gate rerun of the server row: `cargo test -p infraternal-server --features remote --lib saturated_native_control_queue_does_not_hide_primary_terminal_output -- --nocapture` passed 1/1 in 0.10 s (`CONTROL_SATURATION active=true queued=1 native_primary_live=true shadow_output=0`). The parallel workspace run is the only test failure; no Rust sources or dependency features changed in this branch.
kayg commented 2026-10-16 20:36:39 +00:00
Author
Owner
Copy link

Critical-path transfer comparison (same production build recipe, baseline origin/dev 650472b98 vs branch a33f7d093): identity 962,037→963,333 bytes (+1,296), gzip 297,814→298,087 (+273), Brotli 247,707→248,056 (+349). Both remain under the existing transfer budgets; this is growth on the critical path and is recorded for review. Full build and transfer-budget receipts: dist-comparison.log.

Critical-path transfer comparison (same production build recipe, baseline `origin/dev` 650472b98 vs branch `a33f7d093`): identity 962,037→963,333 bytes (+1,296), gzip 297,814→298,087 (+273), Brotli 247,707→248,056 (+349). Both remain under the existing transfer budgets; this is growth on the critical path and is recorded for review. Full build and transfer-budget receipts: [dist-comparison.log](https://git.kayg.org/attachments/2c071364-28f5-48ba-8f11-1151301d41e2).
kayg commented 2026-10-16 20:41:28 +00:00
Author
Owner
Copy link

Final real-server redraw run: production x86_64 binary (21,320,400 bytes) plus production tracer bundle; 10.564 s workload, 191,025 output bytes, 90 OSC titles, 90 clears, 2,250 cursor-addressed rows, 616 sampled frames, 0 pill-status mutations, and 0 client/server socket closes. The 10-second production real-server video and raw trace are attached: video, trace.

The staged agy --prompt-interactive attempt still exits before rendering the sign-in TUI in an isolated HOME, so this recording is the synthetic sign-in redraw fixture and is labeled accordingly; it is not an agy capture.

Final real-server redraw run: production x86_64 binary (21,320,400 bytes) plus production tracer bundle; 10.564 s workload, 191,025 output bytes, 90 OSC titles, 90 clears, 2,250 cursor-addressed rows, 616 sampled frames, 0 pill-status mutations, and 0 client/server socket closes. The 10-second production real-server video and raw trace are attached: [video](https://git.kayg.org/attachments/dc61c677-3bce-4594-97a0-782f4ed24e24), [trace](https://git.kayg.org/attachments/b2c1cb46-49a1-4fb0-b6cc-9ffd24c796da). The staged `agy --prompt-interactive` attempt still exits before rendering the sign-in TUI in an isolated HOME, so this recording is the synthetic sign-in redraw fixture and is labeled accordingly; it is not an agy capture.
kayg commented 2026-10-16 20:53:49 +00:00
Author
Owner
Copy link

Correction to the real-server demo note: frameSamples=616 is the browser harness’s functional trace count from the shared build host, not performance evidence. The attached redraw log supports only workload byte/control counts, stable pill state, and zero closes; timing and main-thread measurements are being taken on the perf VM.

Correction to the real-server demo note: `frameSamples=616` is the browser harness’s functional trace count from the shared build host, not performance evidence. The attached redraw log supports only workload byte/control counts, stable pill state, and zero closes; timing and main-thread measurements are being taken on the perf VM.
kayg commented 2026-10-16 21:09:55 +00:00
Author
Owner
Copy link

Perf-VM recovery harness finding: the first 10 base and 10 branch invocations never reached their initial live precondition, so none produced latency samples. A focused base startup trace shows one WebSocket opened and sent Hello, two Subscribe frames, and Call; the bridge remained open with no attached terminal and no server-to-browser protocol frames before timeout. No performance result is claimed. Diagnostic trace: network-debug-base.log. I am checking warning-level server output to resolve the test setup before retrying the measurement.

Perf-VM recovery harness finding: the first 10 base and 10 branch invocations never reached their initial `live` precondition, so none produced latency samples. A focused base startup trace shows one WebSocket opened and sent `Hello`, two `Subscribe` frames, and `Call`; the bridge remained open with no attached terminal and no server-to-browser protocol frames before timeout. No performance result is claimed. Diagnostic trace: [network-debug-base.log](https://git.kayg.org/attachments/d3f6e16f-912a-4ecc-863a-53bb15517413). I am checking warning-level server output to resolve the test setup before retrying the measurement.
kayg commented 2026-10-16 21:12:30 +00:00
Author
Owner
Copy link

Perf VM real-server startup trace: the page opened its /v1/ws socket, but no AttachTerm was sent and the pill stayed connecting. The embedded agent repeatedly logged infraternal agent failed: address in use; the test server ran as root, whose embedded-agent state resolves to shared /run/infraternal and collided with the host socket. This is a harness isolation issue, not a terminal-byte close. I am rerunning with /run/infraternal isolated in a private mount namespace. Receipt: network-debug-perf-vm.log.

Perf VM real-server startup trace: the page opened its /v1/ws socket, but no AttachTerm was sent and the pill stayed connecting. The embedded agent repeatedly logged `infraternal agent failed: address in use`; the test server ran as root, whose embedded-agent state resolves to shared `/run/infraternal` and collided with the host socket. This is a harness isolation issue, not a terminal-byte close. I am rerunning with `/run/infraternal` isolated in a private mount namespace. Receipt: [network-debug-perf-vm.log](https://git.kayg.org/attachments/8d663e4d-44d3-45b0-b243-4ef05f27cbb9).
kayg commented 2026-10-16 21:15:16 +00:00
Author
Owner
Copy link

The isolated real-server rerun passed. With /run/infraternal private to the test namespace, the browser sent AttachTerm and terminal credit, then startup reached live; the Playwright diagnostic passed (6.8 s). This confirms the prior address-in-use failure was test-host socket sharing. Receipt: network-debug-perf-vm-isolated.log. I am now measuring 10 network-return trials per dist.

The isolated real-server rerun passed. With `/run/infraternal` private to the test namespace, the browser sent `AttachTerm` and terminal credit, then startup reached `live`; the Playwright diagnostic passed (6.8 s). This confirms the prior address-in-use failure was test-host socket sharing. Receipt: [network-debug-perf-vm-isolated.log](https://git.kayg.org/attachments/f4729858-2221-477c-bb2f-38e82bec3ee7). I am now measuring 10 network-return trials per dist.
kayg commented 2026-10-16 21:22:29 +00:00
Author
Owner
Copy link

Build-host real-server replay used the production-equivalent base web dist and real server binary. It emitted 90 redraw frames containing 16 KiB OSC 8 links and lines, OSC 0/2 titles, bracketed-paste/focus toggles, synchronized output, DA/DSR/XTVERSION/keyboard/graphics queries (2,972,055 output bytes). The trace recorded zero WebSocket closes, zero status mutations, and a single replay. This workload does not explain the production close. The staged agy --prompt-interactive attempt still exits before drawing, so no exact sign-in byte capture is available. Receipt: control-replay-base.log.

Build-host real-server replay used the production-equivalent base web dist and real server binary. It emitted 90 redraw frames containing 16 KiB OSC 8 links and lines, OSC 0/2 titles, bracketed-paste/focus toggles, synchronized output, DA/DSR/XTVERSION/keyboard/graphics queries (2,972,055 output bytes). The trace recorded zero WebSocket closes, zero status mutations, and a single replay. This workload does not explain the production close. The staged `agy --prompt-interactive` attempt still exits before drawing, so no exact sign-in byte capture is available. Receipt: [control-replay-base.log](https://git.kayg.org/attachments/7454ec58-10d8-48c1-91ea-08fba2fa8cfb).
kayg commented 2026-10-16 21:24:42 +00:00
Author
Owner
Copy link

Perf VM (infraternal-perf-test, x86_64, 4 vCPU, 7.7 GiB) completed 10 alternating 10-second offline trials per dist before the in-flight-handshake fix. Median return-to-live was 1,575.45 ms for base and 1,866.50 ms for branch (+291.05 ms); ranges were 462.5–8,810.5 ms and 447.3–4,736.3 ms. Load averages were 0.89/0.86/0.84 before and 1.37/1.51/1.18 after. This does not meet the ±100 ms target and confirms the need to wake a stalled opening socket. Receipt: network-recovery-before-fix.log.

Perf VM (infraternal-perf-test, x86_64, 4 vCPU, 7.7 GiB) completed 10 alternating 10-second offline trials per dist before the in-flight-handshake fix. Median return-to-live was 1,575.45 ms for base and 1,866.50 ms for branch (+291.05 ms); ranges were 462.5–8,810.5 ms and 447.3–4,736.3 ms. Load averages were 0.89/0.86/0.84 before and 1.37/1.51/1.18 after. This does not meet the ±100 ms target and confirms the need to wake a stalled opening socket. Receipt: [network-recovery-before-fix.log](https://git.kayg.org/attachments/9526a46c-af49-4fc5-9598-292371ecf0db).
kayg commented 2026-10-16 21:29:09 +00:00
Author
Owner
Copy link

The current production dist/server recording is attached: pill-flicker-real-server-after-fix.mp4. The 10.5-second synthetic sign-in-style redraw produced 611 frame samples, one stable green state, zero status mutations, and zero client/server socket closes. The staged agy command still exits before rendering, so the video is labeled synthetic.

The current production dist/server recording is attached: [pill-flicker-real-server-after-fix.mp4](https://git.kayg.org/attachments/404710b7-0292-4373-aad6-12d00e95b543). The 10.5-second synthetic sign-in-style redraw produced 611 frame samples, one stable green state, zero status mutations, and zero client/server socket closes. The staged agy command still exits before rendering, so the video is labeled synthetic.
kayg commented 2026-10-16 21:31:17 +00:00
Author
Owner
Copy link

git log origin/dev still shows 650472b98 at the tip and no mesh-unreachable merge or #229/R5 checkout-path change. I left both static size caps unchanged as instructed.

`git log origin/dev` still shows `650472b98` at the tip and no mesh-unreachable merge or #229/R5 checkout-path change. I left both static size caps unchanged as instructed.
kayg commented 2026-10-16 21:33:12 +00:00
Author
Owner
Copy link

Final rebuilt dist recording attached: pill-flicker-real-server-final.mp4. The 10.7-second real-server redraw produced 613 frame samples, one constant green state, zero status mutations, and zero client/server socket closes. This is the synthetic sign-in-style workload because the staged agy command exits before rendering.

Final rebuilt dist recording attached: [pill-flicker-real-server-final.mp4](https://git.kayg.org/attachments/fc9f00ac-5353-4dae-9eee-044d602991af). The 10.7-second real-server redraw produced 613 frame samples, one constant green state, zero status mutations, and zero client/server socket closes. This is the synthetic sign-in-style workload because the staged agy command exits before rendering.
kayg commented 2026-10-16 21:40:33 +00:00
Author
Owner
Copy link

Post-fix 10 s outage comparison, 10 alternating trials each on perf VM (4 vCPU, 7.7 GiB; load average 0.88/1.27/1.26 before, 1.16/1.22/1.25 after), current branch server 28212cdd and identical browser build except web variant: base median return-to-live 1,327.45 ms (268.1–3,789.8); branch 3,151.75 ms (1,503.5–5,314.1). This misses the ±100 ms target, so I am tracing the remaining recovery delay before proceeding. Full socket-event log: network-recovery-final.log.

Post-fix 10 s outage comparison, 10 alternating trials each on perf VM (4 vCPU, 7.7 GiB; load average 0.88/1.27/1.26 before, 1.16/1.22/1.25 after), current branch server 28212cdd and identical browser build except web variant: base median return-to-live 1,327.45 ms (268.1–3,789.8); branch 3,151.75 ms (1,503.5–5,314.1). This misses the ±100 ms target, so I am tracing the remaining recovery delay before proceeding. Full socket-event log: [network-recovery-final.log](https://git.kayg.org/attachments/1d51d2fb-86a9-4a3d-a2a7-e513c763e80b).
kayg commented 2026-10-16 21:51:10 +00:00
Author
Owner
Copy link

Exact production server 492e1505a comparison, 10 alternating 10 s outage trials on the 4-vCPU, 7.7 GiB perf VM (load average 0.63/1.19/1.26 before, 1.59/1.56/1.38 after): base median return-to-live 1,878.75 ms (508.2–4,458.0); branch 3,945.0 ms (1,906.2–4,645.1). Socket traces show the online handler closed the in-flight CONNECTING WebSocket with code 1000, discarding a handshake the browser could otherwise resume. I am changing the hint to reset retry state while keeping that socket. Log: network-recovery-prod492e.log.

Exact production server 492e1505a comparison, 10 alternating 10 s outage trials on the 4-vCPU, 7.7 GiB perf VM (load average 0.63/1.19/1.26 before, 1.59/1.56/1.38 after): base median return-to-live 1,878.75 ms (508.2–4,458.0); branch 3,945.0 ms (1,906.2–4,645.1). Socket traces show the online handler closed the in-flight CONNECTING WebSocket with code 1000, discarding a handshake the browser could otherwise resume. I am changing the hint to reset retry state while keeping that socket. Log: [network-recovery-prod492e.log](https://git.kayg.org/attachments/ef4d95c5-9157-497b-9f6d-51f06d6cfd6a).
kayg commented 2026-10-16 22:01:04 +00:00
Author
Owner
Copy link

After preserving an in-flight handshake, the 10-trial comparison on production server artifact 492e1505a measured base median 1,462.45 ms (444.5–2,886.1) and branch 1,849.2 ms (1,102.7–5,316.4), delta +386.75 ms. The online event no longer retires the active socket, but this sample remains outside the ±100 ms target and has wide variance. I am extending the alternating run and inspecting per-trial socket events before changing retry behavior again. Log: network-recovery-prod492e-final2.log.

After preserving an in-flight handshake, the 10-trial comparison on production server artifact 492e1505a measured base median 1,462.45 ms (444.5–2,886.1) and branch 1,849.2 ms (1,102.7–5,316.4), delta +386.75 ms. The online event no longer retires the active socket, but this sample remains outside the ±100 ms target and has wide variance. I am extending the alternating run and inspecting per-trial socket events before changing retry behavior again. Log: [network-recovery-prod492e-final2.log](https://git.kayg.org/attachments/15cc4f91-d19f-4fa9-ae23-f90a16194de7).
kayg commented 2026-10-16 22:10:27 +00:00
Author
Owner
Copy link

Recovery comparison with handshake-preserving online hints: 20 alternating 10 s outage trials per variant against production server artifact 492e1505a (SHA-256 28212cdd96ef716d81502dc3991f8556460f75f8899de4a8f9b8f5e3243d4ea8, 21,320,400 bytes), on the 4-vCPU, 7.7 GiB perf VM. Combined base median 1,593.25 ms (84.5–3,056.1), branch 1,388.5 ms (337.0–5,316.4), delta -204.75 ms. In the second 10-trial window, base was 1,659.0 ms and branch 1,235.6 ms (delta -423.4 ms). Load averages were 1.35/1.40/1.34 to 1.71/1.61/1.45 in the first window and 1.94/1.57/1.45 to 1.98/1.65/1.52 in the extension. The browser-reported online event median was 2.8–2.9 ms. Full socket traces: network-recovery-prod492e-20.log.

Recovery comparison with handshake-preserving online hints: 20 alternating 10 s outage trials per variant against production server artifact 492e1505a (SHA-256 28212cdd96ef716d81502dc3991f8556460f75f8899de4a8f9b8f5e3243d4ea8, 21,320,400 bytes), on the 4-vCPU, 7.7 GiB perf VM. Combined base median 1,593.25 ms (84.5–3,056.1), branch 1,388.5 ms (337.0–5,316.4), delta -204.75 ms. In the second 10-trial window, base was 1,659.0 ms and branch 1,235.6 ms (delta -423.4 ms). Load averages were 1.35/1.40/1.34 to 1.71/1.61/1.45 in the first window and 1.94/1.57/1.45 to 1.98/1.65/1.52 in the extension. The browser-reported online event median was 2.8–2.9 ms. Full socket traces: [network-recovery-prod492e-20.log](https://git.kayg.org/attachments/ea257edc-d39a-4a35-bfc2-ab8d447763d4).
kayg commented 2026-10-16 22:13:41 +00:00
Author
Owner
Copy link

The final mock browser regression ran on the production web build: 2 passed, 0 failed. Across the 2.5 s redraw trace, the machine pill stayed live with no status/class mutations. Under repeated immediate server closes, 4 sockets and 4 attaches produced one live→connecting transition, zero live transitions, zero live frames during reconnect, and the persistent close reason stayed visible. Log: mock-final-browser.log.

The final mock browser regression ran on the production web build: 2 passed, 0 failed. Across the 2.5 s redraw trace, the machine pill stayed live with no status/class mutations. Under repeated immediate server closes, 4 sockets and 4 attaches produced one live→connecting transition, zero live transitions, zero live frames during reconnect, and the persistent close reason stayed visible. Log: [mock-final-browser.log](https://git.kayg.org/attachments/bf6c9b9e-321f-468d-9286-25c64c3d5ed9).
kayg commented 2026-10-16 22:17:07 +00:00
Author
Owner
Copy link

Real-server redraw evidence on server artifact 492e1505a and the final web build: the video run covered 10.38 s and 191,063 output bytes (90 OSC title changes, 2,250 cursor moves, 181 synchronized-output toggles). It recorded 600 frame samples, only the live green color, zero status mutations, zero client closes and zero server closes. The video, sampled frame, and log are attached above.

An earlier build-host run of the same harness saw one server-side transport close at 7.325 s (WebSocket code 1005, no close reason), with a connecting→live transition during the workload. The subsequent REAL_SERVER_LOG=1 repeat and final video run both completed without a close; logs contain only server startup, so the cause of that unrepeatable close remains unknown. The staged sign-in app still does not render here, so this workload is a synthetic sign-in-style redraw, not captured agy output.

Real-server redraw evidence on server artifact 492e1505a and the final web build: the video run covered 10.38 s and 191,063 output bytes (90 OSC title changes, 2,250 cursor moves, 181 synchronized-output toggles). It recorded 600 frame samples, only the live green color, zero status mutations, zero client closes and zero server closes. The video, sampled frame, and log are attached above. An earlier build-host run of the same harness saw one server-side transport close at 7.325 s (WebSocket code 1005, no close reason), with a connecting→live transition during the workload. The subsequent REAL_SERVER_LOG=1 repeat and final video run both completed without a close; logs contain only server startup, so the cause of that unrepeatable close remains unknown. The staged sign-in app still does not render here, so this workload is a synthetic sign-in-style redraw, not captured agy output.
kayg commented 2026-10-16 22:23:19 +00:00
Author
Owner
Copy link

Main-thread output cost on the 4-vCPU, 7.7 GiB perf VM, production server artifact 492e1505a (SHA-256 28212cdd96ef716d81502dc3991f8556460f75f8899de4a8f9b8f5e3243d4ea8), 10 runs per variant: base 2,075 output frames, median 8,016.083 ms/1,000 frames (7,347.330–12,551.627); branch 2,071 frames, median 8,081.567 ms/1,000 (7,557.507–11,805.214), delta +65.483 ms/1,000 (0.065 ms/frame). Load average was 1.57/1.14/1.29 before and 1.21/1.27/1.32 after. The changed code is only on retry/online-event paths; the output-frame path is unchanged. Per-run metrics and full trace: summary, raw log.

Main-thread output cost on the 4-vCPU, 7.7 GiB perf VM, production server artifact 492e1505a (SHA-256 28212cdd96ef716d81502dc3991f8556460f75f8899de4a8f9b8f5e3243d4ea8), 10 runs per variant: base 2,075 output frames, median 8,016.083 ms/1,000 frames (7,347.330–12,551.627); branch 2,071 frames, median 8,081.567 ms/1,000 (7,557.507–11,805.214), delta +65.483 ms/1,000 (0.065 ms/frame). Load average was 1.57/1.14/1.29 before and 1.21/1.27/1.32 after. The changed code is only on retry/online-event paths; the output-frame path is unchanged. Per-run metrics and full trace: [summary](https://git.kayg.org/attachments/fe1640c9-84b9-4e8f-af92-0207dfa1b102), [raw log](https://git.kayg.org/attachments/90fa2cdf-61c1-4556-99a6-fad42b4ab221).
kayg commented 2026-10-16 22:24:03 +00:00
Author
Owner
Copy link

Committed d9142366a: online/foreground hints reset retry state and preserve a handshake already in progress; unit regression was observed failing before the change. Focused TypeScript tests: 11 passed, 0 failed, 42 assertions; bun run check exit 0. Mock browser: 2 passed; close-storm trace held at one live→connecting transition, zero live flashes. Production-server outage comparison: 20 trials per variant, branch median 1,388.5 ms vs base 1,593.25 ms. Main-thread cost: 8,081.567 vs 8,016.083 ms/1,000 frames (+65.483 ms/1,000, within run spread).

Committed d9142366a: online/foreground hints reset retry state and preserve a handshake already in progress; unit regression was observed failing before the change. Focused TypeScript tests: 11 passed, 0 failed, 42 assertions; `bun run check` exit 0. Mock browser: 2 passed; close-storm trace held at one live→connecting transition, zero live flashes. Production-server outage comparison: 20 trials per variant, branch median 1,388.5 ms vs base 1,593.25 ms. Main-thread cost: 8,081.567 vs 8,016.083 ms/1,000 frames (+65.483 ms/1,000, within run spread).
kayg commented 2026-10-16 22:35:35 +00:00
Author
Owner
Copy link

Pre-rebase scripts/verify fast finding. Full gate log: verify-fast-final.log.

The only failed row was the AArch64 static-size gate: 19,517,536 bytes against the unchanged 19,513,440-byte cap (+4,096 bytes). x86_64 passed at its exact 21,320,400-byte cap. I left both caps unchanged. origin/dev now includes cb25dae3b (the mesh recovery merge, #229), so I am rebasing the seven pill-flicker commits onto current dev and will rerun the gate.

Pre-rebase `scripts/verify fast` finding. Full gate log: [verify-fast-final.log](https://git.kayg.org/attachments/1fdb530d-b021-4b40-b704-ae9369220152). The only failed row was the AArch64 static-size gate: 19,517,536 bytes against the unchanged 19,513,440-byte cap (+4,096 bytes). x86_64 passed at its exact 21,320,400-byte cap. I left both caps unchanged. `origin/dev` now includes `cb25dae3b` (the mesh recovery merge, #229), so I am rebasing the seven pill-flicker commits onto current dev and will rerun the gate.
kayg commented 2026-10-16 23:24:36 +00:00
Author
Owner
Copy link

Rebased the seven pill-flicker commits onto origin/dev at 3b2183eaa; this includes the mesh recovery merge cb25dae3b (#229). No Rust source files differ from that base. Final clean gate transcript: verify-fast-final-clean.log.

The gate still fails in inherited build checks. Fresh static artifacts measure x86_64 21,332,784 bytes versus cap 21,300,016 (+32,768) and AArch64 19,549,528 versus cap 19,500,376 (+49,152). The static metadata review also reports the Cargo registry path present in both artifacts. I left both caps unchanged. In cargo test, access_updates could not be executed because its test binary path was missing (No such file or directory); the separate agent_daemon rerun passed 1/1 in 33.94 s.

Final scripts/verify fast summary, verbatim:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | FAIL (101)   |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+
Rebased the seven pill-flicker commits onto `origin/dev` at `3b2183eaa`; this includes the mesh recovery merge `cb25dae3b` (#229). No Rust source files differ from that base. Final clean gate transcript: [verify-fast-final-clean.log](https://git.kayg.org/attachments/332fd7cf-627c-4591-aba9-f66394a1b22d). The gate still fails in inherited build checks. Fresh static artifacts measure x86_64 21,332,784 bytes versus cap 21,300,016 (+32,768) and AArch64 19,549,528 versus cap 19,500,376 (+49,152). The static metadata review also reports the Cargo registry path present in both artifacts. I left both caps unchanged. In `cargo test`, `access_updates` could not be executed because its test binary path was missing (`No such file or directory`); the separate `agent_daemon` rerun passed 1/1 in 33.94 s. Final `scripts/verify fast` summary, verbatim: ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | FAIL (101) | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ```
kayg commented 2026-10-16 23:25:55 +00:00
Author
Owner
Copy link

Finished and pushed pill-flicker at 18ee29012547f809243fd643c8e40a9419dd927b, rebased on origin/dev 3b2183eaa. No deploy. The final screen recording and frame are attached: 10-second production-build recording, final frame. This uses the synthetic sign-in-style workload because the staged agy run exited before rendering; the actual agy bytes remain unavailable.

The reconnect regression and browser checks pass; the final clean scripts/verify fast does not. Its full log is attached in the preceding comment. The pill-flicker commits add no Rust or Cargo inputs relative to origin/dev (13 changed files are tracer TypeScript/tests and taste documentation), so the Rust binary source delta is 0 B. The fresh artifacts still exceed current caps: x86_64 21,332,784 B (+32,768), AArch64 19,549,528 B (+49,152). I did not change the caps.

Final scripts/verify fast output, verbatim:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | FAIL (101)   |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+
Finished and pushed `pill-flicker` at `18ee29012547f809243fd643c8e40a9419dd927b`, rebased on `origin/dev` `3b2183eaa`. No deploy. The final screen recording and frame are attached: [10-second production-build recording](https://git.kayg.org/attachments/b54eb335-ab2b-41b3-90bb-7e3767842306), [final frame](https://git.kayg.org/attachments/2b959a3c-7e4e-4b6a-8d28-a898bcbe20ca). This uses the synthetic sign-in-style workload because the staged agy run exited before rendering; the actual agy bytes remain unavailable. The reconnect regression and browser checks pass; the final clean `scripts/verify fast` does not. Its full log is attached in the preceding comment. The pill-flicker commits add no Rust or Cargo inputs relative to `origin/dev` (13 changed files are tracer TypeScript/tests and taste documentation), so the Rust binary source delta is 0 B. The fresh artifacts still exceed current caps: x86_64 21,332,784 B (+32,768), AArch64 19,549,528 B (+49,152). I did not change the caps. Final `scripts/verify fast` output, verbatim: ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | FAIL (101) | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ```
kayg commented 2026-10-16 23:29:23 +00:00
Author
Owner
Copy link

Start: branch review-pill-flicker-r2 at target 18ee29012547f809243fd643c8e40a9419dd927b, based on origin/dev 3b2183eaa. Read all 109 #227 comments; the issue has no parent link. The target already contains the corrected outage and toast regressions from the prior review, so their cherry-picks were empty/conflicting duplicates and were skipped. Starting the web-only delta review and perf-VM recovery comparison.

Start: branch `review-pill-flicker-r2` at target `18ee29012547f809243fd643c8e40a9419dd927b`, based on `origin/dev` `3b2183eaa`. Read all 109 #227 comments; the issue has no parent link. The target already contains the corrected outage and toast regressions from the prior review, so their cherry-picks were empty/conflicting duplicates and were skipped. Starting the web-only delta review and perf-VM recovery comparison.
kayg commented 2026-10-16 23:30:30 +00:00
Author
Owner
Copy link

Delta check at 18ee29012547f809243fd643c8e40a9419dd927b: the prior outage regression (9.5 s recovery-window case) and toast replacement regression both pass in the target: 11 tests, 42 assertions, 0 failures across connection-restart.test.ts and canvas/status.test.ts. The target already carries these corrected tests; no review change was needed for these cases.

Delta check at `18ee29012547f809243fd643c8e40a9419dd927b`: the prior outage regression (9.5 s recovery-window case) and toast replacement regression both pass in the target: 11 tests, 42 assertions, 0 failures across `connection-restart.test.ts` and `canvas/status.test.ts`. The target already carries these corrected tests; no review change was needed for these cases.
kayg commented 2026-10-16 23:33:39 +00:00
Author
Owner
Copy link

Production web builds completed for the comparison inputs: base 3b2183eaac8a9d8cc52a922dfbbbaecf895fd854 and target 18ee29012547f809243fd643c8e40a9419dd927b. Both bun run build commands passed through the host build-slot wrapper. The built dist directories are staged under /home/kayg/.cache/codex-scratch/pill-flicker-review/build/ pending transfer to the perf VM.

Production web builds completed for the comparison inputs: base `3b2183eaac8a9d8cc52a922dfbbbaecf895fd854` and target `18ee29012547f809243fd643c8e40a9419dd927b`. Both `bun run build` commands passed through the host build-slot wrapper. The built dist directories are staged under `/home/kayg/.cache/codex-scratch/pill-flicker-review/build/` pending transfer to the perf VM.
kayg commented 2026-10-16 23:34:08 +00:00
Author
Owner
Copy link

Build receipt: production dist sizes are base 3,362,136 bytes and target 3,365,171 bytes (+3,035 bytes). Both builds passed; source revisions remain base 3b2183eaa and target 18ee29012.

Build receipt: production dist sizes are base `3,362,136` bytes and target `3,365,171` bytes (+3,035 bytes). Both builds passed; source revisions remain base `3b2183eaa` and target `18ee29012`.
kayg commented 2026-10-16 23:40:52 +00:00
Author
Owner
Copy link

Perf-VM base run completed on 3b2183eaac8a9d8cc52a922dfbbbaecf895fd854: ten 10-second offline trials against server artifact 492e1505a (SHA-256 28212cdd96ef716d81502dc3991f8556460f75f8899de4a8f9b8f5e3243d4ea8). Return-to-live median 1,076.350 ms (628.200–2,658.600); 12–30 WebSocket error events per outage (median 14.5). Perf VM: 4 vCPU, 7.7 GiB; load average 0.54/1.08/1.10 before and 0.95/1.08/1.09 after. Raw receipt is saved under the review scratch evidence directory; head run follows.

Perf-VM base run completed on `3b2183eaac8a9d8cc52a922dfbbbaecf895fd854`: ten 10-second offline trials against server artifact `492e1505a` (SHA-256 `28212cdd96ef716d81502dc3991f8556460f75f8899de4a8f9b8f5e3243d4ea8`). Return-to-live median 1,076.350 ms (628.200–2,658.600); 12–30 WebSocket error events per outage (median 14.5). Perf VM: 4 vCPU, 7.7 GiB; load average 0.54/1.08/1.10 before and 0.95/1.08/1.09 after. Raw receipt is saved under the review scratch evidence directory; head run follows.
kayg commented 2026-10-16 23:43:09 +00:00
Author
Owner
Copy link

Additional targeted checks at target 18ee29012: close-reason and connection-liveness tests passed (2 tests, 15 assertions, 0 failures). The outage and toast regressions remain green (11 tests, 42 assertions).

Additional targeted checks at target `18ee29012`: close-reason and connection-liveness tests passed (2 tests, 15 assertions, 0 failures). The outage and toast regressions remain green (11 tests, 42 assertions).
kayg commented 2026-10-16 23:51:37 +00:00
Author
Owner
Copy link

Head real-browser network-cut measurement complete on the perf VM. Ten 10-second offline trials per build, with the same server artifact: base return-to-live median 1,076.35 ms (628.2–2,658.6); target 1,002.7 ms (541.8–3,175.6). Offline WebSocket error events were base median 14.5 (range 12–30), target median 14 (12–30). VM: 4 vCPU, 7.7 GiB; load average base before/after 0.54/1.08/1.10 and 0.95/1.08/1.09; target 1.59/1.59/1.30 and 1.05/1.38/1.26. Raw logs: /root/infraternal-bench/review-pill-flicker/evidence/network-base.log and network-head.log.

Head real-browser network-cut measurement complete on the perf VM. Ten 10-second offline trials per build, with the same server artifact: base return-to-live median 1,076.35 ms (628.2–2,658.6); target 1,002.7 ms (541.8–3,175.6). Offline WebSocket error events were base median 14.5 (range 12–30), target median 14 (12–30). VM: 4 vCPU, 7.7 GiB; load average base before/after 0.54/1.08/1.10 and 0.95/1.08/1.09; target 1.59/1.59/1.30 and 1.05/1.38/1.26. Raw logs: `/root/infraternal-bench/review-pill-flicker/evidence/network-base.log` and `network-head.log`.
kayg commented 2026-10-16 23:51:42 +00:00
Author
Owner
Copy link

R2-1 (P2): a failed first recovery handshake exits the one quick retry fails on target 18ee29012: after an attached route disconnects, the first fast retry fails before Welcome, and the next delay remains 5 ms (expected at least 50 ms). disconnect resets retryAttempt for unwelcomed sockets while retaining fastRecovery when no server Hello arrived, so immediate handshake failures can keep retrying at the fast interval. Test-only commit: a0068ce77; raw output: /home/kayg/.cache/codex-scratch/pill-flicker-review/retry-edge-test.log. The 10-trial offline browser run did not show a higher error count (base median 14.5, target 14); this finding is scoped to immediate handshake failures.

R2-1 (P2): `a failed first recovery handshake exits the one quick retry` fails on target `18ee29012`: after an attached route disconnects, the first fast retry fails before `Welcome`, and the next delay remains 5 ms (expected at least 50 ms). `disconnect` resets retryAttempt for unwelcomed sockets while retaining fastRecovery when no server Hello arrived, so immediate handshake failures can keep retrying at the fast interval. Test-only commit: `a0068ce77`; raw output: `/home/kayg/.cache/codex-scratch/pill-flicker-review/retry-edge-test.log`. The 10-trial offline browser run did not show a higher error count (base median 14.5, target 14); this finding is scoped to immediate handshake failures.
kayg commented 2026-10-16 23:52:44 +00:00
Author
Owner
Copy link

Finished delta review. Job target: 18ee29012; review branch review-pill-flicker-r2 ends at a0068ce77 with one failing-test-only commit. No implementation files were changed, and nothing was pushed.

Earlier regressions pass: the corrected prolonged-outage retry test and shared-toast ownership test passed in the focused run (11 tests, 42 assertions); the close-reason and liveness tests passed (2 tests, 15 assertions). bun run check passed. Fresh production builds passed: base 3,362,136 bytes, target 3,365,171 bytes (+3,035 bytes).

The 10-trial, 10-second network-cut comparison recorded base/target median return-to-live of 1,076.35/1,002.7 ms and median offline WebSocket error events of 14.5/14. Finding R2-1 is P2: the first failed unwelcomed recovery handshake retains the fast retry interval (5 ms instead of at least 50 ms). Its failing test is committed as a0068ce77 and is recorded in findings.md.

Verdict: MERGE (0 P0, 0 P1, 1 P2). The job report records this scripts/verify fast summary for 18ee29012 (Rust rows excluded from this web-only review):

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | FAIL (101)   |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+
Finished delta review. Job target: `18ee29012`; review branch `review-pill-flicker-r2` ends at `a0068ce77` with one failing-test-only commit. No implementation files were changed, and nothing was pushed. Earlier regressions pass: the corrected prolonged-outage retry test and shared-toast ownership test passed in the focused run (11 tests, 42 assertions); the close-reason and liveness tests passed (2 tests, 15 assertions). `bun run check` passed. Fresh production builds passed: base 3,362,136 bytes, target 3,365,171 bytes (+3,035 bytes). The 10-trial, 10-second network-cut comparison recorded base/target median return-to-live of 1,076.35/1,002.7 ms and median offline WebSocket error events of 14.5/14. Finding R2-1 is P2: the first failed unwelcomed recovery handshake retains the fast retry interval (5 ms instead of at least 50 ms). Its failing test is committed as `a0068ce77` and is recorded in `findings.md`. Verdict: **MERGE** (0 P0, 0 P1, 1 P2). The job report records this `scripts/verify fast` summary for `18ee29012` (Rust rows excluded from this web-only review): ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | FAIL (101) | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ```
kayg commented 2026-10-16 23:55:01 +00:00
Author
Owner
Copy link

Start: branch pill-flicker at 18ee29012547f809243fd643c8e40a9419dd927b, based on origin/dev 3b2183eaac8a9d8cc52a922dfbbbaecf895fd854. Read all 118 #227 comments; issue metadata has no parent link. Continuing only R2-1: cherry-pick the unwelcomed-handshake regression and confirm it fails before changing retry scheduling.

Start: branch `pill-flicker` at `18ee29012547f809243fd643c8e40a9419dd927b`, based on `origin/dev` `3b2183eaac8a9d8cc52a922dfbbbaecf895fd854`. Read all 118 #227 comments; issue metadata has no parent link. Continuing only R2-1: cherry-pick the unwelcomed-handshake regression and confirm it fails before changing retry scheduling.
kayg commented 2026-10-16 23:55:14 +00:00
Author
Owner
Copy link

R2-1 reproduced at 459136e77: after an attached stream drops, the first reconnect fails before Welcome, and the next retry is still scheduled at 5 ms. bun test src/connection-restart.test.ts reports 6 pass, 1 fail; the regression expected >=50 ms and received 5 ms. I’m tracing the unwelcomed-socket reset and one-shot fast-retry state before the fix.

R2-1 reproduced at `459136e77`: after an attached stream drops, the first reconnect fails before `Welcome`, and the next retry is still scheduled at 5 ms. `bun test src/connection-restart.test.ts` reports 6 pass, 1 fail; the regression expected >=50 ms and received 5 ms. I’m tracing the unwelcomed-socket reset and one-shot fast-retry state before the fix.
kayg commented 2026-10-17 00:01:29 +00:00
Author
Owner
Copy link

R2-1 complete and pushed on pill-flicker at 291d431d2638c0cbc351e2f1a9299052a88947bb (test-first commit 459136e77; no deploy). An online recovery handshake that fails before Welcome now consumes the one quick retry and retains the retry count, so repeated failures grow from the 50 ms schedule. Fast probes remain while the browser reports offline; online/visibility hints still make an immediate attempt.

The regression failed before the fix with Expected: >= 50; Received: 5. Final focused connection tests:

7 pass
0 fail
37 expect() calls
Ran 7 tests across 1 file. [132.00ms]

bun run check passed. TypeScript gate results, verbatim:

2 pass
0 fail
12 expect() calls
Ran 2 tests across 1 file. [487.00ms]

213 pass
0 fail
11477 expect() calls
Ran 213 tests across 33 files. [1379.00ms]

37 pass
0 fail
34463 expect() calls
Ran 37 tests across 10 files. [3.14s]

Critical transfer: identity 963430 B, gzip 298140/307200 B, Brotli 248013/256000 B. Full TypeScript gate exited 0. As requested for this web-only continuation, Rust gate rows were not rerun.

Receipts: connection regression log, TypeScript gate log.

R2-1 complete and pushed on `pill-flicker` at `291d431d2638c0cbc351e2f1a9299052a88947bb` (test-first commit `459136e77`; no deploy). An online recovery handshake that fails before Welcome now consumes the one quick retry and retains the retry count, so repeated failures grow from the 50 ms schedule. Fast probes remain while the browser reports offline; online/visibility hints still make an immediate attempt. The regression failed before the fix with `Expected: >= 50; Received: 5`. Final focused connection tests: ```text 7 pass 0 fail 37 expect() calls Ran 7 tests across 1 file. [132.00ms] ``` `bun run check` passed. TypeScript gate results, verbatim: ```text 2 pass 0 fail 12 expect() calls Ran 2 tests across 1 file. [487.00ms] 213 pass 0 fail 11477 expect() calls Ran 213 tests across 33 files. [1379.00ms] 37 pass 0 fail 34463 expect() calls Ran 37 tests across 10 files. [3.14s] ``` Critical transfer: identity 963430 B, gzip 298140/307200 B, Brotli 248013/256000 B. Full TypeScript gate exited 0. As requested for this web-only continuation, Rust gate rows were not rerun. Receipts: [connection regression log](https://git.kayg.org/attachments/39fba04a-9027-4c73-b32e-84c9a5614137), [TypeScript gate log](https://git.kayg.org/attachments/5ff5a85b-174a-44a0-859d-2a5b5e0b55fd).
kayg commented 2026-10-17 00:02:46 +00:00
Author
Owner
Copy link

Live in production 2026-10-17 (web only, no restart) at c47a438be7, merged into dev. Server-side cause still unknown; close-reason logging job queued.

Live in production 2026-10-17 (web only, no restart) at c47a438be76e8cb1e5a5328bd190520d6039b33f, merged into dev. Server-side cause still unknown; close-reason logging job queued.
kayg commented 2026-10-17 04:44:25 +00:00
Author
Owner
Copy link

Start: branch stream-close-log, base b231301396227821a992692410cf55c50abe04ef (origin/dev). Read #227’s complete comment history; it has no parent issue link. The owner’s 2026-10-17 rootless-host case is in scope. Tracing server WebSocket/session and server↔infraternal-agent stream exits before implementation.

Start: branch `stream-close-log`, base `b231301396227821a992692410cf55c50abe04ef` (origin/dev). Read #227’s complete comment history; it has no parent issue link. The owner’s 2026-10-17 rootless-host case is in scope. Tracing server WebSocket/session and server↔infraternal-agent stream exits before implementation.
kayg commented 2026-10-17 04:47:01 +00:00
Author
Owner
Copy link

Finding: terminal close attribution is lost at three boundaries. crates/infraternal-server/src/websocket.rs:1509 logs only agent_stream and labels every attachment embedded, including remote hosts; websocket.rs:2004 discards browser WebSocket close code/reason; remote_link.rs:826-845 converts agent stream read failure to Close 1013 without logging host/session context. The attach path already reads remote_host_id and stores session cursors, so close records can include both IDs and existing counters without adding work to each output frame.

Finding: terminal close attribution is lost at three boundaries. `crates/infraternal-server/src/websocket.rs:1509` logs only `agent_stream` and labels every attachment embedded, including remote hosts; `websocket.rs:2004` discards browser WebSocket close code/reason; `remote_link.rs:826-845` converts agent stream read failure to Close 1013 without logging host/session context. The attach path already reads `remote_host_id` and stores session cursors, so close records can include both IDs and existing counters without adding work to each output frame.
kayg commented 2026-10-17 05:52:25 +00:00
Author
Owner
Copy link

Step 1 complete: 17c4c02d8 adds the content-free session close logger, 60 s summaries, bounded session state, and a 512-close storm test. Focused test: 1 passed, 0 failed. Server crate clippy: PASS.

Step 1 complete: 17c4c02d8 adds the content-free session close logger, 60 s summaries, bounded session state, and a 512-close storm test. Focused test: 1 passed, 0 failed. Server crate clippy: PASS.
kayg commented 2026-10-17 06:03:49 +00:00
Author
Owner
Copy link

Correction: step 1 commit body was amended for readable paragraphs. Current commit is 0e23cc5ea; it replaces 17c4c02d8. The logger test and clippy results are unchanged.

Correction: step 1 commit body was amended for readable paragraphs. Current commit is 0e23cc5ea; it replaces 17c4c02d8. The logger test and clippy results are unchanged.
kayg commented 2026-10-17 06:04:53 +00:00
Author
Owner
Copy link

Finding: the client close-reason table inherited object keys, so an unknown reason such as constructor could bypass the fallback. apps/tracer/src/close-reasons.ts now accepts own keys only; tests cover constructor and __proto__ and unknown text. Client checks: typecheck passed; 17 tests passed.

Finding: the client close-reason table inherited object keys, so an unknown reason such as `constructor` could bypass the fallback. `apps/tracer/src/close-reasons.ts` now accepts own keys only; tests cover `constructor` and `__proto__` and unknown text. Client checks: typecheck passed; 17 tests passed.
kayg commented 2026-10-17 06:05:04 +00:00
Author
Owner
Copy link

Step complete: 182a208c4 updates the existing terminal status/toast mapping with safe close explanations, preserves neutral detach/rotation text, and discards unknown close text. bun run check passed; 17 focused client tests passed (4 files, 71 assertions).

Step complete: 182a208c4 updates the existing terminal status/toast mapping with safe close explanations, preserves neutral detach/rotation text, and discards unknown close text. `bun run check` passed; 17 focused client tests passed (4 files, 71 assertions).
kayg commented 2026-10-17 06:22:47 +00:00
Author
Owner
Copy link

Production queue fix now tracked on #234, branch ping-fifo-fix, base 2a5302cc7, initial regression 54d24be15 from #172 RLR-13. The scheduled Ping bypasses native control FIFO readiness, and Full is propagated as connection failure. Audit also found server Pong admission, host control replies and terminal-command Ping admission need coverage. Fix and natural terminal-flood comparison are in progress; no production deployment.

Production queue fix now tracked on #234, branch ping-fifo-fix, base 2a5302cc7, initial regression 54d24be15 from #172 RLR-13. The scheduled Ping bypasses native control FIFO readiness, and Full is propagated as connection failure. Audit also found server Pong admission, host control replies and terminal-command Ping admission need coverage. Fix and natural terminal-flood comparison are in progress; no production deployment.
kayg commented 2026-10-17 06:25:47 +00:00
Author
Owner
Copy link

Rootless add-host finding: an attached terminal with no live mesh peer could return a terminal-stream refusal while leaving the control socket usable, so the prior logs did not identify why the window stayed reconnecting. The rootless refusal now logs the server as closer, reason, host/session IDs and stream counters, and the test confirms the control socket still answers Ping/Pong. Focused test rootless_remote_host_without_a_live_peer_logs_the_terminal_refusal passed; implementation: crates/infraternal-server/src/websocket.rs, coverage: crates/infraternal-server/src/auth/socket_tests/close_logging.rs.

Rootless add-host finding: an attached terminal with no live mesh peer could return a terminal-stream refusal while leaving the control socket usable, so the prior logs did not identify why the window stayed reconnecting. The rootless refusal now logs the server as closer, reason, host/session IDs and stream counters, and the test confirms the control socket still answers Ping/Pong. Focused test `rootless_remote_host_without_a_live_peer_logs_the_terminal_refusal` passed; implementation: `crates/infraternal-server/src/websocket.rs`, coverage: `crates/infraternal-server/src/auth/socket_tests/close_logging.rs`.
kayg commented 2026-10-17 06:30:24 +00:00
Author
Owner
Copy link

Focused server tests are queued in the required shared build-slot wrapper (15 minutes so far; four slots are occupied). The run is still waiting, not compiling; no owner action is needed. I’m continuing the read-only close-path review and will resume gates when a slot opens.

Focused server tests are queued in the required shared build-slot wrapper (15 minutes so far; four slots are occupied). The run is still waiting, not compiling; no owner action is needed. I’m continuing the read-only close-path review and will resume gates when a slot opens.
kayg commented 2026-10-17 06:34:15 +00:00
Author
Owner
Copy link

Client-map finding: server-generated reasons the terminal reader fell behind and is reconnecting and the terminal on the host stopped responding fell through to the generic unsafe-reason text because the client only mapped the older peer strings. Regression evidence: bun test src/close-reasons.test.ts failed the new assertion at apps/tracer/src/close-reasons.test.ts:18 (4 passed, 1 failed). The mixed-reason summary text also lacked a specific mapping.

Client-map finding: server-generated reasons `the terminal reader fell behind and is reconnecting` and `the terminal on the host stopped responding` fell through to the generic unsafe-reason text because the client only mapped the older peer strings. Regression evidence: `bun test src/close-reasons.test.ts` failed the new assertion at `apps/tracer/src/close-reasons.test.ts:18` (4 passed, 1 failed). The mixed-reason summary text also lacked a specific mapping.
kayg commented 2026-10-17 06:34:52 +00:00
Author
Owner
Copy link

Committed 0fdb45d96d6a4c50e2cf66eedb5666bce642788b: map the server-generated route and mixed-reason text in the shared client status surface. bun test src/close-reasons.test.ts src/canvas/status.test.ts src/connection-restart.test.ts src/connection-liveness.test.ts: 18 passed, 0 failed, 74 assertions. bun run check: exit 0.

Committed `0fdb45d96d6a4c50e2cf66eedb5666bce642788b`: map the server-generated route and mixed-reason text in the shared client status surface. `bun test src/close-reasons.test.ts src/canvas/status.test.ts src/connection-restart.test.ts src/connection-liveness.test.ts`: 18 passed, 0 failed, 74 assertions. `bun run check`: exit 0.
kayg commented 2026-10-17 06:37:34 +00:00
Author
Owner
Copy link

Logger unit tests passed on head 0fdb45d96d6a4c50e2cf66eedb5666bce642788b: 3 passed, 0 failed. The 512-close session storm emitted exactly 2 lines (one first-close record and one 60-second summary); the same suite verifies close counters and rejects untrusted peer text.

Logger unit tests passed on head `0fdb45d96d6a4c50e2cf66eedb5666bce642788b`: 3 passed, 0 failed. The 512-close session storm emitted exactly 2 lines (one first-close record and one 60-second summary); the same suite verifies close counters and rejects untrusted peer text.
kayg commented 2026-10-17 06:38:29 +00:00
Author
Owner
Copy link

Transient test finding: review_full_input_writer_preserves_browser_control_progress missed its attach ACK in the 9-test focused module run (8 passed, 1 failed at crates/infraternal-server/src/auth/socket_tests/review_input_credit.rs:226). The same fully qualified test passed alone on immediate rerun (1 passed, 0 failed, 0.88 s). No code changed for this failure; I’ll recheck the module and note any matching final-gate baseline result if it recurs.

Transient test finding: `review_full_input_writer_preserves_browser_control_progress` missed its attach ACK in the 9-test focused module run (8 passed, 1 failed at `crates/infraternal-server/src/auth/socket_tests/review_input_credit.rs:226`). The same fully qualified test passed alone on immediate rerun (1 passed, 0 failed, 0.88 s). No code changed for this failure; I’ll recheck the module and note any matching final-gate baseline result if it recurs.
kayg commented 2026-10-17 06:39:25 +00:00
Author
Owner
Copy link

Committed 2f2e59f9a2b03c8daa7a7061a6e3639679fdc922: classify and log browser/agent terminal closes, including rootless attach refusal and retained-session resume refusal. Gates: cargo fmt --all -- --check and git diff --check passed; strict server clippy exited 0. Focused tests: logger 3/3 (512 closes -> 2 lines), close paths 4/4, input/credit module 9/9 on rerun, transport end classification 1/1, protocol/size/idle classification 1/1. The initial input/credit module attempt had one transient ACK miss; the exact case and full module passed on rerun, as recorded above. Rootless test asserts host UUID, session UUID, refusal reason and live Ping/Pong.

Committed `2f2e59f9a2b03c8daa7a7061a6e3639679fdc922`: classify and log browser/agent terminal closes, including rootless attach refusal and retained-session resume refusal. Gates: `cargo fmt --all -- --check` and `git diff --check` passed; strict server clippy exited 0. Focused tests: logger 3/3 (512 closes -> 2 lines), close paths 4/4, input/credit module 9/9 on rerun, transport end classification 1/1, protocol/size/idle classification 1/1. The initial input/credit module attempt had one transient ACK miss; the exact case and full module passed on rerun, as recorded above. Rootless test asserts host UUID, session UUID, refusal reason and live Ping/Pong.
kayg commented 2026-10-17 06:54:21 +00:00
Author
Owner
Copy link

/home/kayg/.local/state/codex-jobs/infraternal/orchestrator/build-slot scripts/verify iterate exited 1 with one test failure: daemon::gone_tests::a_vanished_session_gets_a_final_close_and_the_channel_stays_up at crates/infraternal-agent/src/daemon_gone_tests.rs:111. The fixture appended 25 characters and a 47-character rendezvous suffix to the 38-character TMPDIR alias, producing a 110-character Unix socket path (over SUN_LEN). This is a test-root path issue, not a close-log assertion. I’ll rerun with a unique 35-character alias (107-character final path) and use that root for the final gate.

`/home/kayg/.local/state/codex-jobs/infraternal/orchestrator/build-slot scripts/verify iterate` exited 1 with one test failure: `daemon::gone_tests::a_vanished_session_gets_a_final_close_and_the_channel_stays_up` at `crates/infraternal-agent/src/daemon_gone_tests.rs:111`. The fixture appended 25 characters and a 47-character rendezvous suffix to the 38-character TMPDIR alias, producing a 110-character Unix socket path (over SUN_LEN). This is a test-root path issue, not a close-log assertion. I’ll rerun with a unique 35-character alias (107-character final path) and use that root for the final gate.
kayg commented 2026-10-17 06:57:00 +00:00
Author
Owner
Copy link

Correction: the 35-character alias calculation omitted the separator before g<id>; its actual rendezvous path was 108 characters. The isolated agent test passed with a 33-character TMPDIR root, giving a 106-character socket path: 1 passed, 0 failed, 0.03 s. I’ll use that direct short scratch directory for the full iterate rerun.

Correction: the 35-character alias calculation omitted the separator before `g<id>`; its actual rendezvous path was 108 characters. The isolated agent test passed with a 33-character TMPDIR root, giving a 106-character socket path: 1 passed, 0 failed, 0.03 s. I’ll use that direct short scratch directory for the full iterate rerun.
kayg commented 2026-10-17 07:07:18 +00:00
Author
Owner
Copy link

Corrected-root scripts/verify iterate reached the Rust auth/socket binary: 476 passed, 2 failed, 48 ignored. Failures were only the known auth/review3 group: review3_revoked_resume_must_not_welcome (crates/infraternal-server/src/auth/review3_tests.rs:794) and expired_handshake_releases_user_permits (crates/infraternal-server/src/auth/tests.rs:1763). Both asserted close/no-Welcome outcomes and neither is a close-log test. The required dev-baseline comparison will be run if either recurs in final scripts/verify fast.

Corrected-root `scripts/verify iterate` reached the Rust auth/socket binary: 476 passed, 2 failed, 48 ignored. Failures were only the known auth/review3 group: `review3_revoked_resume_must_not_welcome` (`crates/infraternal-server/src/auth/review3_tests.rs:794`) and `expired_handshake_releases_user_permits` (`crates/infraternal-server/src/auth/tests.rs:1763`). Both asserted close/no-Welcome outcomes and neither is a close-log test. The required dev-baseline comparison will be run if either recurs in final `scripts/verify fast`.
kayg commented 2026-10-17 13:04:42 +00:00
Author
Owner
Copy link

Root-cause finding from #234 (ping-fifo-fix, head c179cdcff), measured on the perf VM:

Under a 10-minute saturating terminal flood plus a paste, at 150 ms RTT with 1 % loss, origin/dev closes the host primary 12 times. At LAN it closes it 0 times. Each close comes from the server's liveness deadline expiring, not from a dead peer and not from the full-queue Ping that RLR-13 found. One lost packet on the ordered QUIC control stream holds back every later Pong for about 2 round trips (about 300–450 ms at 150 ms), and the deadline is about 250 ms (srtt + 4·rttvar + 50 ms, floor 250 ms). That fits "The connection to the server ended." and dot flicker on the India↔Germany link.

150 ms + 1 % loss, 10 min primary closes echo p99
origin/dev 12 438 ms
ping-fifo-fix (queue fixes) 12 666 ms
experiment: deadline 2·srtt + 4·rttvar + 75 ms 1 328 ms

The queue fixes on ping-fifo-fix are real P1 gaps with regressions, but under load they do not change the close count. The deadline is an owner decision (D265, about 1-RTT detection). Options A (one-recovery deadline), B (doubt then retire) and C (probes in QUIC datagrams) are on #234.

Root-cause finding from #234 (ping-fifo-fix, head c179cdcff), measured on the perf VM: Under a 10-minute saturating terminal flood plus a paste, at 150 ms RTT with 1 % loss, origin/dev closes the host primary **12 times**. At LAN it closes it 0 times. Each close comes from the server's **liveness deadline expiring**, not from a dead peer and not from the full-queue Ping that RLR-13 found. One lost packet on the ordered QUIC control stream holds back every later Pong for about 2 round trips (about 300–450 ms at 150 ms), and the deadline is about 250 ms (srtt + 4·rttvar + 50 ms, floor 250 ms). That fits "The connection to the server ended." and dot flicker on the India↔Germany link. | 150 ms + 1 % loss, 10 min | primary closes | echo p99 | |---|---|---| | origin/dev | 12 | 438 ms | | ping-fifo-fix (queue fixes) | 12 | 666 ms | | experiment: deadline 2·srtt + 4·rttvar + 75 ms | 1 | 328 ms | The queue fixes on ping-fifo-fix are real P1 gaps with regressions, but under load they do not change the close count. The deadline is an owner decision (D265, about 1-RTT detection). Options A (one-recovery deadline), B (doubt then retire) and C (probes in QUIC datagrams) are on #234.
kayg commented 2026-10-17 13:28:03 +00:00
Author
Owner
Copy link

Finish: head d1c21ad312 on branch stream-close-log (pushed). Continued from 2f2e59f9a.

Added since 2f2e59f9a:

  • 3f982adeb server tests accept the named application Close frame before the native close (fixes review3_revoked_resume_must_not_welcome, expired_handshake_releases_user_permits).
  • 6ee9205bc agent: one rate-limited line per server-link close or refusal (closed-by, link kind, path reason, lifetime); first line per 60 s window, repeats counted.
  • 081c3f8c7 tracer: one rate-limited console.warn per socket close naming code, mapped reason, reconnect attempt and whether the connection was ever established (failed reconnect).
  • server: sweep ended close windows once a second (perf below), one tracing call site, close-test hang bound 10 s, failed-resume test sends Hello+Resume in one message.

Overhead (perf VM 10.69.69.208, release build, flock, load 3): suppressed repeat close (the storm case): 149-162 ns/close. First close of a new session with a full map: 71 us/close before the sweep fix (8,192-window scan per close), 5.4 us after (formats and writes one JSON line); one run 13.8 us under other load. Forwarded frames never call the logger (close_logs.record only at close sites). 512-close storm test: 2 lines. 8,692 distinct sessions: map stays <= 8,192, every first close logged.

Size (D243): static x86_64 21,337,104 B vs dev 7392b925b built here 21,300,016 B = +37,088 B (aarch64 +33,104). OVER the 16 KiB goal and over the exact cap, so the static-size row fails. cargo bloat by function: stream_close ~14.2 KB (logger worker task, map, emit), websocket close handling ~7.8 KB, websocket_transport ~2.8 KB, rest ~11 KB strings/tracing/sqlx/agent. Needs an owner decision: accept and ratchet the cap, or cut (dropping the timer task for lazy summaries and the HashMap for a small Vec would save an estimated 5-7 KB; not enough alone).

Gate (scripts/verify fast, rerun on head):
+---------------------------+--------------+
| Rust gate | Result |
+---------------------------+--------------+
| fmt | PASS |
| VT native fmt | PASS |
| CLI plugin compile-out | PASS |
| clippy | PASS |
| test | PASS |
| cargo-deny | PASS |
| static builds | FAIL (1) |
+---------------------------+--------------+
static builds failure is only the size cap above. Earlier runs on this branch had load-only failures (browser_input_overload_retires_only_the_terminal_stream, saturated_native_control_queue..., close-log 2 s wait, failed-resume test); each passed alone and the last full run passed.

Finish: head d1c21ad312acdfae6535aed6ee61c2bdd4ed4810 on branch stream-close-log (pushed). Continued from 2f2e59f9a. Added since 2f2e59f9a: - 3f982adeb server tests accept the named application Close frame before the native close (fixes review3_revoked_resume_must_not_welcome, expired_handshake_releases_user_permits). - 6ee9205bc agent: one rate-limited line per server-link close or refusal (closed-by, link kind, path reason, lifetime); first line per 60 s window, repeats counted. - 081c3f8c7 tracer: one rate-limited console.warn per socket close naming code, mapped reason, reconnect attempt and whether the connection was ever established (failed reconnect). - server: sweep ended close windows once a second (perf below), one tracing call site, close-test hang bound 10 s, failed-resume test sends Hello+Resume in one message. Overhead (perf VM 10.69.69.208, release build, flock, load 3): suppressed repeat close (the storm case): 149-162 ns/close. First close of a new session with a full map: 71 us/close before the sweep fix (8,192-window scan per close), 5.4 us after (formats and writes one JSON line); one run 13.8 us under other load. Forwarded frames never call the logger (close_logs.record only at close sites). 512-close storm test: 2 lines. 8,692 distinct sessions: map stays <= 8,192, every first close logged. Size (D243): static x86_64 21,337,104 B vs dev 7392b925b built here 21,300,016 B = +37,088 B (aarch64 +33,104). OVER the 16 KiB goal and over the exact cap, so the static-size row fails. cargo bloat by function: stream_close ~14.2 KB (logger worker task, map, emit), websocket close handling ~7.8 KB, websocket_transport ~2.8 KB, rest ~11 KB strings/tracing/sqlx/agent. Needs an owner decision: accept and ratchet the cap, or cut (dropping the timer task for lazy summaries and the HashMap for a small Vec would save an estimated 5-7 KB; not enough alone). Gate (scripts/verify fast, rerun on head): +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ static builds failure is only the size cap above. Earlier runs on this branch had load-only failures (browser_input_overload_retires_only_the_terminal_stream, saturated_native_control_queue..., close-log 2 s wait, failed-resume test); each passed alone and the last full run passed.
kayg commented 2026-10-17 16:44:46 +00:00
Author
Owner
Copy link

Break-it review (Claude Opus) of the close-logging branch started.

Branch: review-stream-close-log, base origin/stream-close-log d66f8041f.
Focus: secrecy of logged/close text, log injection, limiter bypass, memory under churn, summary worker lifetime, blocking on the close path, client reconnect behaviour with the new Close frame.

Break-it review (Claude Opus) of the close-logging branch started. Branch: review-stream-close-log, base origin/stream-close-log d66f8041f. Focus: secrecy of logged/close text, log injection, limiter bypass, memory under churn, summary worker lifetime, blocking on the close path, client reconnect behaviour with the new Close frame.
kayg commented 2026-10-17 17:30:47 +00:00
Author
Owner
Copy link

Break-it review: stream-close-log (#227) at d66f8041f

Reviewer: Claude Opus (independent). Review branch: review-stream-close-log.

Clean (checked, no finding)

  • Secrecy/log injection: every server field is a &'static str from a fixed table, a number, or a UUID.
    Browser WebSocket close reasons, browser stream Close reasons and infraternal agent Close reasons go
    through allow-lists (safe_websocket_reason / safe_agent_close_reason), unknown -> "other". No
    newline/ANSI/huge text can reach the log or the browser. No tokens, terminal content or usernames are
    logged. The agent lines print only enum Debug values and io::ErrorKind. The client prints only a number
    and a mapped key.
  • Close reasons to the browser are static; the client decides reconnect by code (CLOSE_SESSION_GONE),
    never by reason text, so no reason can stop reconnects. Old bundles handle the control Close the same way.
  • Summary worker lifetime: holds only a Weak, Inner::drop aborts it; no leak.

P1-1 Authenticated client can flood the server log, one line per frame (limiter bypass)

AttachTerm with a session id the account does not own: session_host_id -> Ok(None),
owned_writer fails, session_gone -> true -> close_logs.record(AGENT_STREAM_ENDED) + Close 4404,
socket stays up. The stream id can be reused (EndedStreams::absorbs removes it on AttachTerm), so a
single socket can loop forever. The limiter keys on the client-chosen session id, so every frame is a new
"first close" and writes a ~600-byte JSON line to stderr (sync writer, runtime.rs configure_logging) from
the socket's tokio worker. Before #227 this path logged nothing.
Also mislabels a never-existing session as who=server, reason agent_stream_ended, and lets a client write
another account's session id into the log (and pre-open its rate window, suppressing that session's real
first-close line for 60 s): the key has no user component.
Failing test: auth::socket_tests::review_close_log::attaches_to_unowned_session_ids_cannot_flood_the_close_log
(64 attach frames -> 64 lines, bound 8).

P1-2 A full map turns rate limiting off for every session

Once 8,192 live windows exist (P1-1 fills them in seconds), any session not already tracked is "logged but
not tracked" on every close: a real reconnect storm logs one line per close for up to 60 s. The comment
claims memory stays bounded (true) but the log volume is unbounded exactly when it matters.
Failing test: stream_close::review_tests::a_full_map_does_not_turn_off_rate_limiting_for_a_storming_session
(1000 closes of one session -> 1000 lines, bound 2).

P1-3 Client console line is skipped for every close the server names

The claim is "client console warn per close incl. failed reconnects". The server now sends a control-stream
Close before its native close on handshake refusal (1008), protocol/size/idle ends, and teardown (1011).
The client's control() handles Close with forceClose() -> disconnect(); the later onclose sees
this.socket !== socket and returns before closeNotes.note. So failed reconnects refused by the server,
and all server-named closes, print nothing; only native-only/network closes print.
Failing test: apps/tracer/src/review-close-log.test.ts "a reconnect the server refuses with a named Close
frame still prints its console line" (0 warnings, expected 1).

P1-4 The branch breaks an existing tracer test and reverts approved plain wording

apps/tracer/test/close-reasons.test.ts (from cd7b07262 "close reasons ... read plainly") passes on
origin/dev and fails on the branch: "terminal command queue full; ..." now maps to "Connection closed: terminal
input exceeded the host queue limit." (jargon the test forbids). The rewrite of KNOWN also brings back
"flow-control limit", "invalid protocol frame", "control reply" wording and changes the fallback sentence.
The fast gate does not run test/close-reasons.test.ts, src/close-reasons.test.ts or the connection tests,
so scripts/verify fast cannot catch it (that gate gap is itself worth fixing).
Evidence: bun test test/close-reasons.test.ts -> 0 pass / 1 fail on the branch; same file against
origin/dev's close-reasons.ts -> 1 pass.

P2-1 Summary worker rescans the whole map on every first close

record_at calls changed.notify_one() on each new window; run wakes and computes min over all
windows while holding the logger mutex: O(n) per first close (n up to 8,192), contending with record.
The shipped close_path_cost bench runs on a current-thread runtime, so the worker never runs inside the
timed loop; the 5.4 µs figure excludes this cost. Windows are inserted in time order, so a VecDeque of
(deadline, key) gives O(1). Hand bench added: stream_close::review_tests::first_close_cost_with_the_worker_running.
Measured (debug build, loaded build host at load ~35, NOT the perf VM, indicative only):
current-thread runtime 2,131 ns/first close; 4-worker runtime (worker actually runs) 53,607 ns/first close
with 4k-8k live windows: about 25x slower. Needs a perf-VM release number before merge.

P3

  • P3-1 CloseNotes summary says "N more closes in the previous 60 s" even when those closes were hours
    earlier. Failing test in review-close-log.test.ts. Same wording issue in the agent LinkNotes.
  • P3-2 Agent LinkNotes and client CloseNotes have no timer: suppressed closes are reported only if
    another close happens later; a burst followed by a stable link is never summarised (server has a timer).
  • P3-3 run_socket's Inbound::SocketClose | Inbound::Failure arm is dead: the transport read task
    consumes both and only retires the context, recv never yields them. It duplicates the logging code
    (binary size) for nothing.
  • P3-4 Failed-handshake path now runs an untimed SQL query (writer rows for the claimed connection id)
    before the native close; a busy SQLite delays closing a refused socket.
  • P3-5 terminal.reconnecting now shows "Connection closed: ..." which status.ts treats as a connection
    failure toast, for a reconnect that recovers by itself; agent reasons "session ended"/"detached" with a
    non-gone code now show "The terminal session ended." while the client reattaches.

Branch and gate

Head: review-stream-close-log 1025ba97d (pushed), on top of stream-close-log d66f8041f. Only tests were added; no fixes.

scripts/verify fast (verbatim excerpts; it did not reach its final table because the build host disk ran out):

==> Repository artifact size gate
repository artifact gate: PASS (27 changed path(s), merge base b23130139622)
...
failures:
    auth::tests::socket_tests::review_close_log::attaches_to_unowned_session_ids_cannot_flood_the_close_log
    stream_close::review_tests::a_full_map_does_not_turn_off_rate_limiting_for_a_storming_session

test result: FAILED. 479 passed; 2 failed; 50 ignored; 0 measured; 0 filtered out; finished in 409.73s
...
rustc-LLVM ERROR: IO failure on output stream: No space left on device

The only Rust failures are the two intended review tests. The release step failed with ENOSPC (host / was at 97%); I deleted my 22 GB target afterwards. Tracer (bun test src/close-reasons src/connection src/terminal src/review-close): 63 pass, 2 fail (both intended review tests).

Verdict: HOLD, 4 P1, 1 P2, 5 P3.

# Break-it review: stream-close-log (#227) at d66f8041f Reviewer: Claude Opus (independent). Review branch: review-stream-close-log. ## Clean (checked, no finding) - Secrecy/log injection: every server field is a `&'static str` from a fixed table, a number, or a UUID. Browser WebSocket close reasons, browser stream Close reasons and infraternal agent Close reasons go through allow-lists (`safe_websocket_reason` / `safe_agent_close_reason`), unknown -> "other". No newline/ANSI/huge text can reach the log or the browser. No tokens, terminal content or usernames are logged. The agent lines print only enum Debug values and `io::ErrorKind`. The client prints only a number and a mapped key. - Close reasons to the browser are static; the client decides reconnect by code (CLOSE_SESSION_GONE), never by reason text, so no reason can stop reconnects. Old bundles handle the control Close the same way. - Summary worker lifetime: holds only a Weak, `Inner::drop` aborts it; no leak. ## P1-1 Authenticated client can flood the server log, one line per frame (limiter bypass) `AttachTerm` with a session id the account does not own: `session_host_id` -> Ok(None), `owned_writer` fails, `session_gone` -> true -> `close_logs.record(AGENT_STREAM_ENDED)` + Close 4404, socket stays up. The stream id can be reused (`EndedStreams::absorbs` removes it on AttachTerm), so a single socket can loop forever. The limiter keys on the client-chosen session id, so every frame is a new "first close" and writes a ~600-byte JSON line to stderr (sync writer, `runtime.rs` configure_logging) from the socket's tokio worker. Before #227 this path logged nothing. Also mislabels a never-existing session as who=server, reason agent_stream_ended, and lets a client write another account's session id into the log (and pre-open its rate window, suppressing that session's real first-close line for 60 s): the key has no user component. Failing test: `auth::socket_tests::review_close_log::attaches_to_unowned_session_ids_cannot_flood_the_close_log` (64 attach frames -> 64 lines, bound 8). ## P1-2 A full map turns rate limiting off for every session Once 8,192 live windows exist (P1-1 fills them in seconds), any session not already tracked is "logged but not tracked" on every close: a real reconnect storm logs one line per close for up to 60 s. The comment claims memory stays bounded (true) but the log volume is unbounded exactly when it matters. Failing test: `stream_close::review_tests::a_full_map_does_not_turn_off_rate_limiting_for_a_storming_session` (1000 closes of one session -> 1000 lines, bound 2). ## P1-3 Client console line is skipped for every close the server names The claim is "client console warn per close incl. failed reconnects". The server now sends a control-stream Close before its native close on handshake refusal (1008), protocol/size/idle ends, and teardown (1011). The client's `control()` handles Close with `forceClose()` -> `disconnect()`; the later `onclose` sees `this.socket !== socket` and returns before `closeNotes.note`. So failed reconnects refused by the server, and all server-named closes, print nothing; only native-only/network closes print. Failing test: `apps/tracer/src/review-close-log.test.ts` "a reconnect the server refuses with a named Close frame still prints its console line" (0 warnings, expected 1). ## P1-4 The branch breaks an existing tracer test and reverts approved plain wording `apps/tracer/test/close-reasons.test.ts` (from cd7b07262 "close reasons ... read plainly") passes on origin/dev and fails on the branch: "terminal command queue full; ..." now maps to "Connection closed: terminal input exceeded the host queue limit." (jargon the test forbids). The rewrite of `KNOWN` also brings back "flow-control limit", "invalid protocol frame", "control reply" wording and changes the fallback sentence. The fast gate does not run `test/close-reasons.test.ts`, `src/close-reasons.test.ts` or the connection tests, so `scripts/verify fast` cannot catch it (that gate gap is itself worth fixing). Evidence: `bun test test/close-reasons.test.ts` -> 0 pass / 1 fail on the branch; same file against origin/dev's close-reasons.ts -> 1 pass. ## P2-1 Summary worker rescans the whole map on every first close `record_at` calls `changed.notify_one()` on each new window; `run` wakes and computes `min` over all windows while holding the logger mutex: O(n) per first close (n up to 8,192), contending with `record`. The shipped `close_path_cost` bench runs on a current-thread runtime, so the worker never runs inside the timed loop; the 5.4 µs figure excludes this cost. Windows are inserted in time order, so a VecDeque of (deadline, key) gives O(1). Hand bench added: `stream_close::review_tests::first_close_cost_with_the_worker_running`. Measured (debug build, loaded build host at load ~35, NOT the perf VM, indicative only): current-thread runtime 2,131 ns/first close; 4-worker runtime (worker actually runs) 53,607 ns/first close with 4k-8k live windows: about 25x slower. Needs a perf-VM release number before merge. ## P3 - P3-1 `CloseNotes` summary says "N more closes in the previous 60 s" even when those closes were hours earlier. Failing test in `review-close-log.test.ts`. Same wording issue in the agent `LinkNotes`. - P3-2 Agent `LinkNotes` and client `CloseNotes` have no timer: suppressed closes are reported only if another close happens later; a burst followed by a stable link is never summarised (server has a timer). - P3-3 `run_socket`'s `Inbound::SocketClose | Inbound::Failure` arm is dead: the transport read task consumes both and only retires the context, `recv` never yields them. It duplicates the logging code (binary size) for nothing. - P3-4 Failed-handshake path now runs an untimed SQL query (writer rows for the claimed connection id) before the native close; a busy SQLite delays closing a refused socket. - P3-5 `terminal.reconnecting` now shows "Connection closed: ..." which `status.ts` treats as a connection failure toast, for a reconnect that recovers by itself; agent reasons "session ended"/"detached" with a non-gone code now show "The terminal session ended." while the client reattaches. ## Branch and gate Head: review-stream-close-log 1025ba97d (pushed), on top of stream-close-log d66f8041f. Only tests were added; no fixes. `scripts/verify fast` (verbatim excerpts; it did not reach its final table because the build host disk ran out): ``` ==> Repository artifact size gate repository artifact gate: PASS (27 changed path(s), merge base b23130139622) ... failures: auth::tests::socket_tests::review_close_log::attaches_to_unowned_session_ids_cannot_flood_the_close_log stream_close::review_tests::a_full_map_does_not_turn_off_rate_limiting_for_a_storming_session test result: FAILED. 479 passed; 2 failed; 50 ignored; 0 measured; 0 filtered out; finished in 409.73s ... rustc-LLVM ERROR: IO failure on output stream: No space left on device ``` The only Rust failures are the two intended review tests. The release step failed with ENOSPC (host / was at 97%); I deleted my 22 GB target afterwards. Tracer (`bun test src/close-reasons src/connection src/terminal src/review-close`): 63 pass, 2 fail (both intended review tests). **Verdict: HOLD**, 4 P1, 1 P2, 5 P3.
kayg commented 2026-10-17 17:33:16 +00:00
Author
Owner
Copy link

Start: fixing break-it review findings (P1-1..P1-4, P2, P3-1..5) on branch stream-close-log, base d66f8041f, review tests cherry-picked (head d3e967003). Claude Opus job.

Start: fixing break-it review findings (P1-1..P1-4, P2, P3-1..5) on branch stream-close-log, base d66f8041f, review tests cherry-picked (head d3e967003). Claude Opus job.
kayg commented 2026-10-18 03:11:14 +00:00
Author
Owner
Copy link

Finish: break-it review fixes on stream-close-log, head 9c74ce405 (pushed). Base d66f8041f + review tests (d95c35e0d, d3e967003).

Commits, one per finding:

  • P1-4 dd1cccc4b: approved plain close sentences back, new server reasons in plain words, fallback restored; fast gate now runs test/close-reasons, src/close-reasons, src/review-close-log, src/connection*.
  • P1-3 7b9f9309f: console line on every socket-retiring path (forceClose for named Close incl. refused reconnect, liveness timeout, wire violation).
  • P3-1/P3-2 page 3a5b34929: "N more times since the previous line" + window-end timer.
  • P3-5 e946d0e8d: recovering link = routine "… Reconnecting…" status (no toast); non-final stream Close never says "The terminal session ended.".
  • P1-1 0fcb5ef28: limiter key = (account, reason category) from the authenticated user, never the client's session id; unowned attach logged as who=browser unowned_session_attach with session/host "none"; record() = try_send into a 1,024-record queue to one logger thread (drop and count, one warning per minute).
  • P1-2 0464b1d8e: full table -> server-wide window per category (limit_scope "server").
  • P3-2/P3-1 agent bfefd4a33: LinkNotes timer + honest wording.
  • P3-4 fc46ae57d: refused-resume lookup after the close, 250 ms bound.
  • P3-3 cf456442a: dead socket-close arm removed.
  • P2 6a215e2da: deadline queue in start order (in P1-1) + multi-thread bench.
  • 6aa25c248, 9c74ce405: test robustness (2 s authority bound in the flood test; settled() in the input-credit helper).
  • 6456aab86: size caps ratcheted down per D243/D264.

P2 numbers, perf VM (4 vCPU, release, flock, 5 runs): record() on the socket worker 64-78 ns (first close of a new account), 44-64 ns (storm), 181-299 ns (4-worker flood far past the bound, excess dropped). Logger thread: 5.2-7.4 us per JSON line written, 292-383 ns per counted close. Baseline at d3e967003 (review bench, release, same VM): 1,022-1,266 ns current-thread, 1,679-2,642 ns 4 workers, on the socket worker, plus the line itself.

Size: x86_64 21,306,416 B (D301 cap -30,688; +6,400 vs pre-#227), aarch64 19,492,672 B (-40,808; -7,704 vs pre-#227). Caps ratcheted to these.

New findings (evidence):

  1. restart_open_liveness_review::responsive_socket_survives_a_held_terminal_open_with_an_existing_attachment hangs until its 90 s watchdog intermittently: 0/10 at merge base b23130139, 2/10 at 1025ba97d (before these fixes), ~3/10 at head. So the original #227 branch introduced it. The hang is the last step: after advance(25ms) the server's probe after the open never comes. Not fixed; needs its own job.
  2. An authority read missing the 25 ms bound under load tears the socket down labelled PROTOCOL_ERROR ("the terminal stream sent an invalid protocol frame"): the loop cannot tell an authority timeout from a client frame error. Not fixed.
  3. Under host load ~17-20 the full test step flakes on a different test each run (standby control_queue, unchanged from dev; flood test handshake "did not send Welcome"). All pass in isolation (standby 0/10, flood 0/10, input-credit 0/15 failures).

scripts/verify fast at 9c74ce405 (verbatim):

test auth::tests::socket_tests::review_close_log::attaches_to_unowned_session_ids_cannot_flood_the_close_log ... FAILED
test remote_runtime::tests::standby::control_queue::saturated_native_control_queue_does_not_hide_primary_terminal_output ... FAILED
test result: FAILED. 484 passed; 2 failed; 49 ignored; 0 measured; 0 filtered out; finished in 331.41s
PT_INTERP: absent; size: 21306416 bytes; cap: 21306416 bytes; delta: +0 bytes
PT_INTERP: absent; size: 19492672 bytes; cap: 19492672 bytes; delta: +0 bytes

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | FAIL (101)   |
| cargo-deny                | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
rc=1

Tracer gate tests (bun, run separately since the gate stops at the Rust failure): 53 pass, 0 fail.

Finish: break-it review fixes on stream-close-log, head 9c74ce405 (pushed). Base d66f8041f + review tests (d95c35e0d, d3e967003). Commits, one per finding: - P1-4 dd1cccc4b: approved plain close sentences back, new server reasons in plain words, fallback restored; fast gate now runs test/close-reasons, src/close-reasons, src/review-close-log, src/connection*. - P1-3 7b9f9309f: console line on every socket-retiring path (forceClose for named Close incl. refused reconnect, liveness timeout, wire violation). - P3-1/P3-2 page 3a5b34929: "N more times since the previous line" + window-end timer. - P3-5 e946d0e8d: recovering link = routine "… Reconnecting…" status (no toast); non-final stream Close never says "The terminal session ended.". - P1-1 0fcb5ef28: limiter key = (account, reason category) from the authenticated user, never the client's session id; unowned attach logged as who=browser unowned_session_attach with session/host "none"; record() = try_send into a 1,024-record queue to one logger thread (drop and count, one warning per minute). - P1-2 0464b1d8e: full table -> server-wide window per category (limit_scope "server"). - P3-2/P3-1 agent bfefd4a33: LinkNotes timer + honest wording. - P3-4 fc46ae57d: refused-resume lookup after the close, 250 ms bound. - P3-3 cf456442a: dead socket-close arm removed. - P2 6a215e2da: deadline queue in start order (in P1-1) + multi-thread bench. - 6aa25c248, 9c74ce405: test robustness (2 s authority bound in the flood test; settled() in the input-credit helper). - 6456aab86: size caps ratcheted down per D243/D264. P2 numbers, perf VM (4 vCPU, release, flock, 5 runs): record() on the socket worker 64-78 ns (first close of a new account), 44-64 ns (storm), 181-299 ns (4-worker flood far past the bound, excess dropped). Logger thread: 5.2-7.4 us per JSON line written, 292-383 ns per counted close. Baseline at d3e967003 (review bench, release, same VM): 1,022-1,266 ns current-thread, 1,679-2,642 ns 4 workers, on the socket worker, plus the line itself. Size: x86_64 21,306,416 B (D301 cap -30,688; +6,400 vs pre-#227), aarch64 19,492,672 B (-40,808; -7,704 vs pre-#227). Caps ratcheted to these. New findings (evidence): 1. restart_open_liveness_review::responsive_socket_survives_a_held_terminal_open_with_an_existing_attachment hangs until its 90 s watchdog intermittently: 0/10 at merge base b23130139, 2/10 at 1025ba97d (before these fixes), ~3/10 at head. So the original #227 branch introduced it. The hang is the last step: after advance(25ms) the server's probe after the open never comes. Not fixed; needs its own job. 2. An authority read missing the 25 ms bound under load tears the socket down labelled PROTOCOL_ERROR ("the terminal stream sent an invalid protocol frame"): the loop cannot tell an authority timeout from a client frame error. Not fixed. 3. Under host load ~17-20 the full test step flakes on a different test each run (standby control_queue, unchanged from dev; flood test handshake "did not send Welcome"). All pass in isolation (standby 0/10, flood 0/10, input-credit 0/15 failures). scripts/verify fast at 9c74ce405 (verbatim): ``` test auth::tests::socket_tests::review_close_log::attaches_to_unowned_session_ids_cannot_flood_the_close_log ... FAILED test remote_runtime::tests::standby::control_queue::saturated_native_control_queue_does_not_hide_primary_terminal_output ... FAILED test result: FAILED. 484 passed; 2 failed; 49 ignored; 0 measured; 0 filtered out; finished in 331.41s PT_INTERP: absent; size: 21306416 bytes; cap: 21306416 bytes; delta: +0 bytes PT_INTERP: absent; size: 19492672 bytes; cap: 19492672 bytes; delta: +0 bytes +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | FAIL (101) | | cargo-deny | PASS | | static builds | PASS | +---------------------------+--------------+ rc=1 ``` Tracer gate tests (bun, run separately since the gate stops at the Rust failure): 53 pass, 0 fail.
kayg commented 2026-10-18 03:14:41 +00:00
Author
Owner
Copy link

Start (Opus job, two items before review): branch stream-close-log, base head 9c74ce405.

  1. Intermittent hang in restart_open_liveness_review::responsive_socket_survives_a_held_terminal_open_with_an_existing_attachment (0/10 at merge base b23130139, ~3/10 at head): measure base vs head, find the ordering cause, fix product or test.
  2. Authority-read timeout during a frame is logged as protocol_error/invalid frame: give it its own close reason.
Start (Opus job, two items before review): branch `stream-close-log`, base head `9c74ce405`. 1. Intermittent hang in `restart_open_liveness_review::responsive_socket_survives_a_held_terminal_open_with_an_existing_attachment` (0/10 at merge base `b23130139`, ~3/10 at head): measure base vs head, find the ordering cause, fix product or test. 2. Authority-read timeout during a frame is logged as `protocol_error`/invalid frame: give it its own close reason.
kayg commented 2026-10-18 04:22:26 +00:00
Author
Owner
Copy link

Finish (Opus job): head 4ac9e68fb on stream-close-log.

1. Hang in restart_open_liveness_review: fixed in the product (10432b599).
The race is older than this branch: it exists in the proof loop at the merge base. The first attach flips the activity watch while a sink write's proof_changed permit is pending. The unbiased select can wake on proof_changed with the loop's local active still false; the code then read the fresh value (*activity.borrow(), true), sent startup probe 1, and only on the next turn applied the activation, whose reset() clears issued probes. The browser's answer to probe 1 was refused, so the route stayed unsampled with sent=1 and the next probe fell due at origin+1 µs+25 ms: 1 µs after the test's single 25 ms advance, so under the frozen clock no probe ever came. Temporary trace from a hanging run:

wake proof_changed local_active=false
ping nonce=1 active=true sent=1 acked=0 issued=[1]
activity changed active=true before-reset sent=1 acked=0 issued=[1]
pong nonce=1 ok=false acked=0 issued=[]
advance 25ms        (no probe follows)

Fix: probe only when both the applied and the current activation are active (active && *activity.borrow()). The test was right; it was not changed.

Runs (40 s timeout each, same host): base b23130139 18 pass / 2 hang of 20; head 9c74ce405 16 / 4 of 20; head+fix 30 / 0 of 30. No bisect was needed: base hangs too, so the branch only changed timing.

2. Authority timeout gets its own close reason (85b4f987d, 4ac9e68fb).
New AUTHORITY_UNAVAILABLE (server_busy / authority_unavailable, "the server was too busy to confirm the session"; on the page: "The server was too busy to confirm your session; reconnecting."). The frame path records whether the handler or only the authority wrapper failed; teardown confirms it with one bounded non-renewing read (Denied → resume refused, timeout/storage error → authority_unavailable, live → provisional reason). No change to while_authorized/authority_deadline/session_is_live; merge-tree against origin/authority-read-retry shows the same 2 pre-existing conflict hunks before and after. New test close_logging::authority_read_timeout_during_a_frame_logs_authority_unavailable (holds the pool's one connection): 10/10 alone, close_logging module 34/35 (one early failure before the teardown classifier existed; 30/30 since).

Gate scripts/verify fast at head:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Static builds fail only on the exact size caps: x86_64 21307728 vs cap 21306416 (+1312 B), aarch64 19494176 vs 19492672 (+1504 B). The first version was +1888 B on x86_64; folding the classifier into a single teardown await site brought it down. Caps not raised (needs an orchestrator/owner OK, D243).

Finish (Opus job): head `4ac9e68fb` on `stream-close-log`. **1. Hang in restart_open_liveness_review: fixed in the product (`10432b599`).** The race is older than this branch: it exists in the proof loop at the merge base. The first attach flips the activity watch while a sink write's `proof_changed` permit is pending. The unbiased select can wake on `proof_changed` with the loop's local `active` still false; the code then read the fresh value (`*activity.borrow()`, true), sent startup probe 1, and only on the next turn applied the activation, whose `reset()` clears issued probes. The browser's answer to probe 1 was refused, so the route stayed unsampled with sent=1 and the next probe fell due at origin+1 µs+25 ms: 1 µs after the test's single 25 ms advance, so under the frozen clock no probe ever came. Temporary trace from a hanging run: ``` wake proof_changed local_active=false ping nonce=1 active=true sent=1 acked=0 issued=[1] activity changed active=true before-reset sent=1 acked=0 issued=[1] pong nonce=1 ok=false acked=0 issued=[] advance 25ms (no probe follows) ``` Fix: probe only when both the applied and the current activation are active (`active && *activity.borrow()`). The test was right; it was not changed. Runs (40 s timeout each, same host): base b23130139 **18 pass / 2 hang of 20**; head 9c74ce405 **16 / 4 of 20**; head+fix **30 / 0 of 30**. No bisect was needed: base hangs too, so the branch only changed timing. **2. Authority timeout gets its own close reason (`85b4f987d`, `4ac9e68fb`).** New `AUTHORITY_UNAVAILABLE` (server_busy / authority_unavailable, "the server was too busy to confirm the session"; on the page: "The server was too busy to confirm your session; reconnecting."). The frame path records whether the handler or only the authority wrapper failed; teardown confirms it with one bounded non-renewing read (Denied → resume refused, timeout/storage error → authority_unavailable, live → provisional reason). No change to while_authorized/authority_deadline/session_is_live; merge-tree against origin/authority-read-retry shows the same 2 pre-existing conflict hunks before and after. New test close_logging::authority_read_timeout_during_a_frame_logs_authority_unavailable (holds the pool's one connection): 10/10 alone, close_logging module 34/35 (one early failure before the teardown classifier existed; 30/30 since). **Gate** `scripts/verify fast` at head: ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` Static builds fail only on the exact size caps: x86_64 21307728 vs cap 21306416 (**+1312 B**), aarch64 19494176 vs 19492672 (**+1504 B**). The first version was +1888 B on x86_64; folding the classifier into a single teardown await site brought it down. Caps not raised (needs an orchestrator/owner OK, D243).
kayg commented 2026-10-18 05:05:34 +00:00
Author
Owner
Copy link

Break-it review round 2: stream-close-log (#227) at 300a1d6d9

Reviewer: Claude Opus (independent). Review branch: review-stream-close-log-r2 e39549741 (pushed). One commit, tests only, no fixes.

Prior findings

Finding Status Evidence
P1-1 limiter key chosen by the client, blocking write FIXED key = (account hash, reason category); an unowned attach is logged as unowned_session_attach with session/host none; record() = try_send to a 1,024-slot queue read by one thread. review_close_log::attaches_to_unowned_session_ids_cannot_flood_the_close_log passes.
P1-2 a full map turns limiting off FIXED full table → one server-wide window per category. a_full_table_keeps_rate_limiting_a_storm passes; read the code, it is correct (see below).
P1-3 console line skipped on forceClose PARTLY FIXED, see P1-A named Close, liveness timeout, wire violation now print. onerror still does not.
P1-4 jargon, tracer tests not gated FIXED test/close-reasons, src/close-reasons, src/review-close-log, src/connection* now in ts.sh; they pass.
P2-1 summary scan under the lock FIXED the thread owns the limiter; a VecDeque in start order gives O(1). Perf VM numbers on the issue: 44–78 ns per close on the socket worker.
P3-1 / P3-2 wording, no summary timer FIXED page and infraternal agent: "since the previous line" plus a window-end timer.
P3-3 dead arm FIXED (cf456442a)
P3-4 refusal SQL before close FIXED runs after the close, bounded at 250 ms
P3-5 recovering read as failure FIXED (e946d0e8d)

P1-A A reconnect that fails at the network still prints no console line

socket.onerror (connection.ts ~438) calls disconnect(socket) and does not call noteClose. Browsers fire error and then close (1006) in two cases: when a WebSocket cannot be established, and when it drops uncleanly. The close event then fails this.socket !== socket and returns. A failed reconnect (server down, network unreachable, TLS failure), the case the #227 console line was added for, therefore prints nothing. This path is older than the branch, but the branch claims "one console line per close incl. failed reconnects", and 7b9f9309f says "every path that retires a socket for a close calls it".
Failing test: apps/tracer/src/review-close-log-r2.test.ts "a reconnect that fails at the network (error, then close 1006) still prints its console line": expected 1 warning, got 0.

Checked: account-key exhaustion (no finding)

The table holds (account, category) pairs and there are about 11 categories, so filling 8,192 windows takes at least 745 accounts closing within 60 s. One account cannot do it. Once the table is full, a new key goes to the window for its category, and at most about 11 server-scope first lines are written per 60 s. Those windows expire on their own deadline (next_deadline includes them). Per-account tracking resumes as soon as account windows expire. Keys stay consistent: each key in the map has exactly one entry in order, and only expire removes entries. A known design trade-off: during overflow, a real account's first close is folded into the server window.

Checked: the logger thread blocking or dying

  • If stderr blocks, only the close-log thread waits. The queue tops out at 1,024 × CloseRecord (about 200 KiB). Beyond that, closes are dropped and counted, and nothing pushes back on the sockets. The drop warning is written when the thread unblocks. OK.
  • P3-A If the thread dies (a panic in emit or the subscriber) or spawn fails (.ok()? stores None in the OnceLock), every later close is counted into dropped, and nothing ever reports that count, because only the dead thread reports. Close logging then stops for the life of the process without a word. Suggestion: when try_send returns Disconnected, or the queue is None, warn once from the caller with an AtomicBool guard. No test: neither failure can be reached without a fault injection hook.

Checked: teardown reads under a close storm

failure_reason runs only with attachments, an Open transport, and a teardown reason of PROTOCOL_ERROR, AUTHORITY_UNAVAILABLE or SERVER_PROCESSING_FAILED. It does not run for ordinary closes such as client_went_away, idle or size. Its cost is one SELECT, at most AUTHORITY_READ_TIMEOUT (25 ms).

  • P3-B It is followed by session_is_live, a second read, which decides whether to send the 1011. For the frame path that is the same count as before (the old in-loop session_is_live was moved). For exits through the SERVER_PROCESSING_FAILED default (output, plugin and recovery arms whose while_authorized fails), it is a new read. Under a store-wide stall every such socket now queues 2 pool waiters instead of 1, and its close reaches the browser up to 50 ms after the failure instead of 25 ms. failure_reason's result could decide the 1011 directly: live or PROTOCOL_ERROR means send it, RESUME_REFUSED means do not, a timeout means do not.

P3-C A busy store can still be blamed on the browser

frame_failed is true whenever handle_client_frame returns Err. That includes failures inside the handler: session_is_live checks (AttachTerm ×2, refused Call), session_host_id and owned_writer store errors, and session_gone(...)?. A read that misses its 25 ms bound inside the handler, followed by a teardown read that succeeds once the load spike has passed, is logged as protocol_error/invalid_frame. The browser is then told "This window sent something the server did not understand". The session_host_id Err arm still writes PROTOCOL_ERROR inline with who=server. This is narrower than before 85b4f987d, but its claim that a busy server never reads as a misbehaving browser holds only for the wrapper's own reads. No deterministic test: hitting it needs the handler's read to stall while the wrapper's 25 ms tick does not.

P3-D 10432b599 does not cover active → idle → active

let active = active && *activity.borrow(); blocks probes for an activation the loop has not applied, when the applied state is idle. It does not block them when the applied state is active and the watch went false and then true again without the loop seeing it. That happens when the last attachment closes and a new attach follows, which is the routine single-terminal reattach after "reader fell behind" or a stream Close. The borrow then reads true and the local value is true, so a probe goes out. On the next turn changed() runs reset(), which clears issued, and the answer is refused: the same wasted sample and one-tick delay the commit describes. Idle → active → idle and active → idle are handled correctly. Suggested condition: active && !activity.has_changed().unwrap_or(true). No deterministic test: it needs the proof task to be parked inside authority.run while the socket loop detaches and attaches.

P3-E Wording of the new sentence

"The server was too busy to confirm your session; reconnecting." is the only sentence that ends in "; reconnecting." The recovering sentences read ". Reconnecting…". Failing test: review-close-log-r2.test.ts "the busy-server sentence reads like the other recovering sentences".

Gate

scripts/verify fast at e39549741 (verbatim table):

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | FAIL (101)   |
| cargo-deny                | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
rc=1

The test failure is infraternal-plugin-ssh --test config_isolation::configured_identity_paths_are_not_inherited_by_the_command_child ("The SSH helper could not start."). The branch does not touch crates/plugins/ssh, and the test passed when rerun alone (2 pass), so it is a host flake. Cargo stopped at that binary, so I ran cargo test -p infraternal-server --lib separately: 294 passed, 0 failed, 23 ignored. Static size caps PASS at the ratcheted values: 21,307,728 B (x86_64) and 19,494,176 B (aarch64). The gate never reached the tracer step. Run by hand (bun test src/review-close-log-r2.test.ts src/review-close-log.test.ts test/close-reasons.test.ts src/close-reasons.test.ts src/connection): 53 pass, 2 fail, and the 2 failures are the intended review tests.

Verdict: HOLD (1 P1, 0 P2, 5 P3). P1-A is a small fix: noteClose(undefined, "", …) in onerror before disconnect.

# Break-it review round 2: stream-close-log (#227) at 300a1d6d9 Reviewer: Claude Opus (independent). Review branch: `review-stream-close-log-r2` e39549741 (pushed). One commit, tests only, no fixes. ## Prior findings | Finding | Status | Evidence | |---|---|---| | P1-1 limiter key chosen by the client, blocking write | FIXED | key = (account hash, reason category); an unowned attach is logged as `unowned_session_attach` with session/host `none`; `record()` = `try_send` to a 1,024-slot queue read by one thread. `review_close_log::attaches_to_unowned_session_ids_cannot_flood_the_close_log` passes. | | P1-2 a full map turns limiting off | FIXED | full table → one server-wide window per category. `a_full_table_keeps_rate_limiting_a_storm` passes; read the code, it is correct (see below). | | P1-3 console line skipped on forceClose | PARTLY FIXED, see **P1-A** | named Close, liveness timeout, wire violation now print. `onerror` still does not. | | P1-4 jargon, tracer tests not gated | FIXED | `test/close-reasons`, `src/close-reasons`, `src/review-close-log`, `src/connection*` now in `ts.sh`; they pass. | | P2-1 summary scan under the lock | FIXED | the thread owns the limiter; a `VecDeque` in start order gives O(1). Perf VM numbers on the issue: 44–78 ns per close on the socket worker. | | P3-1 / P3-2 wording, no summary timer | FIXED | page and infraternal agent: "since the previous line" plus a window-end timer. | | P3-3 dead arm | FIXED (`cf456442a`) | | | P3-4 refusal SQL before close | FIXED | runs after the close, bounded at 250 ms | | P3-5 recovering read as failure | FIXED (`e946d0e8d`) | | ## P1-A A reconnect that fails at the network still prints no console line `socket.onerror` (connection.ts ~438) calls `disconnect(socket)` and does not call `noteClose`. Browsers fire `error` and then `close` (1006) in two cases: when a WebSocket cannot be established, and when it drops uncleanly. The `close` event then fails `this.socket !== socket` and returns. A failed reconnect (server down, network unreachable, TLS failure), the case the #227 console line was added for, therefore prints nothing. This path is older than the branch, but the branch claims "one console line per close incl. failed reconnects", and 7b9f9309f says "every path that retires a socket for a close calls it". Failing test: `apps/tracer/src/review-close-log-r2.test.ts` "a reconnect that fails at the network (error, then close 1006) still prints its console line": expected 1 warning, got 0. ## Checked: account-key exhaustion (no finding) The table holds (account, category) pairs and there are about 11 categories, so filling 8,192 windows takes at least 745 accounts closing within 60 s. One account cannot do it. Once the table is full, a new key goes to the window for its category, and at most about 11 server-scope first lines are written per 60 s. Those windows expire on their own deadline (`next_deadline` includes them). Per-account tracking resumes as soon as account windows expire. Keys stay consistent: each key in the map has exactly one entry in `order`, and only `expire` removes entries. A known design trade-off: during overflow, a real account's first close is folded into the server window. ## Checked: the logger thread blocking or dying - If stderr blocks, only the `close-log` thread waits. The queue tops out at 1,024 × `CloseRecord` (about 200 KiB). Beyond that, closes are dropped and counted, and nothing pushes back on the sockets. The drop warning is written when the thread unblocks. OK. - **P3-A** If the thread dies (a panic in `emit` or the subscriber) or `spawn` fails (`.ok()?` stores `None` in the `OnceLock`), every later close is counted into `dropped`, and nothing ever reports that count, because only the dead thread reports. Close logging then stops for the life of the process without a word. Suggestion: when `try_send` returns `Disconnected`, or the queue is `None`, warn once from the caller with an `AtomicBool` guard. No test: neither failure can be reached without a fault injection hook. ## Checked: teardown reads under a close storm `failure_reason` runs only with attachments, an Open transport, and a teardown reason of PROTOCOL_ERROR, AUTHORITY_UNAVAILABLE or SERVER_PROCESSING_FAILED. It does not run for ordinary closes such as `client_went_away`, idle or size. Its cost is one SELECT, at most `AUTHORITY_READ_TIMEOUT` (25 ms). - **P3-B** It is followed by `session_is_live`, a second read, which decides whether to send the 1011. For the frame path that is the same count as before (the old in-loop `session_is_live` was moved). For exits through the SERVER_PROCESSING_FAILED default (output, plugin and recovery arms whose `while_authorized` fails), it is a new read. Under a store-wide stall every such socket now queues 2 pool waiters instead of 1, and its close reaches the browser up to 50 ms after the failure instead of 25 ms. `failure_reason`'s result could decide the 1011 directly: `live` or `PROTOCOL_ERROR` means send it, `RESUME_REFUSED` means do not, a timeout means do not. ## P3-C A busy store can still be blamed on the browser `frame_failed` is true whenever `handle_client_frame` returns Err. That includes failures inside the handler: `session_is_live` checks (AttachTerm ×2, refused Call), `session_host_id` and `owned_writer` store errors, and `session_gone(...)?`. A read that misses its 25 ms bound inside the handler, followed by a teardown read that succeeds once the load spike has passed, is logged as `protocol_error`/`invalid_frame`. The browser is then told "This window sent something the server did not understand". The `session_host_id` Err arm still writes `PROTOCOL_ERROR` inline with who=server. This is narrower than before 85b4f987d, but its claim that a busy server never reads as a misbehaving browser holds only for the wrapper's own reads. No deterministic test: hitting it needs the handler's read to stall while the wrapper's 25 ms tick does not. ## P3-D 10432b599 does not cover active → idle → active `let active = active && *activity.borrow();` blocks probes for an activation the loop has not applied, when the applied state is idle. It does not block them when the applied state is active and the watch went false and then true again without the loop seeing it. That happens when the last attachment closes and a new attach follows, which is the routine single-terminal reattach after "reader fell behind" or a stream Close. The borrow then reads true and the local value is true, so a probe goes out. On the next turn `changed()` runs `reset()`, which clears `issued`, and the answer is refused: the same wasted sample and one-tick delay the commit describes. Idle → active → idle and active → idle are handled correctly. Suggested condition: `active && !activity.has_changed().unwrap_or(true)`. No deterministic test: it needs the proof task to be parked inside `authority.run` while the socket loop detaches and attaches. ## P3-E Wording of the new sentence "The server was too busy to confirm your session; reconnecting." is the only sentence that ends in "; reconnecting." The recovering sentences read "<cause>. Reconnecting…". Failing test: `review-close-log-r2.test.ts` "the busy-server sentence reads like the other recovering sentences". ## Gate `scripts/verify fast` at e39549741 (verbatim table): ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | FAIL (101) | | cargo-deny | PASS | | static builds | PASS | +---------------------------+--------------+ rc=1 ``` The test failure is `infraternal-plugin-ssh --test config_isolation::configured_identity_paths_are_not_inherited_by_the_command_child` ("The SSH helper could not start."). The branch does not touch `crates/plugins/ssh`, and the test passed when rerun alone (2 pass), so it is a host flake. Cargo stopped at that binary, so I ran `cargo test -p infraternal-server --lib` separately: **294 passed, 0 failed, 23 ignored**. Static size caps PASS at the ratcheted values: 21,307,728 B (x86_64) and 19,494,176 B (aarch64). The gate never reached the tracer step. Run by hand (`bun test src/review-close-log-r2.test.ts src/review-close-log.test.ts test/close-reasons.test.ts src/close-reasons.test.ts src/connection`): 53 pass, 2 fail, and the 2 failures are the intended review tests. **Verdict: HOLD** (1 P1, 0 P2, 5 P3). P1-A is a small fix: `noteClose(undefined, "", …)` in `onerror` before `disconnect`.
kayg commented 2026-10-18 06:00:27 +00:00
Author
Owner
Copy link

Round-2 fixes pushed to stream-close-log, head 57cf18adbb (base 300a1d6d9; review tests e39549741 cherry-picked as 38f2f1e66).

  • P1-A onerror logs the close; P3-E sentence; P3-A logger fallback; P3-B single teardown read; P3-C authority_unavailable in attach paths; P3-D has_changed gate.

scripts/verify fast (final table, verbatim):


+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Size: static builds FAIL on the caps only: x86 21309072 vs cap 21307728 (+1344), ARM 19495376 vs 19494176 (+1200). Not raised.

Round-2 fixes pushed to stream-close-log, head 57cf18adbbf3b269f0e260d128ea738479e27d6f (base 300a1d6d9; review tests e39549741 cherry-picked as 38f2f1e66). - P1-A onerror logs the close; P3-E sentence; P3-A logger fallback; P3-B single teardown read; P3-C authority_unavailable in attach paths; P3-D has_changed gate. scripts/verify fast (final table, verbatim): ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` Size: static builds FAIL on the caps only: x86 21309072 vs cap 21307728 (+1344), ARM 19495376 vs 19494176 (+1200). Not raised.
kayg commented 2026-10-18 08:34:22 +00:00
Author
Owner
Copy link

Release 3 deployed to production (~16:03 IST): release-prod-live 5b2a1b9ee, server sha256 aacc763d…, MainPID 88336, /healthz ok. Contains Retry after a failed instant setup (#230) and connection-close reasons (#227). Staging: Retry after injected instant failure passed rootless + root on the perf VM (prompt ~350 ms, fully joined ~33 s). CA locked after the restart until the owner unlocks it.

Release 3 deployed to production (~16:03 IST): release-prod-live 5b2a1b9ee, server sha256 aacc763d…, MainPID 88336, /healthz ok. Contains Retry after a failed instant setup (#230) and connection-close reasons (#227). Staging: Retry after injected instant failure passed rootless + root on the perf VM (prompt ~350 ms, fully joined ~33 s). CA locked after the restart until the owner unlocks it.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
j470b-server-mesh-rejoin
mesh-compat-382
j444b-d381-key-wins-flood
j461b-macos-utun-reader-thread
j475b-remove-host-motion
fix-reconnect-new-link
j481-d380
dev
j479-mesh-soak
fix/j476-host-facts
j475-motion-pill-split
j470-mac-node-join
root-upgrade-d376
j474-review-merge
latency-targets-120hz
nightly-suite-shards
gotatun-pin-j469
macos-utun-reader-thread
fix-flood-echo-stall
wip/j416
motion-fast-ease-out
mesh-addhost-j259
takeover-close-terminals-d368
kbd-instant-followups
fix-trust-key-double-click
nightly-suite-partial-report
appwin-icons-j207
rusqlite-slice1
swipe-content-first-d374
mesh-mtu-381
addhost-instant-open
glass-contrast-d365-mock
glass-contrast-d365
signout-followups-442
fix-startup-toast-red
research-gotatun-matrix-j317
themes-sync-j304
fix-load-flakes
fix-nix-zig-patch
fix-compute01-browser-baseline
ratchet-size-baseline-419
predict-auto-d130
revoke-coverage-d367
nightly-suite-queue-timeout
nightly-browser-suite
fix-rustsec-2026-0459
fonts-ioskeley-obel
log-retention-d375
fix-kbd-no-anim-427
fix-dev-red-specs-2
takeover-proof-305-j205
wip/j410-server-pump
ui-consistency-d372
fix-osc52-clipboard
libghostty-vt-1.0
forget-retired-host-d369
j404-merge-426
fix-setup-lock-race
fix-touch-select-426
revoke-access-report-ui
fix-predictive-echo-specs
fix-selinux-agent-exec
wip/j395-checkpoint2
wip/j395
j391-review-scratch
ui-consistency-j195-deferred-rebased
fix-rustc-paths-421
decisions-wording-341
fix-install-suite-red
fix-dev-rtt-specs
fix-pending-removal-align
fix-privileged-skip
fix-install-suite-404
review-j368
mesh-dns-j254
revoke-access-report-d356b
revoke-privilege-round5
fix-removal-confirm
review-j360
gotatun-rotation-367
deps-refresh-340
review-j354
review-J351
review-j350
addhost-takeover-d359
revoke-report-items-347
revoke-end-sessions-d362
review-j348
size-mesh-ssh-346
size-trim-337
revoke-access-diff-d356
j259-pre-rebase-b06c5a6
j305-ended-at-start-spec
addhost-sudo-consent
fix-layout-real-specs-411
fix-shell-open-411
fix-console-hosts-373
fix-shell-exit
fuzz-host-name-353
fix-host-details-410
macos-filter-j197
research-gotatun-tricks
revoke-p3-j216l
stale-state-pill
revoke-followups-j216k
revoke-progress-j216e
backup/macos01-20261106/lume/j197
backup/macos01-20261106/lume/appwin-icons-j207-stash0
backup/macos01-20261106/claude/stash-0
backup/macos01-20261106/claude/stash-1
backup/macos01-20261106/claude/J80-macos-s1
backup/macos01-20261106/claude/J80
backup/macos01-20261106/claude/J80-slice1-20261102
backup/macos01-20261106/claude/J16b-appwin-on-dev
backup/macos01-20261106/claude/appwin-launch-idle-146
backup/macos01-20261106/claude/J80-s1-run
wip/j259
trust-285-p3c
runtime-lock-j216f
mesh-ssh-p3-j175
tun-proof-p3b
trust-285-p3
terminal-polish-j271
tun-proof-p3
ssh-install-flakes-j216i
fuzz-fixes-j264
trust-285-a
trust-285-b
restart-oversized-fixture-j198
backup/macos01-20261106/lume/appwin-icons-j207
restart-bound-j216h
up-preflight-p3-j198
backup/macos01-20261106/lume/j251
up-preflight-j198
wip/up-preflight-j198-j245-p2
wip/up-preflight-j198-j245-p1
agent-fsync-offloop-j196
wip/up-preflight-j198-j245
settings-offscreen-j231
wip/up-preflight-j198-j226-inspection
wip/j103f
themes-j206
wip/up-preflight-j198-j226
flakes-j216c
backup/macos01-20261106/claude/j197
flakes-j216b
backup/macos01-20261106/claude/j197-red
wip/up-preflight-j198-review
flakes-j216
backup/macos01-20261106/claude/appwin-icons-j207
engineering-j208b
appwin-mock-j209
toast-stack-j165
engineering-j208
decisions-d351
decisions-d350
decisions-d349
appearance-presets-mock-j129
decisions-d348
docs-audit-spec
doc-coverage-baseline-raise
decisions-d347
decisions-d346b
decisions-d346
j88h-profile
ui-consistency-j195-deferred
ui-consistency-j195
engineering-practices-j194
preferred-login-ui-j166
signin-pools-263-j174
macos-node-265-s1
decisions-d345
decisions-d344
proxy-cgroup-j164
verify-quick-fuzz-329
proxy-start-io-j168
timing-flakes-j163
up-263-s1
mcp-events-j147
verify-quick-fuzz-j158
proxy-p3-j159
agent-resource-j157-detect
wip/j103d-final
remote-cancel-hang-j155
proxy-traefik-281-j62d
j149-verify-quick
proxy-traefik-281-j62b
j150-drop-unnamed
wip/j79d-red-r4
wip/j103d-d332
rootless-toast-j142
wip/j103d-head
wip/j103d
j125c-mcp
macos-build-j136
backup/macos01-20261106/claude/j136
wip/j79d-red-cli
wip/j79d-red-server
j121-link-liveness
j118c-tracer-failures
wip/j79c-head
release-desktop-j131
j124c-signed-macos
wip/j79c-red-p3
wip/j79c-red-p23
wip/j79c-red-p22
merge-j128-check
perf-deferred-j128
sidebar-toggle-260
j127-shortcut-changed-dot
j125-mcp
j118-tracer-failures
j126-account-summary
settings-sections-mock-j92
j120-pinch-lock
sidebar-292
backup/macos01-20261106/claude/j120
taste-2026-11-03
row-actions-242-215
backup/macos01-20261106/claude/stash-0-untracked
backup/macos01-20261106/claude/J80-265-s1-run
backup/macos01-20261106/claude/J80-macos-s1-20261102
j112-canvas-commands
ui-no-overshoot-scroll
fix-recovery-restart
backup/macos01-20261106/claude/stash-1-untracked
taste-2026-11-02
size-baseline-ratchet
machine-cards-structure-j108
j109-boot-copy-mockups
fix-dev-red
wip/up-263-s1
j95b-settings-rows
proxy-traefik-281
j95-option-rows
docs-human-j91
docs-keys-palette
fix-flood-1013-270-j38d
mesh-ssh-s1
mesh-ssh-s0
addhost-d291-tunnel
test-stability-load
mesh-relay-220
repro-296
shim-thp-off-24
mesh-pins-258-v2
mesh-dns
signin-d311-merged
docs-pages
ui-size-283
fix-ui-batch-specs
wip/j12b-edge-update
gate-on-arm
fix-rename-sync-293
trust-285-j60b
fix-focus-ring-clip
integ-ui-1101
edge-update-v2
signin-d311
taste-2026-11-01
layout-per-tab-250
ui-approved-splitopt-224
ui-approved-splitopt-224-on-dev
icon-text-centre-282
ui-approved-189-212-216
research-reverse-proxy
transfer-budget-320
review-mesh-pins-258-r2
sidebar-pill-259
review-mesh-relay-220
wip/j12-edge-update-v2
nix-package
status-crash-loop-237
fix-dev-test-failures
fix-fixture-socket-path
wip/mesh-relay-220-pre-dev-advance-rebase
fix-canvas-side-clip
research-head-to-head
frosted-real-256
review-mesh-pins-258
wip/mesh-relay-220-pre-final-rebase
review-layout-per-tab-250-r2
broad-suite-triage
nix-devshell
review-status-crash-loop-237-r3
wip/nix-devshell-pre-p3
taste-frosted-real
wip/code-serve-demo-2026-10-30
wip/code-serve-glass-2026-10-30
wip/code-serve-clusters-2026-10-30
wip/code-merge-release94-dev-2026-10-30
wip/rproxy-delta-scratch-2026-10-30
wip/stale-note-257-staged-2026-10-30
wip/d241-size-staged-2026-10-30
review-broad-suite-207
wip/mesh-pins-258-pre-split
wip/mesh-relay-220-pre-rebase
mesh-measure-220
plan-211-update
review-edge-autoupdate-r3
perf24-reconnect
review-stale-note-r2
release-prod-live
d241-shell-marker
review-mesh-pins-server-258
review-revocation-r3
review-setup-token-r2
review-sync-journal-r3
wip/review-mesh-pins-258-prune-review-1995fee46
frosted-scroll-256
sync-journal-r3
revocation-per-account-wip
review-tokio-asyncfd-0386
review-sync-p3s
wip/stash-8by64-stash0-2026-10-30
mesh-drag-r2
setup-token-data-wip
release-9-4
auth-busy-1013
mesh-view-overlap
sync-p3s
review-auth-busy-wake
review-readd-r8-wip
mesh-drag-bounds
audit-auth-93
app-windows-stream-start
review-authority-cap-r3
app-windows-r7-measure
readd-minimal-r6
readd-minimal-r5
app-windows-r7-soak
readd-minimal-r4
app-windows-r7
app-windows-r6
readd-minimal-r3
readd-minimal-r2
readd-minimal
merge-release8-dev
review-busy-page-liveness-r2
review-busy-page-liveness
mesh-view-trim
review-authority-read-retry-r2
review-remove-uninstall-r2
release-retry-reattach
release-retry
review-remove-uninstall
review-authority-read-retry
app-windows-ship
mesh-autoswitch-p2
mesh-ssh
mesh-relay
mesh-macos-node
screen-and-control
session-labels
review-real-logins-r5
review-real-logins-r4
review-real-logins-r3
shim-memory-r2
review-real-logins-r2
shim-memory
shadow-addhost-instant
shadow-real-logins
shadow-same-session-flood
shadow-prod-perf-fix
hosts-from-server
app-windows-real
app-windows-transport
release-prod-6w
mesh-policy-verify
quiet-close-toast
dev-merge-r4
review-mesh-agent-transport
review-app-windows-transport
settings-roundtrip
review-release-pipeline-r4
mesh-agent-transport
ended-strip-fix
review-audit-ca-update-d244
settings-rows
phone-width
signout-clear
fix-restart-tests
dev-merge-rp
signer-hardening
review-audit-ca-update
terminal-fidelity
review-audit-ca-update-wip
mesh-integrate
review-mesh-agent-transport-515d3712
review-mesh-dataplane-feec1988
mesh-dataplane
signout
research-canvas-node
wip/review-renderer-cost
wip/review-terminal-fidelity
padlock-glyph
review-release-pipeline-2e95b689
phone-polish
sidebar-polish
inbox-generic
review-mesh-dataplane-378d4055
prod-batch
ui-inbox
ui-ca-unlock
ui-machine-sections
review-mesh-dataplane-a5f744fb
tls-session-prod
session-bounds-prod
fix-dead-session
session-bounds
tls-session
palette-polish
shortcuts-d222
research-appmeta
phone-gutter
ui-centre-button
wip/ui-centre-button
wip/tls-session
review-addhost-breakit
ui-platform
renderer-cost
remote-agents
mesh-api
ssh-bootstrap
prod-hotfix2
prod-fixes
review-remote-agents
review-mesh-dataplane
session-life
research-browser
review-mesh-api
research-capture
review-mesh-api-round3
review-mesh-api-round1
gate-stability
review-gate-stability
review-relay-direct
review-mesh-api-findings
proto-sessions
font-picker
proto-addhost
proto-windows
mesh-core
latency-gate
research-bsd
research-discovery
keytray-mods
mock-core
ui-tweaks
fix-motion-sel
fix-motion
integrate2
fix-robust
t-web
t-server
vt-ghostty
fix-snap-echo
t-web-auth
review-vt2
t-agent
lane-protocol
lane-shim
lane-auth
s1-base
s1-contracts
s1-workspace
No results found.
Labels
Clear labels   
launch-qa
Findings from launch QA.

No labels
launch-qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/infraternal#227
No description provided.