kayg/infraternal
Watch
1
0
Fork
You've already forked infraternal
 0
Code Issues 356 Pull requests Projects Releases Packages Wiki Activity Actions

Review: mobile-class performance of the deployed build #42

New issue
Open
opened 2026-10-07 01:06:49 +00:00 by kayg · 53 comments
kayg commented 2026-10-07 01:06:49 +00:00
Owner
Copy link

Overnight performance review of the deployed build (dev 7d2d0a5) under mobile-class conditions: Chromium with iPhone emulation and 4x CPU throttle. First paint, setup and sign-in, terminal attach, keystroke latency, scroll, the theme switch, 8 sessions, and memory over 30 minutes of mixed use. Findings with numbers land here, and fixes follow.

Overnight performance review of the deployed build (dev 7d2d0a5) under mobile-class conditions: Chromium with iPhone emulation and 4x CPU throttle. First paint, setup and sign-in, terminal attach, keystroke latency, scroll, the theme switch, 8 sessions, and memory over 30 minutes of mixed use. Findings with numbers land here, and fixes follow.
kayg commented 2026-10-07 01:08:33 +00:00
Author
Owner
Copy link

Started mobile performance review on branch fix-perf, from base 7d2d0a535f. Isolated worktree: /home/kayg/.cache/codex-scratch/review-perf/wt.

Started mobile performance review on branch fix-perf, from base 7d2d0a535fe315fb010da133d665c68458f23162. Isolated worktree: /home/kayg/.cache/codex-scratch/review-perf/wt.
kayg commented 2026-10-07 01:30:51 +00:00
Author
Owner
Copy link

Finding: the real static-server terminal.open action is refused. Production iPhone-profile page shows 'The server could not open this terminal.'; scripts/browser-auth/check-merged.mjs on the same 7d2d0a5 static binary reports 'real PTY command output missing' (298 bridge attempts, 0 PTY output). Root cause is under investigation; PTY latency/soak measurements depend on resolving it.

Finding: the real static-server terminal.open action is refused. Production iPhone-profile page shows 'The server could not open this terminal.'; scripts/browser-auth/check-merged.mjs on the same 7d2d0a5 static binary reports 'real PTY command output missing' (298 bridge attempts, 0 PTY output). Root cause is under investigation; PTY latency/soak measurements depend on resolving it.
kayg commented 2026-10-07 01:34:46 +00:00
Author
Owner
Copy link

Correction to the terminal-open finding: diagnosis showed my nested XDG_RUNTIME_DIR produced a PTY Unix-socket path over SUN_LEN; using a 45-character private runtime under the scratch job makes production terminal.open attach successfully. The prior failure is a harness-path artifact, not a product finding. The real-server 500-key probe is now in progress.

Correction to the terminal-open finding: diagnosis showed my nested XDG_RUNTIME_DIR produced a PTY Unix-socket path over SUN_LEN; using a 45-character private runtime under the scratch job makes production terminal.open attach successfully. The prior failure is a harness-path artifact, not a product finding. The real-server 500-key probe is now in progress.
kayg commented 2026-10-07 01:38:55 +00:00
Author
Owner
Copy link

Finding: terminal sessions drop immediately after attach. Evidence: production 4x iPhone run completed AttachTerm, then the page entered 'Connection lost; resuming…' and the 500-key probe stayed disabled; bridge closed after 123 server-to-client / 159 client-to-server bytes. Source confirms the mismatch: apps/tracer/src/terminal.ts startPings sends Ping on TERMINAL_STREAM, while crates/infraternal-server/src/websocket.rs handles client Ping on CONTROL_STREAM. Fixing the client stream now.

Finding: terminal sessions drop immediately after attach. Evidence: production 4x iPhone run completed AttachTerm, then the page entered 'Connection lost; resuming…' and the 500-key probe stayed disabled; bridge closed after 123 server-to-client / 159 client-to-server bytes. Source confirms the mismatch: apps/tracer/src/terminal.ts startPings sends Ping on TERMINAL_STREAM, while crates/infraternal-server/src/websocket.rs handles client Ping on CONTROL_STREAM. Fixing the client stream now.
kayg commented 2026-10-07 01:43:04 +00:00
Author
Owner
Copy link

Finding: idle violates Decision 81. The production 4x iPhone 15 CDP trace records 600 FireAnimationFrame and 24 TimerFire events in 10 s; 1x records the same. Raw traces: /home/kayg/.cache/codex-scratch/review-perf/evidence/traces/idle-4x.json and idle-1x.json. Investigating the persistent renderer frame loop and timer sources; detailed before/after follows.

Finding: idle violates Decision 81. The production 4x iPhone 15 CDP trace records 600 FireAnimationFrame and 24 TimerFire events in 10 s; 1x records the same. Raw traces: /home/kayg/.cache/codex-scratch/review-perf/evidence/traces/idle-4x.json and idle-1x.json. Investigating the persistent renderer frame loop and timer sources; detailed before/after follows.
kayg commented 2026-10-07 01:44:38 +00:00
Author
Owner
Copy link

Pilot finding: cold FCP is above DECISIONS.md decision 41’s <1 s budget under both measured profiles. Real production static-server sample: iPhone 15, Chromium 149.0.7827.55; FCP 1,136 ms at 4x CPU and 1,136 ms at 1x (n=1 each, preliminary). firstPaint was 204/140 ms; JS EvaluateScript before first contentful paint totaled 1.922/0.242 ms. Traces: /home/kayg/.cache/codex-scratch/review-perf/evidence/traces/cold-4x.json and cold-1x.json. Running the required 30-sample set next.

Pilot finding: cold FCP is above DECISIONS.md decision 41’s <1 s budget under both measured profiles. Real production static-server sample: iPhone 15, Chromium 149.0.7827.55; FCP 1,136 ms at 4x CPU and 1,136 ms at 1x (n=1 each, preliminary). firstPaint was 204/140 ms; JS EvaluateScript before first contentful paint totaled 1.922/0.242 ms. Traces: /home/kayg/.cache/codex-scratch/review-perf/evidence/traces/cold-4x.json and cold-1x.json. Running the required 30-sample set next.
kayg commented 2026-10-07 01:49:27 +00:00
Author
Owner
Copy link

Idle-render step validated on the production server: the 10 s 4x trace now has 0 FireAnimationFrame, 0 RequestAnimationFrame, and 0 TimerFire events (before: 600, 600, 24). The real 500-key probe painted 500/500 at 4x (pilot p50 16.2 ms, p99 32.1 ms); local keydown→onData p99 was 1.8 ms and encode→send p99 3.1 ms. Trace: /home/kayg/.cache/codex-scratch/review-perf/evidence/traces/idle-4x.json; screenshot: /home/kayg/.cache/codex-scratch/review-perf/evidence/screens/terminal-4x.png. This is a pilot; I’m expanding each metric to the required sample size.

Idle-render step validated on the production server: the 10 s 4x trace now has 0 FireAnimationFrame, 0 RequestAnimationFrame, and 0 TimerFire events (before: 600, 600, 24). The real 500-key probe painted 500/500 at 4x (pilot p50 16.2 ms, p99 32.1 ms); local keydown→onData p99 was 1.8 ms and encode→send p99 3.1 ms. Trace: /home/kayg/.cache/codex-scratch/review-perf/evidence/traces/idle-4x.json; screenshot: /home/kayg/.cache/codex-scratch/review-perf/evidence/screens/terminal-4x.png. This is a pilot; I’m expanding each metric to the required sample size.
kayg commented 2026-10-07 01:51:34 +00:00
Author
Owner
Copy link

Cold-load baseline complete on the real static server, 30 fresh Chromium contexts per profile with tracing enabled: 4x first paint 172/204 ms p50/p99; FCP 1,276/1,372 ms; EvaluateScript before first paint 0.970/1.703 ms. 1x first paint 144/152 ms; FCP 1,152/1,180 ms; JS before first paint 0.251/0.583 ms. FCP misses decision 41’s 1 s budget at both rates while script work before paint is <2 ms; the Geist Mono font-face uses font-display:block, which blocks first text paint. All 60 raw traces: /home/kayg/.cache/codex-scratch/review-perf/evidence/traces/cold-4x-run-.json and cold-1x-run-.json. I’m measuring font-display:swap against this baseline.

Cold-load baseline complete on the real static server, 30 fresh Chromium contexts per profile with tracing enabled: 4x first paint 172/204 ms p50/p99; FCP 1,276/1,372 ms; EvaluateScript before first paint 0.970/1.703 ms. 1x first paint 144/152 ms; FCP 1,152/1,180 ms; JS before first paint 0.251/0.583 ms. FCP misses decision 41’s 1 s budget at both rates while script work before paint is <2 ms; the Geist Mono font-face uses font-display:block, which blocks first text paint. All 60 raw traces: /home/kayg/.cache/codex-scratch/review-perf/evidence/traces/cold-4x-run-*.json and cold-1x-run-*.json. I’m measuring font-display:swap against this baseline.
kayg commented 2026-10-07 01:54:07 +00:00
Author
Owner
Copy link

The font-display-only trial did not improve cold FCP (30-run medians: 4x 1,276→1,312 ms; 1x 1,152→1,256 ms, with a higher host load during the second pass). Trace/source follow-up found the actual critical-path wait: main.ts awaits both /auth/status and document.fonts.load (capped by an 800 ms timeout) before mounting any text; index.html otherwise has only an empty #app. I’m removing that font wait and keeping swap so the access screen can paint with fallback text while Geist loads.

The font-display-only trial did not improve cold FCP (30-run medians: 4x 1,276→1,312 ms; 1x 1,152→1,256 ms, with a higher host load during the second pass). Trace/source follow-up found the actual critical-path wait: main.ts awaits both /auth/status and document.fonts.load (capped by an 800 ms timeout) before mounting any text; index.html otherwise has only an empty #app. I’m removing that font wait and keeping swap so the access screen can paint with fallback text while Geist loads.
kayg commented 2026-10-07 01:58:09 +00:00
Author
Owner
Copy link

Cold-load experiment status: the final startup path still misses the 1 s FCP budget (baseline n=30 p50/p99: 4x 1,276/1,372 ms; 1x 1,152/1,180 ms). Removing the fontsReady wait plus font-display:swap did not demonstrate a gain; changed-build medians were 4x 1,492 ms and 1x 1,160 ms while host load rose from 1.54 to 9.84. I reverted those startup edits; no speculative FCP change remains in the branch. Baseline and trial summaries/traces remain under /home/kayg/.cache/codex-scratch/review-perf/evidence/.

Cold-load experiment status: the final startup path still misses the 1 s FCP budget (baseline n=30 p50/p99: 4x 1,276/1,372 ms; 1x 1,152/1,180 ms). Removing the fontsReady wait plus font-display:swap did not demonstrate a gain; changed-build medians were 4x 1,492 ms and 1x 1,160 ms while host load rose from 1.54 to 9.84. I reverted those startup edits; no speculative FCP change remains in the branch. Baseline and trial summaries/traces remain under /home/kayg/.cache/codex-scratch/review-perf/evidence/.
kayg commented 2026-10-07 02:00:17 +00:00
Author
Owner
Copy link

Finding: the mobile touch scroll did not move the terminal. On the real static-server build with 10,000 lines loaded, 30 dispatched iPhone 15 touch swipes left #terminal-host data-scrolled=false throughout; the initial 60 fps/no-long-task numbers were only the sampler loop and are not valid scroll-smoothness results. Source evidence: the pinned renderer listens for wheel events; the terminal screen/host are overflow:hidden with touch-action:pan-y, and the client has no touch-to-scroll handler. Replacing the scroll probe with a verified wheel baseline and checking a mobile swipe fix.

Finding: the mobile touch scroll did not move the terminal. On the real static-server build with 10,000 lines loaded, 30 dispatched iPhone 15 touch swipes left #terminal-host data-scrolled=false throughout; the initial 60 fps/no-long-task numbers were only the sampler loop and are not valid scroll-smoothness results. Source evidence: the pinned renderer listens for wheel events; the terminal screen/host are overflow:hidden with touch-action:pan-y, and the client has no touch-to-scroll handler. Replacing the scroll probe with a verified wheel baseline and checking a mobile swipe fix.
kayg commented 2026-10-07 02:06:54 +00:00
Author
Owner
Copy link

Finding: iPhone-profile touch swipes previously left the terminal at its bottom on 30/30 attempts because the canvas host had no scroll overflow and the client handled only wheel input. Added touch-driven Ghostty scrollback movement; a 4x production-server pilot changed scroll state on all 30 alternating swipes, with 1,070 sampled frame intervals (p50 16.7 ms, p99 33.4 ms), 0 long tasks at the median, and a 4-long-task p99 per swipe. Regression check passed on behemoth-prefix-82541.bin: swipe up entered scrollback, reverse swipe returned to bottom, and the canary painted. The full 30-run comparison follows.

Finding: iPhone-profile touch swipes previously left the terminal at its bottom on 30/30 attempts because the canvas host had no scroll overflow and the client handled only wheel input. Added touch-driven Ghostty scrollback movement; a 4x production-server pilot changed scroll state on all 30 alternating swipes, with 1,070 sampled frame intervals (p50 16.7 ms, p99 33.4 ms), 0 long tasks at the median, and a 4-long-task p99 per swipe. Regression check passed on behemoth-prefix-82541.bin: swipe up entered scrollback, reverse swipe returned to bottom, and the canary painted. The full 30-run comparison follows.
kayg commented 2026-10-07 03:00:10 +00:00
Author
Owner
Copy link

Finding: in the modified-build 1x journey, sample 2 attempt 1 lost the terminal WebSocket during the 10,000-line command. The completion marker timed out at 60s; bridge diagnostics recorded 728 closes (1005/1005) after 28,167 server-to-client bytes, with no server stderr. Attempt 2 passed. Evidence: evidence/journey-1x-final.log and evidence/journey-1x-progress.json in the review scratch directory; collecting protocol frame counts on subsequent diagnostics.

Finding: in the modified-build 1x journey, sample 2 attempt 1 lost the terminal WebSocket during the 10,000-line command. The completion marker timed out at 60s; bridge diagnostics recorded 728 closes (1005/1005) after 28,167 server-to-client bytes, with no server stderr. Attempt 2 passed. Evidence: evidence/journey-1x-final.log and evidence/journey-1x-progress.json in the review scratch directory; collecting protocol frame counts on subsequent diagnostics.
kayg commented 2026-10-07 03:01:27 +00:00
Author
Owner
Copy link

Update: the same 1x modified-build scroll failure recurred on sample 4 attempt 1: 690 WebSocket closes, 25,125 server-to-client bytes, no marker by 60s; the retry is running. This currently reproduces in 2 of the first 4 samples. Captured screenshots show the shell prompt, not completed seq output. Evidence remains in evidence/journey-1x-final.log and evidence/screens/10k-scroll-setup-failure.png.

Update: the same 1x modified-build scroll failure recurred on sample 4 attempt 1: 690 WebSocket closes, 25,125 server-to-client bytes, no marker by 60s; the retry is running. This currently reproduces in 2 of the first 4 samples. Captured screenshots show the shell prompt, not completed seq output. Evidence remains in evidence/journey-1x-final.log and evidence/screens/10k-scroll-setup-failure.png.
kayg commented 2026-10-07 03:03:38 +00:00
Author
Owner
Copy link

4x journey measurement complete on the real static server: 30 valid runs from 34 attempts (the earlier byte-threshold gate rejected 3 attempts; one attempt also showed a disconnect). Host: i9-9900K, 8 logical CPUs, 32 GiB RAM; Chromium 149.0.7827.55, iPhone 15 emulation at 393x659, 4x CPU throttle. Load average (1/5/15m) moved from 5.82/5.94/5.27 to 5.02/5.68/5.21.\n\n| Measure | p50 | p99 |\n| --- | ---: | ---: |\n| Setup: token to identity | 345 ms | 669 ms |\n| Setup: identity to passkey | 257 ms | 509 ms |\n| Setup: passkey to recovery | 392 ms | 971 ms |\n| Setup: recovery to welcome | 214 ms | 696 ms |\n| Setup to terminal attach | 3,746 ms | 4,361 ms |\n| Sign-in to terminal attach | 896 ms | 1,065 ms |\n| 500-key run p50 distribution | 11.0 ms | 16.8 ms |\n| 500-key run p99 distribution | 21.9 ms | 42.8 ms |\n| Forced reconnect to painted marker | 108 ms | 161 ms |\n| Palette open to paint | 124 ms | 167 ms |\n| Sidebar open to paint | 83 ms | 148 ms |\n| Theme switch to paint | 149 ms | 261 ms |\n| Scroll frame interval over 10k lines | 16.7 ms | 33.4 ms |\n| Dropped frames per swipe | 0 | 10 |\n| Long tasks per swipe | 0 | 3 |\n\nAll 900 alternating swipe checks changed scroll state. The 4x echo p99 exceeds Decision 41 5ms; reconnect p99 remains below 300ms. Scroll interval p99 reached 33.4ms. Full per-run details: evidence/diagnostics/journey-4x-final-summary.json; raw traces: evidence/traces/scroll-sample.json.

4x journey measurement complete on the real static server: 30 valid runs from 34 attempts (the earlier byte-threshold gate rejected 3 attempts; one attempt also showed a disconnect). Host: i9-9900K, 8 logical CPUs, 32 GiB RAM; Chromium 149.0.7827.55, iPhone 15 emulation at 393x659, 4x CPU throttle. Load average (1/5/15m) moved from 5.82/5.94/5.27 to 5.02/5.68/5.21.\n\n| Measure | p50 | p99 |\n| --- | ---: | ---: |\n| Setup: token to identity | 345 ms | 669 ms |\n| Setup: identity to passkey | 257 ms | 509 ms |\n| Setup: passkey to recovery | 392 ms | 971 ms |\n| Setup: recovery to welcome | 214 ms | 696 ms |\n| Setup to terminal attach | 3,746 ms | 4,361 ms |\n| Sign-in to terminal attach | 896 ms | 1,065 ms |\n| 500-key run p50 distribution | 11.0 ms | 16.8 ms |\n| 500-key run p99 distribution | 21.9 ms | 42.8 ms |\n| Forced reconnect to painted marker | 108 ms | 161 ms |\n| Palette open to paint | 124 ms | 167 ms |\n| Sidebar open to paint | 83 ms | 148 ms |\n| Theme switch to paint | 149 ms | 261 ms |\n| Scroll frame interval over 10k lines | 16.7 ms | 33.4 ms |\n| Dropped frames per swipe | 0 | 10 |\n| Long tasks per swipe | 0 | 3 |\n\nAll 900 alternating swipe checks changed scroll state. The 4x echo p99 exceeds Decision 41 5ms; reconnect p99 remains below 300ms. Scroll interval p99 reached 33.4ms. Full per-run details: evidence/diagnostics/journey-4x-final-summary.json; raw traces: evidence/traces/scroll-sample.json.
kayg commented 2026-10-07 03:03:56 +00:00
Author
Owner
Copy link

4x journey measurement complete on the real static server: 30 valid runs from 34 attempts. Three attempts were rejected by the earlier byte threshold; one also showed a disconnect. Host: i9-9900K, 8 logical CPUs, 32 GiB RAM; Chromium 149.0.7827.55, iPhone 15 emulation at 393x659, 4x CPU throttle. Load average (1/5/15m) moved from 5.82/5.94/5.27 to 5.02/5.68/5.21.

Measure p50 p99
Setup: token to identity 345 ms 669 ms
Setup: identity to passkey 257 ms 509 ms
Setup: passkey to recovery 392 ms 971 ms
Setup: recovery to welcome 214 ms 696 ms
Setup to terminal attach 3,746 ms 4,361 ms
Sign-in to terminal attach 896 ms 1,065 ms
500-key run p50 distribution 11.0 ms 16.8 ms
500-key run p99 distribution 21.9 ms 42.8 ms
Forced reconnect to painted marker 108 ms 161 ms
Palette open to paint 124 ms 167 ms
Sidebar open to paint 83 ms 148 ms
Theme switch to paint 149 ms 261 ms
Scroll frame interval over 10k lines 16.7 ms 33.4 ms
Dropped frames per swipe 0 10
Long tasks per swipe 0 3

All 900 alternating swipe checks changed scroll state. Decision 41: 4x echo p99 exceeds 5 ms; reconnect p99 remains below 300 ms. Scroll interval p99 reached 33.4 ms. Full per-run details: evidence/diagnostics/journey-4x-final-summary.json; raw scroll trace: evidence/traces/scroll-sample.json. Previous comment accidentally included literal backslash-n separators; this comment has the same measurements formatted as a table.

4x journey measurement complete on the real static server: 30 valid runs from 34 attempts. Three attempts were rejected by the earlier byte threshold; one also showed a disconnect. Host: i9-9900K, 8 logical CPUs, 32 GiB RAM; Chromium 149.0.7827.55, iPhone 15 emulation at 393x659, 4x CPU throttle. Load average (1/5/15m) moved from 5.82/5.94/5.27 to 5.02/5.68/5.21. | Measure | p50 | p99 | | --- | ---: | ---: | | Setup: token to identity | 345 ms | 669 ms | | Setup: identity to passkey | 257 ms | 509 ms | | Setup: passkey to recovery | 392 ms | 971 ms | | Setup: recovery to welcome | 214 ms | 696 ms | | Setup to terminal attach | 3,746 ms | 4,361 ms | | Sign-in to terminal attach | 896 ms | 1,065 ms | | 500-key run p50 distribution | 11.0 ms | 16.8 ms | | 500-key run p99 distribution | 21.9 ms | 42.8 ms | | Forced reconnect to painted marker | 108 ms | 161 ms | | Palette open to paint | 124 ms | 167 ms | | Sidebar open to paint | 83 ms | 148 ms | | Theme switch to paint | 149 ms | 261 ms | | Scroll frame interval over 10k lines | 16.7 ms | 33.4 ms | | Dropped frames per swipe | 0 | 10 | | Long tasks per swipe | 0 | 3 | All 900 alternating swipe checks changed scroll state. Decision 41: 4x echo p99 exceeds 5 ms; reconnect p99 remains below 300 ms. Scroll interval p99 reached 33.4 ms. Full per-run details: evidence/diagnostics/journey-4x-final-summary.json; raw scroll trace: evidence/traces/scroll-sample.json. Previous comment accidentally included literal backslash-n separators; this comment has the same measurements formatted as a table.
kayg commented 2026-10-07 03:18:21 +00:00
Author
Owner
Copy link

1x journey measurement complete on the real static server: 30 valid runs from 32 attempts. Two attempts timed out waiting for the 10k output marker during the early high-load part of the pass; both retries passed. Host: i9-9900K, 8 logical CPUs, 32 GiB RAM; Chromium 149.0.7827.55, iPhone 15 emulation at 393x659, 1x CPU. Load average (1/5/15m) moved from 1.74/4.09/4.82 to 2.44/2.19/3.30; the two rejects occurred while sampled 1m load was about 8.7.

Measure p50 p99
Setup: token to identity 196 ms 357 ms
Setup: identity to passkey 291 ms 361 ms
Setup: passkey to recovery 375 ms 492 ms
Setup: recovery to welcome 87 ms 146 ms
Setup to terminal attach 3,284 ms 3,469 ms
Sign-in to terminal attach 541 ms 743 ms
500-key run p50 distribution 15.1 ms 15.2 ms
500-key run p99 distribution 15.8 ms 28.5 ms
Forced reconnect to painted marker 96 ms 112 ms
Palette open to paint 148 ms 278 ms
Sidebar open to paint 42 ms 56 ms
Theme switch to paint 88 ms 106 ms
Scroll frame interval over 10k lines 16.7 ms 16.8 ms
Dropped frames per swipe 0 1
Long tasks per swipe 0 0

All 900 alternating swipe checks changed scroll state. Decision 41: 1x echo p99 exceeds 5 ms; reconnect p99 remains below 300 ms. Details: evidence/diagnostics/journey-1x-final-summary.json; failures: evidence/diagnostics/journey-1x-failed-run.log; scroll trace: evidence/traces/scroll-sample.json.

1x journey measurement complete on the real static server: 30 valid runs from 32 attempts. Two attempts timed out waiting for the 10k output marker during the early high-load part of the pass; both retries passed. Host: i9-9900K, 8 logical CPUs, 32 GiB RAM; Chromium 149.0.7827.55, iPhone 15 emulation at 393x659, 1x CPU. Load average (1/5/15m) moved from 1.74/4.09/4.82 to 2.44/2.19/3.30; the two rejects occurred while sampled 1m load was about 8.7. | Measure | p50 | p99 | | --- | ---: | ---: | | Setup: token to identity | 196 ms | 357 ms | | Setup: identity to passkey | 291 ms | 361 ms | | Setup: passkey to recovery | 375 ms | 492 ms | | Setup: recovery to welcome | 87 ms | 146 ms | | Setup to terminal attach | 3,284 ms | 3,469 ms | | Sign-in to terminal attach | 541 ms | 743 ms | | 500-key run p50 distribution | 15.1 ms | 15.2 ms | | 500-key run p99 distribution | 15.8 ms | 28.5 ms | | Forced reconnect to painted marker | 96 ms | 112 ms | | Palette open to paint | 148 ms | 278 ms | | Sidebar open to paint | 42 ms | 56 ms | | Theme switch to paint | 88 ms | 106 ms | | Scroll frame interval over 10k lines | 16.7 ms | 16.8 ms | | Dropped frames per swipe | 0 | 1 | | Long tasks per swipe | 0 | 0 | All 900 alternating swipe checks changed scroll state. Decision 41: 1x echo p99 exceeds 5 ms; reconnect p99 remains below 300 ms. Details: evidence/diagnostics/journey-1x-final-summary.json; failures: evidence/diagnostics/journey-1x-failed-run.log; scroll trace: evidence/traces/scroll-sample.json.
kayg commented 2026-10-07 03:19:05 +00:00
Author
Owner
Copy link

Idle change measurement complete on the production static-server build. The original build had 600 FireAnimationFrame, 600 RequestAnimationFrame and 24 TimerFire events in each 10s trace at both rates. The updated build has zero of those events at 1x and 4x. CrRendererMain RunTask time fell from 1,102.826 ms to 1.189 ms per 10s at 1x, and from 4,787.051 ms to 62.180 ms per 10s at 4x. That is a 99.9% and 98.7% reduction respectively; residual trace task time remains, so I’m reporting the zero scheduled-work result separately from a literal zero-CPU claim. The browser regression also observed zero pending rAF, timeout and interval handles after output settled.

Raw traces: evidence/traces/idle-before-1x.json, idle-before-4x.json, idle-1x.json, idle-4x.json. The updated 1x capture used the production Chromium 149.0.7827.55 iPhone 15 profile at 393x659; host load average was 2.30/2.17/3.28 before and 2.08/2.13/3.22 after.

Idle change measurement complete on the production static-server build. The original build had 600 FireAnimationFrame, 600 RequestAnimationFrame and 24 TimerFire events in each 10s trace at both rates. The updated build has zero of those events at 1x and 4x. CrRendererMain RunTask time fell from 1,102.826 ms to 1.189 ms per 10s at 1x, and from 4,787.051 ms to 62.180 ms per 10s at 4x. That is a 99.9% and 98.7% reduction respectively; residual trace task time remains, so I’m reporting the zero scheduled-work result separately from a literal zero-CPU claim. The browser regression also observed zero pending rAF, timeout and interval handles after output settled. Raw traces: evidence/traces/idle-before-1x.json, idle-before-4x.json, idle-1x.json, idle-4x.json. The updated 1x capture used the production Chromium 149.0.7827.55 iPhone 15 profile at 393x659; host load average was 2.30/2.17/3.28 before and 2.08/2.13/3.22 after.
kayg commented 2026-10-07 03:23:44 +00:00
Author
Owner
Copy link

Exact-base cold capture complete: 30 fresh Chromium contexts per rate, production static server, trace saved per run. This corrects my earlier comment that called FCP a Decision 41 budget miss: the decision says first paint, and first-paint p99 is below 1s at both rates. FCP remains above 1s as a separate contentful-paint result.

Base build metric 4x p50 4x p99 1x p50 1x p99
First Paint 168 ms 220 ms 140 ms 192 ms
FCP 1,268 ms 1,384 ms 1,148 ms 1,284 ms
EvaluateScript before First Paint 0.909 ms 3.820 ms 0.232 ms 0.862 ms
EvaluateScript before FCP 0.909 ms 3.820 ms 0.232 ms 0.862 ms

Host: i9-9900K, 8 logical CPUs, 32 GiB RAM; Chromium 149.0.7827.55. Load average during the 4x pass was 2.05/2.12/3.19 before and 2.92/2.35/3.21 after; during 1x it was 3.01/2.37/3.21 before and 3.64/2.60/3.25 after. Raw traces: evidence/traces/cold-base-recaptured-4x-run-.json and cold-base-recaptured-1x-run-.json. Summary: evidence/diagnostics/cold-base-recaptured-final-summary.json.

Exact-base cold capture complete: 30 fresh Chromium contexts per rate, production static server, trace saved per run. This corrects my earlier comment that called FCP a Decision 41 budget miss: the decision says first paint, and first-paint p99 is below 1s at both rates. FCP remains above 1s as a separate contentful-paint result. | Base build metric | 4x p50 | 4x p99 | 1x p50 | 1x p99 | | --- | ---: | ---: | ---: | ---: | | First Paint | 168 ms | 220 ms | 140 ms | 192 ms | | FCP | 1,268 ms | 1,384 ms | 1,148 ms | 1,284 ms | | EvaluateScript before First Paint | 0.909 ms | 3.820 ms | 0.232 ms | 0.862 ms | | EvaluateScript before FCP | 0.909 ms | 3.820 ms | 0.232 ms | 0.862 ms | Host: i9-9900K, 8 logical CPUs, 32 GiB RAM; Chromium 149.0.7827.55. Load average during the 4x pass was 2.05/2.12/3.19 before and 2.92/2.35/3.21 after; during 1x it was 3.01/2.37/3.21 before and 3.64/2.60/3.25 after. Raw traces: evidence/traces/cold-base-recaptured-4x-run-*.json and cold-base-recaptured-1x-run-*.json. Summary: evidence/diagnostics/cold-base-recaptured-final-summary.json.
kayg commented 2026-10-07 03:29:38 +00:00
Author
Owner
Copy link

Eight-session idle measurement (8 authenticated iframe sessions in one iPhone 15 tab; 30 one-second samples per rate) confirms a high renderer CPU cost at 4x throttling. /proc and Chromium SystemInfo.getProcessInfo.cpuTime agree: 83.5% p50 / 87.1% p99 at 4x, versus 2.0% / 5.9% at 1x. At 4x the renderer uses 264.8 MB JS heap, 58.7 MB WASM and 339.8/343.8 MB RSS p50/p99; at 1x it uses 249.6 MB JS, 58.7 MB WASM and 344.0/344.7 MB RSS. Server RSS is about 16.3 MB. Eight frame instrumentation observed zero pending rAFs, timeouts or intervals and zero rAF activity during the 30-second sample. The finding is therefore high CPU outside scheduled page callbacks; evidence: evidence/diagnostics/concurrent-4x-cpu-validation-summary.json, concurrent-1x-cpu-validation-summary.json, and evidence/traces/concurrent-idle-4x.json in the review scratch directory. I’m retaining this as an open ranked finding while completing the soak.

Eight-session idle measurement (8 authenticated iframe sessions in one iPhone 15 tab; 30 one-second samples per rate) confirms a high renderer CPU cost at 4x throttling. `/proc` and Chromium `SystemInfo.getProcessInfo.cpuTime` agree: 83.5% p50 / 87.1% p99 at 4x, versus 2.0% / 5.9% at 1x. At 4x the renderer uses 264.8 MB JS heap, 58.7 MB WASM and 339.8/343.8 MB RSS p50/p99; at 1x it uses 249.6 MB JS, 58.7 MB WASM and 344.0/344.7 MB RSS. Server RSS is about 16.3 MB. Eight frame instrumentation observed zero pending rAFs, timeouts or intervals and zero rAF activity during the 30-second sample. The finding is therefore high CPU outside scheduled page callbacks; evidence: `evidence/diagnostics/concurrent-4x-cpu-validation-summary.json`, `concurrent-1x-cpu-validation-summary.json`, and `evidence/traces/concurrent-idle-4x.json` in the review scratch directory. I’m retaining this as an open ranked finding while completing the soak.
kayg commented 2026-10-07 03:33:58 +00:00
Author
Owner
Copy link

Decision #41's keystroke-to-echo p99 target is <5 ms. The 30-run, 500-key sequential synthetic probe reports painted-echo latency p50-of-run-p99 / p99-of-run-p99 of 15.8 / 28.5 ms at 1x and 21.9 / 42.8 ms at 4x; the p50-of-run-p50 values are 15.1 and 11.0 ms. Thus the latency target is not met on this host/profile. The base deployed build could not provide a comparable 500-key sample because its Ping used the wrong mux stream and the terminal detached; the corrected build produced 30 valid samples. Full per-run data: evidence/diagnostics/journey-1x-final-summary.json and journey-4x-final-summary.json in scratch. Reconnect-to-painted-marker p99 is 112 ms at 1x and 161 ms at 4x, within #41's 300 ms budget.

Decision #41's keystroke-to-echo p99 target is <5 ms. The 30-run, 500-key sequential synthetic probe reports painted-echo latency p50-of-run-p99 / p99-of-run-p99 of 15.8 / 28.5 ms at 1x and 21.9 / 42.8 ms at 4x; the p50-of-run-p50 values are 15.1 and 11.0 ms. Thus the latency target is not met on this host/profile. The base deployed build could not provide a comparable 500-key sample because its Ping used the wrong mux stream and the terminal detached; the corrected build produced 30 valid samples. Full per-run data: `evidence/diagnostics/journey-1x-final-summary.json` and `journey-4x-final-summary.json` in scratch. Reconnect-to-painted-marker p99 is 112 ms at 1x and 161 ms at 4x, within #41's 300 ms budget.
kayg commented 2026-10-07 03:36:36 +00:00
Author
Owner
Copy link

Correction to the eight-session CPU finding above: the 4x raw renderer process CPU (83.5% p50 / 87.1% p99) is not a tracer CPU regression. Chromium's Linux/POSIX CPU-throttling implementation sends a signal to the renderer thread every 200 µs and busy-waits in that thread's signal handler to add the slowdown; that emulator work is included in both /proc and SystemInfo.cpuTime counters. Source: Chromium ThreadCPUThrottler. Treat the 4x process-CPU series as throttle overhead, not application CPU. At 1x, the same 30-second sample is 2.0% p50 / 5.9% p99 (including per-sample instrumentation). The app-side idle evidence remains: all eight frames had zero rAF/timer/interval activity, and the production 4x idle trace recorded 62 ms of renderer-main RunTask time in 10 s (1x: 1.2 ms in 10 s). I withdraw the earlier ranking of 4x process CPU as a product finding.

Correction to the eight-session CPU finding above: the 4x raw renderer process CPU (83.5% p50 / 87.1% p99) is not a tracer CPU regression. Chromium's Linux/POSIX CPU-throttling implementation sends a signal to the renderer thread every 200 µs and busy-waits in that thread's signal handler to add the slowdown; that emulator work is included in both `/proc` and `SystemInfo.cpuTime` counters. Source: [Chromium ThreadCPUThrottler](https://chromium.googlesource.com/chromium/src/+/b5e9a80d71482c9f173bee4ad3ac77ca39d3ab60/third_party/blink/renderer/platform/scheduler/common/thread_cpu_throttler.cc#139). Treat the 4x process-CPU series as throttle overhead, not application CPU. At 1x, the same 30-second sample is 2.0% p50 / 5.9% p99 (including per-sample instrumentation). The app-side idle evidence remains: all eight frames had zero rAF/timer/interval activity, and the production 4x idle trace recorded 62 ms of renderer-main RunTask time in 10 s (1x: 1.2 ms in 10 s). I withdraw the earlier ranking of 4x process CPU as a product finding.
kayg commented 2026-10-07 03:37:41 +00:00
Author
Owner
Copy link

Completed ping protocol step: commit ecb7cbb (fix(tracer): route liveness ping through control stream). The deployed client sent Ping on the terminal stream while the server accepts it on the control stream, so the client detached; the test server now rejects terminal-stream Ping. One ping on attach and reconnect replaces the two-second idle interval. The production build yielded 30 valid 500-key journey samples; reconnect-to-painted-marker p99 is 112 ms at 1x and 161 ms at 4x, below Decision #41's 300 ms budget. Evidence: evidence/diagnostics/journey-1x-final-summary.json, journey-4x-final-summary.json.

Completed ping protocol step: commit `ecb7cbb` (`fix(tracer): route liveness ping through control stream`). The deployed client sent Ping on the terminal stream while the server accepts it on the control stream, so the client detached; the test server now rejects terminal-stream Ping. One ping on attach and reconnect replaces the two-second idle interval. The production build yielded 30 valid 500-key journey samples; reconnect-to-painted-marker p99 is 112 ms at 1x and 161 ms at 4x, below Decision #41's 300 ms budget. Evidence: `evidence/diagnostics/journey-1x-final-summary.json`, `journey-4x-final-summary.json`.
kayg commented 2026-10-07 03:39:29 +00:00
Author
Owner
Copy link

Completed idle-rendering step: commit 8847828 (perf(tracer): paint terminal only on changes). The deployed 10 s traces had 600 animation-frame callbacks and 24 timer fires per rate; the fixed build has zero rAFs and timers at 1x and 4x. Renderer-main RunTask time changed 1,102.8 → 1.2 ms/10 s at 1x and 4,787.1 → 62.2 ms/10 s at 4x. The idle CDP test also asserts zero pending rAF, timeout and interval work after output settles. Decision #41/#81 evidence: evidence/traces/idle-before-{1x,4x}.json, idle-{1x,4x}.json, and evidence/diagnostics/idle-1x-post-pilot.log in scratch. Note: raw 4x process CPU is emulation overhead as corrected above; these renderer trace/task and schedule counters are app-side evidence.

Completed idle-rendering step: commit `8847828` (`perf(tracer): paint terminal only on changes`). The deployed 10 s traces had 600 animation-frame callbacks and 24 timer fires per rate; the fixed build has zero rAFs and timers at 1x and 4x. Renderer-main RunTask time changed 1,102.8 → 1.2 ms/10 s at 1x and 4,787.1 → 62.2 ms/10 s at 4x. The idle CDP test also asserts zero pending rAF, timeout and interval work after output settles. Decision #41/#81 evidence: `evidence/traces/idle-before-{1x,4x}.json`, `idle-{1x,4x}.json`, and `evidence/diagnostics/idle-1x-post-pilot.log` in scratch. Note: raw 4x process CPU is emulation overhead as corrected above; these renderer trace/task and schedule counters are app-side evidence.
kayg commented 2026-10-07 03:39:39 +00:00
Author
Owner
Copy link

Completed mobile-scroll step: commit 794a92a (fix(tracer): map mobile touch gestures to scrollback). Before the change, 0/30 iPhone-profile swipes moved the terminal viewport; after it, 900/900 swipes across 30 runs entered or left scrollback. At 1x/4x, frame-interval p50/p99 is 16.7/16.8 ms and 16.7/33.4 ms; dropped-frame p99 is 1 and 10 per swipe. Evidence: evidence/diagnostics/journey-1x-final-summary.json, journey-4x-final-summary.json, and evidence/traces/scroll-sample.json in scratch.

Completed mobile-scroll step: commit `794a92a` (`fix(tracer): map mobile touch gestures to scrollback`). Before the change, 0/30 iPhone-profile swipes moved the terminal viewport; after it, 900/900 swipes across 30 runs entered or left scrollback. At 1x/4x, frame-interval p50/p99 is 16.7/16.8 ms and 16.7/33.4 ms; dropped-frame p99 is 1 and 10 per swipe. Evidence: `evidence/diagnostics/journey-1x-final-summary.json`, `journey-4x-final-summary.json`, and `evidence/traces/scroll-sample.json` in scratch.
kayg commented 2026-10-07 03:49:58 +00:00
Author
Owner
Copy link

Finding: soak ls -R /usr closes the terminal stream before the scheduled network cut. In evidence/diagnostics/soak-reconnect-pilot-diagnostic.log, bridge 1 sent 61 Output frames / 42,624 bytes then received terminal Close/1013; three Resume attempts then each hit another Close/1013 (the third also control Close/1011). crates/infraternal-server/src/embedded.rs retires a stream when its four-item OUTPUT_CAPACITY try_send queue fills. This makes burst output fail to reconnect on the deployed build; I am testing a bounded queue/batching correction before repeating the soak.

Finding: soak ls -R /usr closes the terminal stream before the scheduled network cut. In evidence/diagnostics/soak-reconnect-pilot-diagnostic.log, bridge 1 sent 61 Output frames / 42,624 bytes then received terminal Close/1013; three Resume attempts then each hit another Close/1013 (the third also control Close/1011). crates/infraternal-server/src/embedded.rs retires a stream when its four-item OUTPUT_CAPACITY try_send queue fills. This makes burst output fail to reconnect on the deployed build; I am testing a bounded queue/batching correction before repeating the soak.
kayg commented 2026-10-07 03:55:05 +00:00
Author
Owner
Copy link

Step complete: 74c4fd6 raises the bounded attachment and WebSocket forwarding queues from 4 to 64 items. The focused server regression passed with the measured 61-frame / 42,624-byte burst; the x86_64 musl release server build also passed. Next: production tracer/browser burst probe, then the required 30-minute soak and final gates.

Step complete: 74c4fd6 raises the bounded attachment and WebSocket forwarding queues from 4 to 64 items. The focused server regression passed with the measured 61-frame / 42,624-byte burst; the x86_64 musl release server build also passed. Next: production tracer/browser burst probe, then the required 30-minute soak and final gates.
kayg commented 2026-10-07 04:00:13 +00:00
Author
Owner
Copy link

Update: the first 64-slot change is insufficient for the complete ls -R /usr burst. The real 4x probe still ended with terminal Close/1013 after 661 Output frames and 395,727 WebSocket bytes; three retries had previously repeated 1013. I am measuring a batch/coalesce path to reduce per-frame server authorization and sink work, then I will amend the queue regression to cover the observed 661-frame burst.

Update: the first 64-slot change is insufficient for the complete ls -R /usr burst. The real 4x probe still ended with terminal Close/1013 after 661 Output frames and 395,727 WebSocket bytes; three retries had previously repeated 1013. I am measuring a batch/coalesce path to reduce per-frame server authorization and sink work, then I will amend the queue regression to cover the observed 661-frame burst.
kayg commented 2026-10-07 04:09:55 +00:00
Author
Owner
Copy link

Step complete: 066a20a coalesces contiguous same-stream Output frames up to 64 KiB while preserving the single-frame path. On the 4x iPhone 15 production probe, ls -R /usr finished at 33.536 s; 1,016,842 WebSocket bytes and 1,340 Output frames crossed the initial and resumed bridges without 1013. The scheduled network cut at 90 s reattached in 107 ms. Summary: evidence/diagnostics/soak-batch-probe-summary.json; screenshot: evidence/screens/terminal-soak-end.png. Starting the 30-minute mixed-use soak now.

Step complete: 066a20a coalesces contiguous same-stream Output frames up to 64 KiB while preserving the single-frame path. On the 4x iPhone 15 production probe, ls -R /usr finished at 33.536 s; 1,016,842 WebSocket bytes and 1,340 Output frames crossed the initial and resumed bridges without 1013. The scheduled network cut at 90 s reattached in 107 ms. Summary: evidence/diagnostics/soak-batch-probe-summary.json; screenshot: evidence/screens/terminal-soak-end.png. Starting the 30-minute mixed-use soak now.
kayg commented 2026-10-07 04:25:32 +00:00
Author
Owner
Copy link

Finding during 30-minute soak: the forced cut at 600.35 s appeared to reattach in 273 ms, but the 650.24 s and 770.21 s samples both recorded connected=false. Memory stayed bounded (JS heap 3.63→4.26 MB; WASM 12.44 MiB; renderer RSS 184.7→198.4 MiB). The soak is running to its next reconnect checkpoint to capture close/frame diagnostics; I will fix and repeat if this is persistent.

Finding during 30-minute soak: the forced cut at 600.35 s appeared to reattach in 273 ms, but the 650.24 s and 770.21 s samples both recorded connected=false. Memory stayed bounded (JS heap 3.63→4.26 MB; WASM 12.44 MiB; renderer RSS 184.7→198.4 MiB). The soak is running to its next reconnect checkpoint to capture close/frame diagnostics; I will fix and repeat if this is persistent.
kayg commented 2026-10-07 04:36:17 +00:00
Author
Owner
Copy link

Reconnect finding from the 1,200 s checkpoint: 7,490 WebSocket attempts; all sockets were closed. Each of the last 10 sent Hello+Resume (plus AttachTerm/Credit/Resize) and received only the server Hello before close 1005. Evidence: evidence/diagnostics/soak-4x-reconnect-diagnostics.json and evidence/screens/terminal-soak-4x-reconnect-failure.png. + "TerminalClient.connect" + optimistically calls tryAttach onopen for resumable sessions, resetting retry backoff and sending AttachTerm before Resume is accepted. I am changing that ordering and adding a delayed-Welcome regression before repeating the soak.

Reconnect finding from the 1,200 s checkpoint: 7,490 WebSocket attempts; all sockets were closed. Each of the last 10 sent Hello+Resume (plus AttachTerm/Credit/Resize) and received only the server Hello before close 1005. Evidence: evidence/diagnostics/soak-4x-reconnect-diagnostics.json and evidence/screens/terminal-soak-4x-reconnect-failure.png. ` + "`TerminalClient.connect`" + ` optimistically calls tryAttach onopen for resumable sessions, resetting retry backoff and sending AttachTerm before Resume is accepted. I am changing that ordering and adding a delayed-Welcome regression before repeating the soak.
kayg commented 2026-10-07 04:43:14 +00:00
Author
Owner
Copy link

Reconnect ordering fix committed as c5b0074758. Resume no longer sends AttachTerm before Welcome; retry backoff resets after terminal Ack. The 800 ms delayed-Welcome browser regression passed: connected remained false, 0 premature terminal frames, then reattached. The prior run had 7,490 closed attempts after the 600 s reconnect; full soak rerun is pending.

Reconnect ordering fix committed as c5b0074758389879b7710470b3aca438b3b07da8. Resume no longer sends AttachTerm before Welcome; retry backoff resets after terminal Ack. The 800 ms delayed-Welcome browser regression passed: connected remained false, 0 premature terminal frames, then reattached. The prior run had 7,490 closed attempts after the 600 s reconnect; full soak rerun is pending.
kayg commented 2026-10-07 04:43:33 +00:00
Author
Owner
Copy link

Delayed-Welcome regression committed as 583de4e201. Focused browser test passed (1/1): with Welcome held 800 ms, connection stayed false, premature terminal frames remained 0, and resume completed. bun run check passed after making the browser timer wrapper IDs explicit.

Delayed-Welcome regression committed as 583de4e2013f00a3e14146175cfd8a9bdb558528. Focused browser test passed (1/1): with Welcome held 800 ms, connection stayed false, premature terminal frames remained 0, and resume completed. bun run check passed after making the browser timer wrapper IDs explicit.
kayg commented 2026-10-07 04:48:41 +00:00
Author
Owner
Copy link

Second 4x soak is currently invalid: terminal was connected at 30.24 s when ls -R /usr started, false by 40.17 s, and remained false at 230.07 s. Heap 3.16→4.03 MB, WASM 7.73 MB, renderer RSS 182.9→176.1 MB in that interval. Evidence: evidence/soak-progress.json and evidence/diagnostics/soak-4x-rerun.log. I am keeping the run through the 600 s checkpoint to capture bridge frame counts and close codes; cause not yet established.

Second 4x soak is currently invalid: terminal was connected at 30.24 s when `ls -R /usr` started, false by 40.17 s, and remained false at 230.07 s. Heap 3.16→4.03 MB, WASM 7.73 MB, renderer RSS 182.9→176.1 MB in that interval. Evidence: evidence/soak-progress.json and evidence/diagnostics/soak-4x-rerun.log. I am keeping the run through the 600 s checkpoint to capture bridge frame counts and close codes; cause not yet established.
kayg commented 2026-10-07 04:50:20 +00:00
Author
Owner
Copy link

Additional 230 s evidence: the current run SQLite database had exactly one websocket_connections row and it remained active=1 (queried without reading token fields). The browser socket was closed, so the server-side connection had not reached its inactive/resume cleanup; this is consistent with subsequent Resume being rejected before Welcome. Per-run DB: state-222204-soak-4x/data/infraternal.sqlite. Close/frame confirmation is pending at the harness checkpoint.

Additional 230 s evidence: the current run SQLite database had exactly one `websocket_connections` row and it remained `active=1` (queried without reading token fields). The browser socket was closed, so the server-side connection had not reached its inactive/resume cleanup; this is consistent with subsequent Resume being rejected before Welcome. Per-run DB: state-222204-soak-4x/data/infraternal.sqlite. Close/frame confirmation is pending at the harness checkpoint.
kayg commented 2026-10-07 04:56:54 +00:00
Author
Owner
Copy link

600 s checkpoint confirms a second issue. First bridge: 873 Output frames, 728,664 WebSocket bytes, terminal Close/1013 followed by control Close/1011; client had 13 Credit frames. Evidence: evidence/diagnostics/soak-4x-rerun.log and evidence/screens/terminal-soak-reconnect-precondition.png. The browser made only 2 bridges in 600 s (previous failed run: 7,490); the second had received Hello+Welcome, confirming the backoff fix. In crates/infraternal-server/src/embedded.rs, a full 64-event per-attachment output queue retires the stream; I am working on output batching/backpressure before another soak.

600 s checkpoint confirms a second issue. First bridge: 873 Output frames, 728,664 WebSocket bytes, terminal Close/1013 followed by control Close/1011; client had 13 Credit frames. Evidence: evidence/diagnostics/soak-4x-rerun.log and evidence/screens/terminal-soak-reconnect-precondition.png. The browser made only 2 bridges in 600 s (previous failed run: 7,490); the second had received Hello+Welcome, confirming the backoff fix. In crates/infraternal-server/src/embedded.rs, a full 64-event per-attachment output queue retires the stream; I am working on output batching/backpressure before another soak.
kayg commented 2026-10-07 05:02:13 +00:00
Author
Owner
Copy link

The embedded relay burst guard is committed as 25e2c70a1b; regression is separate at fb47f1e1c4. mobile_render_burst_fits_bounded_output_queue passed with 128 contiguous 16 KiB chunks (2 MiB); cargo fmt --all -- --check passed. Static x86_64-musl release build passed in 1m20s. Before measurement was 873 Output frames / 728,664 bytes then Close/1013; updated real-server probe is next.

The embedded relay burst guard is committed as 25e2c70a1b2a74d062f76212388217bab9fdeb83; regression is separate at fb47f1e1c49b9aec0fffe79497f7007086fed3e1. `mobile_render_burst_fits_bounded_output_queue` passed with 128 contiguous 16 KiB chunks (2 MiB); `cargo fmt --all -- --check` passed. Static x86_64-musl release build passed in 1m20s. Before measurement was 873 Output frames / 728,664 bytes then Close/1013; updated real-server probe is next.
kayg commented 2026-10-07 05:05:32 +00:00
Author
Owner
Copy link

The 256-event queue improves the real production run: ls -R /usr completed in 33.544 s with 1,014,820 WebSocket bytes and 1,181 Output frames on one bridge; the bridge had no close and all 15 samples through 150 s remained connected. Before, the first bridge closed at 40.17 s after 873 frames / 728,664 bytes with Close/1013. After summary: evidence/diagnostics/soak-outputqueue-150s-summary.json; progress: evidence/diagnostics/soak-outputqueue-150s-progress.json. Starting the full soak with reconnects at 600 and 1,200 s.

The 256-event queue improves the real production run: `ls -R /usr` completed in 33.544 s with 1,014,820 WebSocket bytes and 1,181 Output frames on one bridge; the bridge had no close and all 15 samples through 150 s remained connected. Before, the first bridge closed at 40.17 s after 873 frames / 728,664 bytes with Close/1013. After summary: evidence/diagnostics/soak-outputqueue-150s-summary.json; progress: evidence/diagnostics/soak-outputqueue-150s-progress.json. Starting the full soak with reconnects at 600 and 1,200 s.
kayg commented 2026-10-07 05:17:55 +00:00
Author
Owner
Copy link

Full soak reconnect checkpoint at 600.5 s completed in 172 ms; terminal was connected at 600.673 s. It was disconnected again by the 650.132 s sample (heap 4.61→4.60 MB, WASM 13.04 MB, renderer RSS 181.9→181.9 MB). This is separate from the initial output-queue failure, which the 256-frame buffer prevented through 650 s. Evidence: evidence/soak-progress.json and evidence/diagnostics/soak-4x-final-run.log. I am continuing to the 1,200 s checkpoint to capture the later bridge close.

Full soak reconnect checkpoint at 600.5 s completed in 172 ms; terminal was connected at 600.673 s. It was disconnected again by the 650.132 s sample (heap 4.61→4.60 MB, WASM 13.04 MB, renderer RSS 181.9→181.9 MB). This is separate from the initial output-queue failure, which the 256-frame buffer prevented through 650 s. Evidence: evidence/soak-progress.json and evidence/diagnostics/soak-4x-final-run.log. I am continuing to the 1,200 s checkpoint to capture the later bridge close.
kayg commented 2026-10-07 05:27:40 +00:00
Author
Owner
Copy link

1,200 s reconnect diagnostic identifies the resume failure: 703 browser bridges; each recent attempt sent Hello+Resume then AttachTerm/Credit/Resize/Input, received Hello+Welcome, then the server closed with control Close/1011. Evidence: evidence/diagnostics/soak-4x-final-run.log, evidence/screens/terminal-soak-reconnect-precondition.png. TerminalClient.tryAttach puts nextInSeq in AttachTerm.lastInSeq, while the server rejects values above persisted input_acked; queued keystrokes therefore make each resume fail. I am switching the attach cursor to ackedInSeq and adding a queued-input resume regression.

1,200 s reconnect diagnostic identifies the resume failure: 703 browser bridges; each recent attempt sent Hello+Resume then AttachTerm/Credit/Resize/Input, received Hello+Welcome, then the server closed with control Close/1011. Evidence: evidence/diagnostics/soak-4x-final-run.log, evidence/screens/terminal-soak-reconnect-precondition.png. `TerminalClient.tryAttach` puts `nextInSeq` in AttachTerm.lastInSeq, while the server rejects values above persisted input_acked; queued keystrokes therefore make each resume fail. I am switching the attach cursor to ackedInSeq and adding a queued-input resume regression.
kayg commented 2026-10-07 05:36:01 +00:00
Author
Owner
Copy link

Queued-input resume fix committed as 0f2b68e3d9; focused mock regression committed separately as a9f77c8386. bun run check passed and test/resume-input.spec.ts passed 1/1: while Welcome was held 2 s, the queued z stayed unsent, then replayed once after attach (accepted bytes 1→2, echo a→az, 0 premature terminal frames). Production soak is being repeated after this fix.

Queued-input resume fix committed as 0f2b68e3d94301f0649d80716b2cdef322ded91e; focused mock regression committed separately as a9f77c8386be1d169b6e86fe6c8c13337bc244e. `bun run check` passed and `test/resume-input.spec.ts` passed 1/1: while Welcome was held 2 s, the queued `z` stayed unsent, then replayed once after attach (accepted bytes 1→2, echo `a`→`az`, 0 premature terminal frames). Production soak is being repeated after this fix.
kayg commented 2026-10-07 05:40:36 +00:00
Author
Owner
Copy link

Production cursor-fix probe passed: 180 s iPhone 15 / 4x run, with keystrokes deliberately queued offline at 60 and 120 s. Reattach-to-Input+Output took 205 ms and 252 ms; 3 bridges total, 1,346 Output frames / 1,017,476 WebSocket bytes, no Close/1011, and ls -R /usr completed in 33.453 s. Summary: evidence/diagnostics/soak-queued-resume-180s-summary.json; progress: evidence/diagnostics/soak-queued-resume-180s-progress.json. Before the cursor fix the 1,200 s trace showed 703 bridges, each recent attempt ending Close/1011. Starting the final 30-minute soak with queued-input reconnect checkpoints.

Production cursor-fix probe passed: 180 s iPhone 15 / 4x run, with keystrokes deliberately queued offline at 60 and 120 s. Reattach-to-Input+Output took 205 ms and 252 ms; 3 bridges total, 1,346 Output frames / 1,017,476 WebSocket bytes, no Close/1011, and `ls -R /usr` completed in 33.453 s. Summary: evidence/diagnostics/soak-queued-resume-180s-summary.json; progress: evidence/diagnostics/soak-queued-resume-180s-progress.json. Before the cursor fix the 1,200 s trace showed 703 bridges, each recent attempt ending Close/1011. Starting the final 30-minute soak with queued-input reconnect checkpoints.
kayg commented 2026-10-07 05:53:08 +00:00
Author
Owner
Copy link

Finding: the final 4x soak lost the terminal during the 600s queued-input reconnect. After 590s it had 60 memory samples; reconnect timed out at 30s with the UI still disconnected and the last three bridges closing 1005. Evidence: evidence/diagnostics/soak-4x-complete.log and evidence/screens/terminal-soak-reconnect-failure.png in the review-perf scratch directory. Investigating before ranking this as a product defect.

Finding: the final 4x soak lost the terminal during the 600s queued-input reconnect. After 590s it had 60 memory samples; reconnect timed out at 30s with the UI still disconnected and the last three bridges closing 1005. Evidence: evidence/diagnostics/soak-4x-complete.log and evidence/screens/terminal-soak-reconnect-failure.png in the review-perf scratch directory. Investigating before ranking this as a product defect.
kayg commented 2026-10-07 06:08:44 +00:00
Author
Owner
Copy link

Finding: in the standard 4x soak, a forced reconnect returned in 101ms at 600s, then the terminal reported disconnected in every sample from 620–710s. The 600s theme rebuild ran immediately before the reconnect; this run is not a valid 30-minute soak. Evidence: evidence/diagnostics/soak-4x-standard-complete.log and evidence/diagnostics/soak-4x-theme-reconnect-failure-progress.json. I’m separating the two actions and rerunning to isolate the cause.

Finding: in the standard 4x soak, a forced reconnect returned in 101ms at 600s, then the terminal reported disconnected in every sample from 620–710s. The 600s theme rebuild ran immediately before the reconnect; this run is not a valid 30-minute soak. Evidence: evidence/diagnostics/soak-4x-standard-complete.log and evidence/diagnostics/soak-4x-theme-reconnect-failure-progress.json. I’m separating the two actions and rerunning to isolate the cause.
kayg commented 2026-10-07 06:23:30 +00:00
Author
Owner
Copy link

Finding confirmed: the isolated 4x soak (theme actions kept 35s away) also failed at 600s. Reconnect initially appeared up, then the terminal disconnected; the server emitted control Close/1011 after Hello+Welcome and the client AttachTerm, with no terminal Ack. Evidence: evidence/diagnostics/soak-4x-isolated-reconnect.log and evidence/soak-disconnect-progress.json. This is an attach rejection requiring a fix; investigating server/agent state before rerunning the 30m soak.

Finding confirmed: the isolated 4x soak (theme actions kept 35s away) also failed at 600s. Reconnect initially appeared up, then the terminal disconnected; the server emitted control Close/1011 after Hello+Welcome and the client AttachTerm, with no terminal Ack. Evidence: evidence/diagnostics/soak-4x-isolated-reconnect.log and evidence/soak-disconnect-progress.json. This is an attach rejection requiring a fix; investigating server/agent state before rerunning the 30m soak.
kayg commented 2026-10-07 06:42:00 +00:00
Author
Owner
Copy link

Root cause: after 600s online, the tracer expired the resume credential 120s after Welcome, although the server starts that window on disconnect. It opened a fresh writer (cursor 0) while AttachTerm sent the prior durable cursor 85; server rejected it with Close/1011. Cursor and aggregate DB evidence are in evidence/soak-disconnect-progress.json (no raw DB retained). Fix in progress: keep credentials live while attached and start the local deadline on socket close; bun run check passes.

Root cause: after 600s online, the tracer expired the resume credential 120s after Welcome, although the server starts that window on disconnect. It opened a fresh writer (cursor 0) while AttachTerm sent the prior durable cursor 85; server rejected it with Close/1011. Cursor and aggregate DB evidence are in evidence/soak-disconnect-progress.json (no raw DB retained). Fix in progress: keep credentials live while attached and start the local deadline on socket close; bun run check passes.
kayg commented 2026-10-07 06:43:05 +00:00
Author
Owner
Copy link

Step complete: 17adc5a fixes resume expiry to start when the socket closes, matching the server's 120s disconnect window. At 600s online, diagnostics showed AttachTerm.lastInSeq=85 against a fresh writer cursor 0, followed by control Close/1011; bun run check passed. Added the long-lived connection regression next; it is not committed yet.

Step complete: 17adc5a fixes resume expiry to start when the socket closes, matching the server's 120s disconnect window. At 600s online, diagnostics showed AttachTerm.lastInSeq=85 against a fresh writer cursor 0, followed by control Close/1011; bun run check passed. Added the long-lived connection regression next; it is not committed yet.
kayg commented 2026-10-07 06:44:00 +00:00
Author
Owner
Copy link

Step complete: c65cae6 adds deterministic coverage by advancing the browser clock 180s while connected, then forcing a delayed-Welcome resume with queued input. passed; focused Playwright passed 1/1 in 5.7s on TRACER_TEST_PORT=4481. Source fix is 17adc5a; next is a fresh production 30m soak.

Step complete: c65cae6 adds deterministic coverage by advancing the browser clock 180s while connected, then forcing a delayed-Welcome resume with queued input. passed; focused Playwright passed 1/1 in 5.7s on TRACER_TEST_PORT=4481. Source fix is 17adc5a; next is a fresh production 30m soak.
kayg commented 2026-10-07 06:44:08 +00:00
Author
Owner
Copy link

Step complete: c65cae6 adds deterministic coverage by advancing the browser clock 180s while connected, then forcing a delayed-Welcome resume with queued input. bun run check passed; focused Playwright passed 1/1 in 5.7s on TRACER_TEST_PORT=4481. Source fix is 17adc5a; next is a fresh production 30m soak.

Step complete: c65cae6 adds deterministic coverage by advancing the browser clock 180s while connected, then forcing a delayed-Welcome resume with queued input. bun run check passed; focused Playwright passed 1/1 in 5.7s on TRACER_TEST_PORT=4481. Source fix is 17adc5a; next is a fresh production 30m soak.
kayg commented 2026-10-07 07:16:17 +00:00
Author
Owner
Copy link

Step complete: final 4x production soak ran 1,800s with 180 ten-second samples; all samples stayed connected. Forced reconnects at 600/1,200s completed in 105/103ms and remained attached for 35s. ls -R /usr finished in 33.488s; htop ran 120s. JS heap p50/p99 4.17/4.96MB; WASM reached 12.78MB by 240s and stayed flat thereafter; renderer RSS p50/p99 179.7/188.3MB (first 210.4MB, last 184.4MB). Evidence: evidence/soak-summary.json, evidence/soak-progress.json, evidence/diagnostics/soak-4x-final.log.

Step complete: final 4x production soak ran 1,800s with 180 ten-second samples; all samples stayed connected. Forced reconnects at 600/1,200s completed in 105/103ms and remained attached for 35s. ls -R /usr finished in 33.488s; htop ran 120s. JS heap p50/p99 4.17/4.96MB; WASM reached 12.78MB by 240s and stayed flat thereafter; renderer RSS p50/p99 179.7/188.3MB (first 210.4MB, last 184.4MB). Evidence: evidence/soak-summary.json, evidence/soak-progress.json, evidence/diagnostics/soak-4x-final.log.
kayg commented 2026-10-07 07:42:20 +00:00
Author
Owner
Copy link

Completed paint scheduling step: commit 967d278 (perf(tracer): draw output within its paint frame). The synchronous renderer draw in the WebSocket task delayed Ghostty's next-frame write callback. Moving the draw into that callback changed the focused 500-key probe at 1x from p50/p99 18.2/48.4 ms to 15.4/19.7 ms; term.write → callback p99 changed 39.3 → 17.9 ms. The existing <20 ms browser assertion now passes at 1x. A 4x spot measured p99 20.3 ms; Decision #41's stricter 5 ms echo target remains unmet.

A 30-run repeated mobile-size sample had p50/p99-of-run-p99 31.7/66 ms while host load varied from 6 to 15 on 8 CPUs, so I retain it as noisy context rather than a stable after comparison: evidence/diagnostics/key-after-mobile-1x.log. Focused 1x and 4x logs: playwright-terminal-render-in-raf.log and playwright-terminal-render-in-raf-4x.log in the same diagnostics directory.

Completed paint scheduling step: commit `967d278` (`perf(tracer): draw output within its paint frame`). The synchronous renderer draw in the WebSocket task delayed Ghostty's next-frame write callback. Moving the draw into that callback changed the focused 500-key probe at 1x from p50/p99 18.2/48.4 ms to 15.4/19.7 ms; `term.write → callback` p99 changed 39.3 → 17.9 ms. The existing `<20 ms` browser assertion now passes at 1x. A 4x spot measured p99 20.3 ms; Decision #41's stricter 5 ms echo target remains unmet. A 30-run repeated mobile-size sample had p50/p99-of-run-p99 31.7/66 ms while host load varied from 6 to 15 on 8 CPUs, so I retain it as noisy context rather than a stable after comparison: `evidence/diagnostics/key-after-mobile-1x.log`. Focused 1x and 4x logs: `playwright-terminal-render-in-raf.log` and `playwright-terminal-render-in-raf-4x.log` in the same diagnostics directory.
kayg commented 2026-10-07 07:50:54 +00:00
Author
Owner
Copy link

Complete: deployed mobile performance review and fix-perf

Branch fix-perf, based on 7d2d0a535fe315fb010da133d665c68458f23162; final HEAD 967d278116bfdc43148c624ab54ae4669c4fd39e. Source worktree was read-only and remains unchanged. No deploy or merge.

Machine and load

Intel i9-9900K @ 3.60 GHz, 8 logical CPUs, 32 GiB RAM, Linux 6.12.111, Chromium 149.0.7827.55. Mobile measurements used iPhone 15, 393×659 CSS px, at 1x and 4x CPU throttle. Cold capture load (1/5/15m) ranged 2.05/2.12/3.19 to 3.64/2.60/3.25; journey passes ranged 1.74/4.09/4.82 to 5.82/5.94/5.27. The final soak began at 3.85/4.84/6.49 and ended at 1.93/3.27/4.07.

Cold production build, 30 runs per rate

Measure 1x p50 / p99 4x p50 / p99
First paint 140 / 192 ms 168 / 220 ms
FCP 1,148 / 1,284 ms 1,268 / 1,384 ms
JS EvaluateScript before first paint 0.232 / 0.862 ms 0.909 / 3.820 ms

Decision #41 first-paint p99 <1s passes at both rates. FCP is shown separately from that first-paint budget. Traces: evidence/traces/cold-base-recaptured-{1x,4x}-run-*.json; summary: evidence/diagnostics/cold-base-recaptured-final-summary.json.

Production journey, 30 valid runs per rate

Setup journey attempts: 30/32 valid at 1x and 30/34 at 4x. Scroll metrics aggregate 900 swipes per rate. Values below are p50 / p99 in milliseconds unless noted.

Measure 1x p50 / p99 4x p50 / p99
Setup: token → identity 196 / 357 345 / 669
Setup: identity → passkey 291 / 361 257 / 509
Setup: passkey → recovery 375 / 492 392 / 971
Setup: recovery → welcome 87 / 146 214 / 696
Setup → terminal attach 3,284 / 3,469 3,746 / 4,361
Sign-in → terminal attach 541 / 743 896 / 1,065
500-key run p50, distribution across run p50s 15.1 / 15.2 11.0 / 16.8
500-key run p99, distribution across run p99s 15.8 / 28.5 21.9 / 42.8
Reconnect → painted marker 96 / 112 108 / 161
Palette open → paint 148 / 278 124 / 167
Sidebar open → paint 42 / 56 83 / 148
Theme switch → paint 88 / 106 149 / 261
Scroll frame interval 16.7 / 16.8 16.7 / 33.4
Dropped frames per swipe, p50 / p99 0 / 1 0 / 10
Long tasks per swipe, p50 / p99 0 / 0 0 / 3

All 900 swipes changed scroll state after the mobile-scroll fix. Decision #41 reconnect p99 <300ms passes. The original <5ms key-echo p99 budget remains missed. Journey summaries: evidence/diagnostics/journey-{1x,4x}-final-summary.json; scroll trace: evidence/traces/scroll-sample.json.

Eight idle sessions in one tab, 30 one-second samples per rate

Totals are for all eight sessions. Memory is MiB; CPU is percent.

Measure 1x p50 / p99 4x p50 / p99
JS heap 249.6 / 249.6 264.8 / 264.8
WASM 58.7 / 58.7 58.7 / 58.7
Renderer RSS 344.0 / 344.7 339.8 / 343.8
Server RSS 16.8 / 16.8 16.3 / 16.3
Browser CPU 1.0 / 1.0 1.0 / 1.9
Renderer process CPU 2.0 / 5.9 83.5 / 87.1*

All eight frames had zero pending rAFs, timeouts, or intervals and zero scheduled work in the 30-second observation. *Chromium's Linux CPU throttler busy-waits in a renderer signal handler, so raw 4x process CPU is emulation overhead; it is not tracer CPU. Evidence: evidence/diagnostics/concurrent-{1x,4x}-cpu-validation-summary.json, evidence/traces/concurrent-idle-{1x,4x}.json.

Idle trace and 30-minute mixed-use soak

The deployed idle trace had 600 animation-frame callbacks and 24 timer fires in each 10-second capture. After the idle-render change those counts are zero at both rates. RendererMain RunTask fell from 1,102.826 → 1.189 ms/10s at 1x and 4,787.051 → 62.180 ms/10s at 4x (99.9% / 98.7% reduction).

The 1,800-second soak ran on c65cae669377ea31873e23ff10d613f57b2fde1e, immediately before the final frame-timing commit. It completed 180 ten-second samples with 0 disconnected samples; ls -R /usr completed in 33.488s, htop ran 120s, and forced reconnects at about 600s and 1,200s completed in 105ms and 103ms and stayed attached for 35s.

Soak measure p50 / p99 First → last
JS heap 4.17 / 4.96 MB 4.79 → 4.98 MB (+0.19 MB)
WASM memory 12.78 / 12.78 MB 7.34 → 12.78 MB, then flat after minute 4
Renderer RSS 179.7 / 188.3 MB 210.4 → 184.4 MB
Server RSS 14.93 / 15.00 MB 14.29 → 15.00 MB
Browser CPU at 4x 0 / 4.65% —
Renderer process CPU at 4x* 81.3 / 95.3% —

No sustained heap, WASM, or renderer-RSS leak was observed; WASM grew during warmup and then stayed flat. *The 4x renderer CPU includes Chromium throttler overhead described above. Soak summary and samples: evidence/soak-summary.json, evidence/soak-progress.json; trace/log: evidence/diagnostics/soak-4x-final.log. Idle traces: evidence/traces/idle-before-{1x,4x}.json and idle-{1x,4x}.json.

Ranked findings

  1. Painted keystroke latency still exceeds Decision #41's 5ms p99 target. The 30-run production journey was 28.5ms p99-of-run-p99 at 1x and 42.8ms at 4x. Moving the explicit draw into Ghostty's already-scheduled frame reduced the focused 500-key run from 18.2/48.4ms p50/p99 to 15.4/19.7ms; the final full suite measured 15.6/16.2ms and passed its <20ms assertion. A 4x spot measured 20.3ms. The strict 5ms target remains open. Focused logs: evidence/diagnostics/playwright-terminal-render-in-raf{,-4x}.log; final suite: evidence/diagnostics/playwright-full-final.log.
  2. Scrolling 10k lines has visible 4x jank. Frame interval p99 is 33.4ms, dropped-frame p99 is 10 per swipe, and long-task p99 is 3 per swipe. At 1x, interval p99 is 16.8ms and long-task p99 is zero.
  3. Theme rebuild is the slowest measured mobile interaction. At 4x its p99 is 261ms; palette and sidebar p99 are 167ms and 148ms. No budget for these transitions is stated in Decisions #41/#81.
  4. Eight open sessions consume about 340MB renderer RSS. This was stable during the idle sample; the 30-minute single-session soak showed no continuing renderer-RSS growth.

Fixes and before/after evidence

Change Before → after measurement
Stop idle Ghostty frame loop (8847828) 600 rAF callbacks + 24 timer fires/10s → zero; RendererMain RunTask 1,102.826 → 1.189ms/10s at 1x and 4,787.051 → 62.180ms/10s at 4x.
Restore mobile touch scroll (794a92a) 0/30 swipes moved scrollback → 900/900 swipe checks changed scroll state.
Buffer/coalesce terminal bursts (74c4fd6, 066a20a, 25e2c70) ls -R /usr closed with 1013 around 40.17s → 1,014,820 WebSocket bytes delivered and command completed in 33.488s.
Route liveness Ping on the control stream (ecb7cbb) Terminal-stream Ping detached the client and a two-second idle timer kept firing → one Ping on attach/reconnect, zero idle timers, reconnect p99 112ms/161ms at 1x/4x.
Start resume expiry at disconnect and replay acknowledged input (0f2b68e, 17adc5a; regressions a9f77c8, c65cae6) At a 600s live session the client resumed from cursor 0 while acknowledged input was 85 and got Close 1011 → credentials remain eligible while connected; final 30-minute soak had no disconnects and reconnects at 105/103ms.
Draw in the write callback frame (967d278) Focused 500-key p50/p99 18.2/48.4ms → 15.4/19.7ms; term.write-to-callback p99 39.3 → 17.9ms.

A repeated same-tab 30-run mobile-sized supplement is retained at evidence/diagnostics/key-after-mobile-1x.log; host load varied from 6–15 on 8 cores, so its tail is noisy and is not used as a stable before/after claim.

Final gates

scripts/verify fast passed. Verbatim gate summary:

advisories ok, bans ok, licenses ok, sources ok

==> static builds
    Finished `release` profile [optimized] target(s) in 0.40s
target/x86_64-unknown-linux-musl/release/infraternal: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, stripped
PT_INTERP: absent; size: 15078912 bytes
    Finished `release` profile [optimized] target(s) in 0.34s
target/aarch64-unknown-linux-musl/release/infraternal: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, stripped
PT_INTERP: absent; size: 12642000 bytes
server subcommand: exit 2, not implemented
agent subcommand: exit 2, not implemented

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+

==> Vendored Ghostty artifact checksums
dist/ghostty-web.js: OK
dist/ghostty-vt.wasm: OK
dist/index.d.ts: OK

==> TypeScript protocol compiler

==> Rust stream oracle (for the differential test)

==> TypeScript protocol tests
bun test v1.4.2 (744846f84)

src/codec.test.ts:
(pass) wire protocol golden vectors > encodes and decodes each frame exactly as the shared manifest describes [5.23ms]
(pass) wire protocol golden vectors > returns borrowed payload views and writes into reusable buffers [0.28ms]
(pass) wire protocol golden vectors > decodes muxed messages split across WebSocket messages [1.13ms]
(pass) wire protocol golden vectors > rejects oversized, non-canonical, unknown, and malformed frame bytes [0.35ms]
(pass) split mux header preserves trailing Close [0.34ms]

src/stream.test.ts:
stream differential: 134 streams, 11439 schedules, zero divergences
(pass) Rust and TypeScript preserve every frame at all boundaries and random multi-splits [2637.58ms]
(pass) maximum frame size and fragmented oversized prefix obey the allocation boundary [7.11ms]

 7 pass
 0 fail
 34390 expect() calls
Ran 7 tests across 2 files. [2.66s]

Full Playwright suite on TRACER_TEST_PORT=4482:

21 passed (2.3m)

Full fast-gate log: evidence/diagnostics/verify-fast-final.log. Trace and log directory: /home/kayg/.cache/codex-scratch/review-perf/evidence/.

## Complete: deployed mobile performance review and `fix-perf` Branch `fix-perf`, based on `7d2d0a535fe315fb010da133d665c68458f23162`; final HEAD `967d278116bfdc43148c624ab54ae4669c4fd39e`. Source worktree was read-only and remains unchanged. No deploy or merge. ### Machine and load Intel i9-9900K @ 3.60 GHz, 8 logical CPUs, 32 GiB RAM, Linux 6.12.111, Chromium 149.0.7827.55. Mobile measurements used iPhone 15, 393×659 CSS px, at 1x and 4x CPU throttle. Cold capture load (1/5/15m) ranged 2.05/2.12/3.19 to 3.64/2.60/3.25; journey passes ranged 1.74/4.09/4.82 to 5.82/5.94/5.27. The final soak began at 3.85/4.84/6.49 and ended at 1.93/3.27/4.07. ### Cold production build, 30 runs per rate | Measure | 1x p50 / p99 | 4x p50 / p99 | | --- | ---: | ---: | | First paint | 140 / 192 ms | 168 / 220 ms | | FCP | 1,148 / 1,284 ms | 1,268 / 1,384 ms | | JS EvaluateScript before first paint | 0.232 / 0.862 ms | 0.909 / 3.820 ms | Decision #41 first-paint p99 <1s passes at both rates. FCP is shown separately from that first-paint budget. Traces: `evidence/traces/cold-base-recaptured-{1x,4x}-run-*.json`; summary: `evidence/diagnostics/cold-base-recaptured-final-summary.json`. ### Production journey, 30 valid runs per rate Setup journey attempts: 30/32 valid at 1x and 30/34 at 4x. Scroll metrics aggregate 900 swipes per rate. Values below are p50 / p99 in milliseconds unless noted. | Measure | 1x p50 / p99 | 4x p50 / p99 | | --- | ---: | ---: | | Setup: token → identity | 196 / 357 | 345 / 669 | | Setup: identity → passkey | 291 / 361 | 257 / 509 | | Setup: passkey → recovery | 375 / 492 | 392 / 971 | | Setup: recovery → welcome | 87 / 146 | 214 / 696 | | Setup → terminal attach | 3,284 / 3,469 | 3,746 / 4,361 | | Sign-in → terminal attach | 541 / 743 | 896 / 1,065 | | 500-key run p50, distribution across run p50s | 15.1 / 15.2 | 11.0 / 16.8 | | 500-key run p99, distribution across run p99s | 15.8 / 28.5 | 21.9 / 42.8 | | Reconnect → painted marker | 96 / 112 | 108 / 161 | | Palette open → paint | 148 / 278 | 124 / 167 | | Sidebar open → paint | 42 / 56 | 83 / 148 | | Theme switch → paint | 88 / 106 | 149 / 261 | | Scroll frame interval | 16.7 / 16.8 | 16.7 / 33.4 | | Dropped frames per swipe, p50 / p99 | 0 / 1 | 0 / 10 | | Long tasks per swipe, p50 / p99 | 0 / 0 | 0 / 3 | All 900 swipes changed scroll state after the mobile-scroll fix. Decision #41 reconnect p99 <300ms passes. The original <5ms key-echo p99 budget remains missed. Journey summaries: `evidence/diagnostics/journey-{1x,4x}-final-summary.json`; scroll trace: `evidence/traces/scroll-sample.json`. ### Eight idle sessions in one tab, 30 one-second samples per rate Totals are for all eight sessions. Memory is MiB; CPU is percent. | Measure | 1x p50 / p99 | 4x p50 / p99 | | --- | ---: | ---: | | JS heap | 249.6 / 249.6 | 264.8 / 264.8 | | WASM | 58.7 / 58.7 | 58.7 / 58.7 | | Renderer RSS | 344.0 / 344.7 | 339.8 / 343.8 | | Server RSS | 16.8 / 16.8 | 16.3 / 16.3 | | Browser CPU | 1.0 / 1.0 | 1.0 / 1.9 | | Renderer process CPU | 2.0 / 5.9 | 83.5 / 87.1* | All eight frames had zero pending rAFs, timeouts, or intervals and zero scheduled work in the 30-second observation. *Chromium's Linux CPU throttler busy-waits in a renderer signal handler, so raw 4x process CPU is emulation overhead; it is not tracer CPU. Evidence: `evidence/diagnostics/concurrent-{1x,4x}-cpu-validation-summary.json`, `evidence/traces/concurrent-idle-{1x,4x}.json`. ### Idle trace and 30-minute mixed-use soak The deployed idle trace had 600 animation-frame callbacks and 24 timer fires in each 10-second capture. After the idle-render change those counts are zero at both rates. RendererMain RunTask fell from 1,102.826 → 1.189 ms/10s at 1x and 4,787.051 → 62.180 ms/10s at 4x (99.9% / 98.7% reduction). The 1,800-second soak ran on `c65cae669377ea31873e23ff10d613f57b2fde1e`, immediately before the final frame-timing commit. It completed 180 ten-second samples with 0 disconnected samples; `ls -R /usr` completed in 33.488s, `htop` ran 120s, and forced reconnects at about 600s and 1,200s completed in 105ms and 103ms and stayed attached for 35s. | Soak measure | p50 / p99 | First → last | | --- | ---: | ---: | | JS heap | 4.17 / 4.96 MB | 4.79 → 4.98 MB (+0.19 MB) | | WASM memory | 12.78 / 12.78 MB | 7.34 → 12.78 MB, then flat after minute 4 | | Renderer RSS | 179.7 / 188.3 MB | 210.4 → 184.4 MB | | Server RSS | 14.93 / 15.00 MB | 14.29 → 15.00 MB | | Browser CPU at 4x | 0 / 4.65% | — | | Renderer process CPU at 4x* | 81.3 / 95.3% | — | No sustained heap, WASM, or renderer-RSS leak was observed; WASM grew during warmup and then stayed flat. *The 4x renderer CPU includes Chromium throttler overhead described above. Soak summary and samples: `evidence/soak-summary.json`, `evidence/soak-progress.json`; trace/log: `evidence/diagnostics/soak-4x-final.log`. Idle traces: `evidence/traces/idle-before-{1x,4x}.json` and `idle-{1x,4x}.json`. ### Ranked findings 1. **Painted keystroke latency still exceeds Decision #41's 5ms p99 target.** The 30-run production journey was 28.5ms p99-of-run-p99 at 1x and 42.8ms at 4x. Moving the explicit draw into Ghostty's already-scheduled frame reduced the focused 500-key run from 18.2/48.4ms p50/p99 to 15.4/19.7ms; the final full suite measured 15.6/16.2ms and passed its `<20ms` assertion. A 4x spot measured 20.3ms. The strict 5ms target remains open. Focused logs: `evidence/diagnostics/playwright-terminal-render-in-raf{,-4x}.log`; final suite: `evidence/diagnostics/playwright-full-final.log`. 2. **Scrolling 10k lines has visible 4x jank.** Frame interval p99 is 33.4ms, dropped-frame p99 is 10 per swipe, and long-task p99 is 3 per swipe. At 1x, interval p99 is 16.8ms and long-task p99 is zero. 3. **Theme rebuild is the slowest measured mobile interaction.** At 4x its p99 is 261ms; palette and sidebar p99 are 167ms and 148ms. No budget for these transitions is stated in Decisions #41/#81. 4. **Eight open sessions consume about 340MB renderer RSS.** This was stable during the idle sample; the 30-minute single-session soak showed no continuing renderer-RSS growth. ### Fixes and before/after evidence | Change | Before → after measurement | | --- | --- | | Stop idle Ghostty frame loop (`8847828`) | 600 rAF callbacks + 24 timer fires/10s → zero; RendererMain RunTask 1,102.826 → 1.189ms/10s at 1x and 4,787.051 → 62.180ms/10s at 4x. | | Restore mobile touch scroll (`794a92a`) | 0/30 swipes moved scrollback → 900/900 swipe checks changed scroll state. | | Buffer/coalesce terminal bursts (`74c4fd6`, `066a20a`, `25e2c70`) | `ls -R /usr` closed with 1013 around 40.17s → 1,014,820 WebSocket bytes delivered and command completed in 33.488s. | | Route liveness Ping on the control stream (`ecb7cbb`) | Terminal-stream Ping detached the client and a two-second idle timer kept firing → one Ping on attach/reconnect, zero idle timers, reconnect p99 112ms/161ms at 1x/4x. | | Start resume expiry at disconnect and replay acknowledged input (`0f2b68e`, `17adc5a`; regressions `a9f77c8`, `c65cae6`) | At a 600s live session the client resumed from cursor 0 while acknowledged input was 85 and got Close 1011 → credentials remain eligible while connected; final 30-minute soak had no disconnects and reconnects at 105/103ms. | | Draw in the write callback frame (`967d278`) | Focused 500-key p50/p99 18.2/48.4ms → 15.4/19.7ms; term.write-to-callback p99 39.3 → 17.9ms. | A repeated same-tab 30-run mobile-sized supplement is retained at `evidence/diagnostics/key-after-mobile-1x.log`; host load varied from 6–15 on 8 cores, so its tail is noisy and is not used as a stable before/after claim. ### Final gates `scripts/verify fast` passed. Verbatim gate summary: ```text advisories ok, bans ok, licenses ok, sources ok ==> static builds Finished `release` profile [optimized] target(s) in 0.40s target/x86_64-unknown-linux-musl/release/infraternal: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, stripped PT_INTERP: absent; size: 15078912 bytes Finished `release` profile [optimized] target(s) in 0.34s target/aarch64-unknown-linux-musl/release/infraternal: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, stripped PT_INTERP: absent; size: 12642000 bytes server subcommand: exit 2, not implemented agent subcommand: exit 2, not implemented +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | PASS | +---------------------------+--------------+ ==> Vendored Ghostty artifact checksums dist/ghostty-web.js: OK dist/ghostty-vt.wasm: OK dist/index.d.ts: OK ==> TypeScript protocol compiler ==> Rust stream oracle (for the differential test) ==> TypeScript protocol tests bun test v1.4.2 (744846f84) src/codec.test.ts: (pass) wire protocol golden vectors > encodes and decodes each frame exactly as the shared manifest describes [5.23ms] (pass) wire protocol golden vectors > returns borrowed payload views and writes into reusable buffers [0.28ms] (pass) wire protocol golden vectors > decodes muxed messages split across WebSocket messages [1.13ms] (pass) wire protocol golden vectors > rejects oversized, non-canonical, unknown, and malformed frame bytes [0.35ms] (pass) split mux header preserves trailing Close [0.34ms] src/stream.test.ts: stream differential: 134 streams, 11439 schedules, zero divergences (pass) Rust and TypeScript preserve every frame at all boundaries and random multi-splits [2637.58ms] (pass) maximum frame size and fragmented oversized prefix obey the allocation boundary [7.11ms] 7 pass 0 fail 34390 expect() calls Ran 7 tests across 2 files. [2.66s] ``` Full Playwright suite on `TRACER_TEST_PORT=4482`: ```text 21 passed (2.3m) ``` Full fast-gate log: `evidence/diagnostics/verify-fast-final.log`. Trace and log directory: `/home/kayg/.cache/codex-scratch/review-perf/evidence/`.
kayg commented 2026-10-07 07:53:06 +00:00
Author
Owner
Copy link

review-perf finished (fix-perf 967d278, based on 7d2d0a5): idle rAF and timers went from 600 and 24 per 10 s to 0; 30-min soak clean; 500-key p50/p99 15.6/16.2 ms (the D41 5 ms budget is about overhead and excludes the display frame; overhead is about 1 ms). Its 14 commits include real resume fixes (resume from the acknowledged input cursor, wait for resume acceptance, resume window starts at disconnect) and server output coalescing.

Merge plan: a merge job after mobile-recovery lands (it overlaps touch scrolling #44). The coalescing (066a20a, 74c4fd6, 25e2c70) must be checked against D130 (no added batching delay for interactive bytes): coalescing is allowed only within the same paint frame or under backpressure, never as a timer that delays echo. Measured at 300 ms RTT.

review-perf finished (fix-perf 967d278, based on 7d2d0a5): idle rAF and timers went from 600 and 24 per 10 s to 0; 30-min soak clean; 500-key p50/p99 15.6/16.2 ms (the D41 5 ms budget is about overhead and excludes the display frame; overhead is about 1 ms). Its 14 commits include real resume fixes (resume from the acknowledged input cursor, wait for resume acceptance, resume window starts at disconnect) and server output coalescing. Merge plan: a merge job after mobile-recovery lands (it overlaps touch scrolling #44). The coalescing (066a20a, 74c4fd6, 25e2c70) must be checked against D130 (no added batching delay for interactive bytes): coalescing is allowed only within the same paint frame or under backpressure, never as a timer that delays echo. Measured at 300 ms RTT.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
j470b-server-mesh-rejoin
mesh-compat-382
j444b-d381-key-wins-flood
j461b-macos-utun-reader-thread
j475b-remove-host-motion
fix-reconnect-new-link
j481-d380
dev
j479-mesh-soak
fix/j476-host-facts
j475-motion-pill-split
j470-mac-node-join
root-upgrade-d376
j474-review-merge
latency-targets-120hz
nightly-suite-shards
gotatun-pin-j469
macos-utun-reader-thread
fix-flood-echo-stall
wip/j416
motion-fast-ease-out
mesh-addhost-j259
takeover-close-terminals-d368
kbd-instant-followups
fix-trust-key-double-click
nightly-suite-partial-report
appwin-icons-j207
rusqlite-slice1
swipe-content-first-d374
mesh-mtu-381
addhost-instant-open
glass-contrast-d365-mock
glass-contrast-d365
signout-followups-442
fix-startup-toast-red
research-gotatun-matrix-j317
themes-sync-j304
fix-load-flakes
fix-nix-zig-patch
fix-compute01-browser-baseline
ratchet-size-baseline-419
predict-auto-d130
revoke-coverage-d367
nightly-suite-queue-timeout
nightly-browser-suite
fix-rustsec-2026-0459
fonts-ioskeley-obel
log-retention-d375
fix-kbd-no-anim-427
fix-dev-red-specs-2
takeover-proof-305-j205
wip/j410-server-pump
ui-consistency-d372
fix-osc52-clipboard
libghostty-vt-1.0
forget-retired-host-d369
j404-merge-426
fix-setup-lock-race
fix-touch-select-426
revoke-access-report-ui
fix-predictive-echo-specs
fix-selinux-agent-exec
wip/j395-checkpoint2
wip/j395
j391-review-scratch
ui-consistency-j195-deferred-rebased
fix-rustc-paths-421
decisions-wording-341
fix-install-suite-red
fix-dev-rtt-specs
fix-pending-removal-align
fix-privileged-skip
fix-install-suite-404
review-j368
mesh-dns-j254
revoke-access-report-d356b
revoke-privilege-round5
fix-removal-confirm
review-j360
gotatun-rotation-367
deps-refresh-340
review-j354
review-J351
review-j350
addhost-takeover-d359
revoke-report-items-347
revoke-end-sessions-d362
review-j348
size-mesh-ssh-346
size-trim-337
revoke-access-diff-d356
j259-pre-rebase-b06c5a6
j305-ended-at-start-spec
addhost-sudo-consent
fix-layout-real-specs-411
fix-shell-open-411
fix-console-hosts-373
fix-shell-exit
fuzz-host-name-353
fix-host-details-410
macos-filter-j197
research-gotatun-tricks
revoke-p3-j216l
stale-state-pill
revoke-followups-j216k
revoke-progress-j216e
backup/macos01-20261106/lume/j197
backup/macos01-20261106/lume/appwin-icons-j207-stash0
backup/macos01-20261106/claude/stash-0
backup/macos01-20261106/claude/stash-1
backup/macos01-20261106/claude/J80-macos-s1
backup/macos01-20261106/claude/J80
backup/macos01-20261106/claude/J80-slice1-20261102
backup/macos01-20261106/claude/J16b-appwin-on-dev
backup/macos01-20261106/claude/appwin-launch-idle-146
backup/macos01-20261106/claude/J80-s1-run
wip/j259
trust-285-p3c
runtime-lock-j216f
mesh-ssh-p3-j175
tun-proof-p3b
trust-285-p3
terminal-polish-j271
tun-proof-p3
ssh-install-flakes-j216i
fuzz-fixes-j264
trust-285-a
trust-285-b
restart-oversized-fixture-j198
backup/macos01-20261106/lume/appwin-icons-j207
restart-bound-j216h
up-preflight-p3-j198
backup/macos01-20261106/lume/j251
up-preflight-j198
wip/up-preflight-j198-j245-p2
wip/up-preflight-j198-j245-p1
agent-fsync-offloop-j196
wip/up-preflight-j198-j245
settings-offscreen-j231
wip/up-preflight-j198-j226-inspection
wip/j103f
themes-j206
wip/up-preflight-j198-j226
flakes-j216c
backup/macos01-20261106/claude/j197
flakes-j216b
backup/macos01-20261106/claude/j197-red
wip/up-preflight-j198-review
flakes-j216
backup/macos01-20261106/claude/appwin-icons-j207
engineering-j208b
appwin-mock-j209
toast-stack-j165
engineering-j208
decisions-d351
decisions-d350
decisions-d349
appearance-presets-mock-j129
decisions-d348
docs-audit-spec
doc-coverage-baseline-raise
decisions-d347
decisions-d346b
decisions-d346
j88h-profile
ui-consistency-j195-deferred
ui-consistency-j195
engineering-practices-j194
preferred-login-ui-j166
signin-pools-263-j174
macos-node-265-s1
decisions-d345
decisions-d344
proxy-cgroup-j164
verify-quick-fuzz-329
proxy-start-io-j168
timing-flakes-j163
up-263-s1
mcp-events-j147
verify-quick-fuzz-j158
proxy-p3-j159
agent-resource-j157-detect
wip/j103d-final
remote-cancel-hang-j155
proxy-traefik-281-j62d
j149-verify-quick
proxy-traefik-281-j62b
j150-drop-unnamed
wip/j79d-red-r4
wip/j103d-d332
rootless-toast-j142
wip/j103d-head
wip/j103d
j125c-mcp
macos-build-j136
backup/macos01-20261106/claude/j136
wip/j79d-red-cli
wip/j79d-red-server
j121-link-liveness
j118c-tracer-failures
wip/j79c-head
release-desktop-j131
j124c-signed-macos
wip/j79c-red-p3
wip/j79c-red-p23
wip/j79c-red-p22
merge-j128-check
perf-deferred-j128
sidebar-toggle-260
j127-shortcut-changed-dot
j125-mcp
j118-tracer-failures
j126-account-summary
settings-sections-mock-j92
j120-pinch-lock
sidebar-292
backup/macos01-20261106/claude/j120
taste-2026-11-03
row-actions-242-215
backup/macos01-20261106/claude/stash-0-untracked
backup/macos01-20261106/claude/J80-265-s1-run
backup/macos01-20261106/claude/J80-macos-s1-20261102
j112-canvas-commands
ui-no-overshoot-scroll
fix-recovery-restart
backup/macos01-20261106/claude/stash-1-untracked
taste-2026-11-02
size-baseline-ratchet
machine-cards-structure-j108
j109-boot-copy-mockups
fix-dev-red
wip/up-263-s1
j95b-settings-rows
proxy-traefik-281
j95-option-rows
docs-human-j91
docs-keys-palette
fix-flood-1013-270-j38d
mesh-ssh-s1
mesh-ssh-s0
addhost-d291-tunnel
test-stability-load
mesh-relay-220
repro-296
shim-thp-off-24
mesh-pins-258-v2
mesh-dns
signin-d311-merged
docs-pages
ui-size-283
fix-ui-batch-specs
wip/j12b-edge-update
gate-on-arm
fix-rename-sync-293
trust-285-j60b
fix-focus-ring-clip
integ-ui-1101
edge-update-v2
signin-d311
taste-2026-11-01
layout-per-tab-250
ui-approved-splitopt-224
ui-approved-splitopt-224-on-dev
icon-text-centre-282
ui-approved-189-212-216
research-reverse-proxy
transfer-budget-320
review-mesh-pins-258-r2
sidebar-pill-259
review-mesh-relay-220
wip/j12-edge-update-v2
nix-package
status-crash-loop-237
fix-dev-test-failures
fix-fixture-socket-path
wip/mesh-relay-220-pre-dev-advance-rebase
fix-canvas-side-clip
research-head-to-head
frosted-real-256
review-mesh-pins-258
wip/mesh-relay-220-pre-final-rebase
review-layout-per-tab-250-r2
broad-suite-triage
nix-devshell
review-status-crash-loop-237-r3
wip/nix-devshell-pre-p3
taste-frosted-real
wip/code-serve-demo-2026-10-30
wip/code-serve-glass-2026-10-30
wip/code-serve-clusters-2026-10-30
wip/code-merge-release94-dev-2026-10-30
wip/rproxy-delta-scratch-2026-10-30
wip/stale-note-257-staged-2026-10-30
wip/d241-size-staged-2026-10-30
review-broad-suite-207
wip/mesh-pins-258-pre-split
wip/mesh-relay-220-pre-rebase
mesh-measure-220
plan-211-update
review-edge-autoupdate-r3
perf24-reconnect
review-stale-note-r2
release-prod-live
d241-shell-marker
review-mesh-pins-server-258
review-revocation-r3
review-setup-token-r2
review-sync-journal-r3
wip/review-mesh-pins-258-prune-review-1995fee46
frosted-scroll-256
sync-journal-r3
revocation-per-account-wip
review-tokio-asyncfd-0386
review-sync-p3s
wip/stash-8by64-stash0-2026-10-30
mesh-drag-r2
setup-token-data-wip
release-9-4
auth-busy-1013
mesh-view-overlap
sync-p3s
review-auth-busy-wake
review-readd-r8-wip
mesh-drag-bounds
audit-auth-93
app-windows-stream-start
review-authority-cap-r3
app-windows-r7-measure
readd-minimal-r6
readd-minimal-r5
app-windows-r7-soak
readd-minimal-r4
app-windows-r7
app-windows-r6
readd-minimal-r3
readd-minimal-r2
readd-minimal
merge-release8-dev
review-busy-page-liveness-r2
review-busy-page-liveness
mesh-view-trim
review-authority-read-retry-r2
review-remove-uninstall-r2
release-retry-reattach
release-retry
review-remove-uninstall
review-authority-read-retry
app-windows-ship
mesh-autoswitch-p2
mesh-ssh
mesh-relay
mesh-macos-node
screen-and-control
session-labels
review-real-logins-r5
review-real-logins-r4
review-real-logins-r3
shim-memory-r2
review-real-logins-r2
shim-memory
shadow-addhost-instant
shadow-real-logins
shadow-same-session-flood
shadow-prod-perf-fix
hosts-from-server
app-windows-real
app-windows-transport
release-prod-6w
mesh-policy-verify
quiet-close-toast
dev-merge-r4
review-mesh-agent-transport
review-app-windows-transport
settings-roundtrip
review-release-pipeline-r4
mesh-agent-transport
ended-strip-fix
review-audit-ca-update-d244
settings-rows
phone-width
signout-clear
fix-restart-tests
dev-merge-rp
signer-hardening
review-audit-ca-update
terminal-fidelity
review-audit-ca-update-wip
mesh-integrate
review-mesh-agent-transport-515d3712
review-mesh-dataplane-feec1988
mesh-dataplane
signout
research-canvas-node
wip/review-renderer-cost
wip/review-terminal-fidelity
padlock-glyph
review-release-pipeline-2e95b689
phone-polish
sidebar-polish
inbox-generic
review-mesh-dataplane-378d4055
prod-batch
ui-inbox
ui-ca-unlock
ui-machine-sections
review-mesh-dataplane-a5f744fb
tls-session-prod
session-bounds-prod
fix-dead-session
session-bounds
tls-session
palette-polish
shortcuts-d222
research-appmeta
phone-gutter
ui-centre-button
wip/ui-centre-button
wip/tls-session
review-addhost-breakit
ui-platform
renderer-cost
remote-agents
mesh-api
ssh-bootstrap
prod-hotfix2
prod-fixes
review-remote-agents
review-mesh-dataplane
session-life
research-browser
review-mesh-api
research-capture
review-mesh-api-round3
review-mesh-api-round1
gate-stability
review-gate-stability
review-relay-direct
review-mesh-api-findings
proto-sessions
font-picker
proto-addhost
proto-windows
mesh-core
latency-gate
research-bsd
research-discovery
keytray-mods
mock-core
ui-tweaks
fix-motion-sel
fix-motion
integrate2
fix-robust
t-web
t-server
vt-ghostty
fix-snap-echo
t-web-auth
review-vt2
t-agent
lane-protocol
lane-shim
lane-auth
s1-base
s1-contracts
s1-workspace
No results found.
Labels
Clear labels   
launch-qa
Findings from launch QA.

No labels
launch-qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/infraternal#42
No description provided.