kayg/infraternal
Watch
1
0
Fork
You've already forked infraternal
 0
Code Issues 356 Pull requests Projects Releases Packages Wiki Activity Actions

Latency gate: D375 120 Hz target table in PROCESS §6 and verify latency (J407) #447

New issue
Open
opened 2026-11-09 06:31:26 +00:00 by kayg · 20 comments
kayg commented 2026-11-09 06:31:26 +00:00
Owner
Copy link

D375 (owner, 2026-11-09) rewrites D41's budgets as a 120 Hz target table (one frame = 8.3 ms). This job (J407, branch latency-targets-120hz, base 80a09ad02) brings PROCESS.md §6 and the latency gate in line with it.

Scope:

  • PROCESS.md §6: the D375 table replaces "RTT × 2 + 300 ms" and the 60 Hz frame wording.
  • scripts/verify latency measures and reports every D375 row on every run, gated exactly as D375 says (own overhead gates at today's ~25 ms baseline until 8.3 ms is reached).
  • Move hard-coded 60 Hz (16.67 ms) budgets to 120 Hz. The renderer's actual frame pacing is left alone; what would need to change is listed here with its cost.
  • One latency-profile run on dev to record today's number per row.

No merge, no deploy.

D375 (owner, 2026-11-09) rewrites D41's budgets as a 120 Hz target table (one frame = 8.3 ms). This job (J407, branch `latency-targets-120hz`, base `80a09ad02`) brings PROCESS.md §6 and the latency gate in line with it. Scope: - PROCESS.md §6: the D375 table replaces "RTT × 2 + 300 ms" and the 60 Hz frame wording. - `scripts/verify latency` measures and reports every D375 row on every run, gated exactly as D375 says (own overhead gates at today's ~25 ms baseline until 8.3 ms is reached). - Move hard-coded 60 Hz (16.67 ms) *budgets* to 120 Hz. The renderer's actual frame pacing is left alone; what would need to change is listed here with its cost. - One latency-profile run on dev to record today's number per row. No merge, no deploy.
kayg commented 2026-11-09 06:31:34 +00:00
Author
Owner
Copy link

Start: branch latency-targets-120hz from origin/dev 80a09ad02.

Plan (about 3 h):

  1. PROCESS.md §6 + §1 + perf/README: the D375 table; drop "RTT × 2 + 300 ms" and the 60 Hz frame (30 min).
  2. One D375 metric format (METRIC name=d375_<row> value_ms= target_ms= gate_ms= result=) and scripts/latency-report.py, which prints the 10-row table at the end of verify latency and fails on a missing or failed row (30 min).
  3. Native rows in scripts/verify-latency.py (real agent + PTY): real echo (RTT + jitter bound + 16.7), flood (+8.3), own overhead at zero network, resumed-link reattach (1 RTT + 50), idle agent RSS (45 min).
  4. Browser rows in the tracer scenario: predicted glyph (8.3), network-change reconnect (4 RTT + 50, replaces RTT × 2 + 300), last screen on drop, browser share of own overhead; a latency-profile spec for 120 fps at 10 MB/s and phone first paint (CPU ×4); perf-VM asserts follow (45 min).
  5. 60 Hz inventory: budgets moved, pacing listed with cost, not changed.
  6. Gate + one latency-profile run on dev; table posted here.
Start: branch `latency-targets-120hz` from origin/dev `80a09ad02`. Plan (about 3 h): 1. PROCESS.md §6 + §1 + perf/README: the D375 table; drop "RTT × 2 + 300 ms" and the 60 Hz frame (30 min). 2. One D375 metric format (`METRIC name=d375_<row> value_ms= target_ms= gate_ms= result=`) and `scripts/latency-report.py`, which prints the 10-row table at the end of `verify latency` and fails on a missing or failed row (30 min). 3. Native rows in `scripts/verify-latency.py` (real agent + PTY): real echo (RTT + jitter bound + 16.7), flood (+8.3), own overhead at zero network, resumed-link reattach (1 RTT + 50), idle agent RSS (45 min). 4. Browser rows in the tracer scenario: predicted glyph (8.3), network-change reconnect (4 RTT + 50, replaces RTT × 2 + 300), last screen on drop, browser share of own overhead; a latency-profile spec for 120 fps at 10 MB/s and phone first paint (CPU ×4); perf-VM asserts follow (45 min). 5. 60 Hz inventory: budgets moved, pacing listed with cost, not changed. 6. Gate + one latency-profile run on dev; table posted here.
kayg commented 2026-11-09 06:44:29 +00:00
Author
Owner
Copy link

Finding 1: the 60 Hz inventory (code at 9850758c3).

60 Hz budgets moved to 120 Hz (in this branch):

Where Was Now
scripts/verify-latency.py flood growth 1000/60 (16.67 ms) one 120 Hz frame (8.33 ms)
scripts/verify-latency.py echo components RTT + 34 ms round trip + 16.7 ms (2 frames)
apps/tracer/test/terminal-scenario.ts predicted glyph 17 ms 8.33 ms
same, real echo RTT + jitter + 34 ms round trip + 16.7 ms
same, reconnect RTT × 2 + 300 ms 4 round trips + 50 ms (D267)
apps/tracer/perf/terminal.spec.ts predicted paint assert 17 ms 8.33 ms
apps/tracer/perf/synchronized-output-input.spec.ts held-output input send 16 ms 8.33 ms
apps/tracer/test/terminal-theme.spec.ts theme-switch JS p99 16.67 ms 8.33 ms

Frame pacing (left unchanged, as the task says):
The renderer does not cap at 60 Hz. Every paint path is requestAnimationFrame: Ghostty-web's render loop (vendor/ghostty-web/dist/ghostty-web.js), predictive echo's overlay (predictive-echo.ts:701,745), synchronized-output release (synchronized-output.ts:293), touch momentum (terminal-touch.ts:91) and the canvas springs. rAF runs at the display's rate, so a 120 Hz display already gets 120 Hz pacing. Nothing has to change for pacing. These are the places that assume 60 Hz, and what changing them would cost:

Where 60 Hz assumption Change and cost
src/canvas/springs.ts restAt rest velocity delta * 60 ("a quarter pixel per frame at 60 Hz") At 120 Hz a spring comes to rest at 2× the per-frame motion, so the tail can stop about one 60 Hz frame earlier. Changing it to delta * 120 costs one more spring frame per flight. This is visual only, not latency. Leave it.
src/terminal-touch.ts momentum 0.95^(dt/16) Already time-based and correct at both rates. No change.
src/predictive-echo.ts CONFIRMATION_FRAME_BUDGET_MS = 34 2 × 60 Hz frames before a pending prediction counts as late This is a misprediction timeout, not a latency budget. At 16.7 ms, a slow paint (the measured browser real-echo p99 is below) would trigger false rollbacks and flicker. Cost to change: more fallbacks, which is a correctness risk. Leave it.
src/perf-hud.on-demand.ts:440 16.7 ms fallback when the refresh rate is unknown Display only. No change.
test/motion-audit.spec.ts:121 counts missed frames in 1000/60 units Headless Chromium runs at 60 Hz, so the unit matches the venue. Move it only together with a frame-rate-unlocked browser.
real/switch.spec.ts medians (34/50, 70/100 ms) "two frames" bounds on a shared host The comment now names the 120 Hz target. The numbers stay loose because the host is shared. The D375 last-screen row is gated in verify latency.
TASTE.md / canvas/camera, mesh/graph "60 fps rule" design rule text Owner's document. Not touched. Flag for the owner if the rule should read 120.

The 120 fps cost (#260-style): the budget per frame halves, from 16.7 to 8.3 ms. The rendering row measures it directly: 1000 / p99 animation-frame interval with the frame-rate limit off while the terminal takes 10 MB/s. It is in the run table, posted next. If p99 frame work is above 8.3 ms, the cost is engine work (parse and raster per frame), not pacing. Unverified platform note: iOS Safari has historically capped rAF at 60 Hz behind a user setting. If it still does, a phone PWA cannot show 120 fps whatever the code does. That needs a ProMotion device to confirm.

Finding 1: the 60 Hz inventory (code at `9850758c3`). **60 Hz budgets moved to 120 Hz (in this branch):** | Where | Was | Now | |---|---|---| | `scripts/verify-latency.py` flood growth | `1000/60` (16.67 ms) | one 120 Hz frame (8.33 ms) | | `scripts/verify-latency.py` echo components | RTT + 34 ms | round trip + 16.7 ms (2 frames) | | `apps/tracer/test/terminal-scenario.ts` predicted glyph | 17 ms | 8.33 ms | | same, real echo | RTT + jitter + 34 ms | round trip + 16.7 ms | | same, reconnect | RTT × 2 + 300 ms | 4 round trips + 50 ms (D267) | | `apps/tracer/perf/terminal.spec.ts` predicted paint assert | 17 ms | 8.33 ms | | `apps/tracer/perf/synchronized-output-input.spec.ts` held-output input send | 16 ms | 8.33 ms | | `apps/tracer/test/terminal-theme.spec.ts` theme-switch JS p99 | 16.67 ms | 8.33 ms | **Frame pacing (left unchanged, as the task says):** The renderer does not cap at 60 Hz. Every paint path is `requestAnimationFrame`: Ghostty-web's render loop (`vendor/ghostty-web/dist/ghostty-web.js`), predictive echo's overlay (`predictive-echo.ts:701,745`), synchronized-output release (`synchronized-output.ts:293`), touch momentum (`terminal-touch.ts:91`) and the canvas springs. rAF runs at the display's rate, so a 120 Hz display already gets 120 Hz pacing. Nothing has to change for pacing. These are the places that assume 60 Hz, and what changing them would cost: | Where | 60 Hz assumption | Change and cost | |---|---|---| | `src/canvas/springs.ts` `restAt` | rest velocity `delta * 60` ("a quarter pixel per frame at 60 Hz") | At 120 Hz a spring comes to rest at 2× the per-frame motion, so the tail can stop about one 60 Hz frame earlier. Changing it to `delta * 120` costs one more spring frame per flight. This is visual only, not latency. Leave it. | | `src/terminal-touch.ts` momentum | `0.95^(dt/16)` | Already time-based and correct at both rates. No change. | | `src/predictive-echo.ts` `CONFIRMATION_FRAME_BUDGET_MS = 34` | 2 × 60 Hz frames before a pending prediction counts as late | This is a misprediction timeout, not a latency budget. At 16.7 ms, a slow paint (the measured browser real-echo p99 is below) would trigger false rollbacks and flicker. Cost to change: more fallbacks, which is a correctness risk. Leave it. | | `src/perf-hud.on-demand.ts:440` | 16.7 ms fallback when the refresh rate is unknown | Display only. No change. | | `test/motion-audit.spec.ts:121` | counts missed frames in 1000/60 units | Headless Chromium runs at 60 Hz, so the unit matches the venue. Move it only together with a frame-rate-unlocked browser. | | `real/switch.spec.ts` medians (34/50, 70/100 ms) | "two frames" bounds on a shared host | The comment now names the 120 Hz target. The numbers stay loose because the host is shared. The D375 last-screen row is gated in `verify latency`. | | `TASTE.md` / `canvas/camera`, `mesh/graph` "60 fps rule" | design rule text | Owner's document. Not touched. Flag for the owner if the rule should read 120. | **The 120 fps cost (#260-style):** the budget per frame halves, from 16.7 to 8.3 ms. The rendering row measures it directly: 1000 / p99 animation-frame interval with the frame-rate limit off while the terminal takes 10 MB/s. It is in the run table, posted next. If p99 frame work is above 8.3 ms, the cost is engine work (parse and raster per frame), not pacing. Unverified platform note: iOS Safari has historically capped rAF at 60 Hz behind a user setting. If it still does, a phone PWA cannot show 120 fps whatever the code does. That needs a ProMotion device to confirm.
kayg commented 2026-11-09 07:06:04 +00:00
Author
Owner
Copy link

Finding 2: a native trial run on compute01, before the full profile. This is evidence only, the profile run comes later.

Binary: a static build from j404-merged (dev, 06:46 UTC), with this branch's scripts/verify-latency.py at 9850758c3, nice 10, on a shared, loaded VM. Log: claude@infraternal-dev-01-compute01:~/.cache/codex-scratch/j407/native-trial.log.

METRIC name=d375_own_overhead ... n=100 value=0.74 unit=ms target=8.33 gate=25.00 target_met=yes result=PASS p50_ms=0.39 rtt_ms=0
METRIC name=d375_idle_agent_rss ... value=10.00 unit=MB target=15.00 result=PASS no_session_kb=6328 one_quiet_session_kb=9764
METRIC name=keystroke_pty_echo_baseline ... p50_ms=296.36 p99_ms=324.04 budget_ms=346.67 result=PASS
METRIC name=flood_isolation_other_session ... growth_ms=-21.56 budget_ms=8.33 result=PASS
METRIC name=d375_reconnect_resumed_link ... n=20 value=318.66 unit=ms target=380.00 result=PASS p50_ms=302.33 first_key_echo_p99_ms=626.54
METRIC name=keystroke_pty_echo_flood_same_session ... p50_ms=283.52 p99_ms=472.03 budget_ms=346.67 result=FAIL
METRIC name=d375_flood_effect ... value=147.99 unit=ms target=8.33 result=FAIL other_session_ms=-21.56 same_session_ms=147.99
  • Same-session flood is the real gap. When yes floods the same PTY you type in, echo p99 grows by 148 ms. This fails the old budgets too (472 ms against the old RTT + 34 = 334 ms), so the failure is not caused by the 120 Hz change. A flood in a different session costs nothing (−22 ms, which is noise).
  • The agent's own overhead is tiny. It measured 0.74 ms p99, but that covers only the agent and PTY, with no browser. Reported alone, it would say the 8.3 ms target is met while leaving out the browser. In 0e7b61962 the row is now the agent's p99 plus the browser's own spans p99 (keydown to send, plus echo arrival to paint), an upper bound. D296's ~25 ms was a different workload: the pooled shell echo on the perf VM. So the gate stays at the 25 ms baseline until a perf-VM run of this composed row reaches 8.3 ms.
  • The resumed-link reattach takes about one round trip (p99 319 ms against a 380 ms budget at 300 ± 30 ms RTT).
Finding 2: a native trial run on compute01, before the full profile. This is evidence only, the profile run comes later. Binary: a static build from `j404-merged` (dev, 06:46 UTC), with this branch's `scripts/verify-latency.py` at `9850758c3`, nice 10, on a shared, loaded VM. Log: `claude@infraternal-dev-01-compute01:~/.cache/codex-scratch/j407/native-trial.log`. ``` METRIC name=d375_own_overhead ... n=100 value=0.74 unit=ms target=8.33 gate=25.00 target_met=yes result=PASS p50_ms=0.39 rtt_ms=0 METRIC name=d375_idle_agent_rss ... value=10.00 unit=MB target=15.00 result=PASS no_session_kb=6328 one_quiet_session_kb=9764 METRIC name=keystroke_pty_echo_baseline ... p50_ms=296.36 p99_ms=324.04 budget_ms=346.67 result=PASS METRIC name=flood_isolation_other_session ... growth_ms=-21.56 budget_ms=8.33 result=PASS METRIC name=d375_reconnect_resumed_link ... n=20 value=318.66 unit=ms target=380.00 result=PASS p50_ms=302.33 first_key_echo_p99_ms=626.54 METRIC name=keystroke_pty_echo_flood_same_session ... p50_ms=283.52 p99_ms=472.03 budget_ms=346.67 result=FAIL METRIC name=d375_flood_effect ... value=147.99 unit=ms target=8.33 result=FAIL other_session_ms=-21.56 same_session_ms=147.99 ``` - **Same-session flood is the real gap.** When `yes` floods the same PTY you type in, echo p99 grows by 148 ms. This fails the old budgets too (472 ms against the old RTT + 34 = 334 ms), so the failure is not caused by the 120 Hz change. A flood in a *different* session costs nothing (−22 ms, which is noise). - **The agent's own overhead is tiny.** It measured 0.74 ms p99, but that covers only the agent and PTY, with no browser. Reported alone, it would say the 8.3 ms target is met while leaving out the browser. In `0e7b61962` the row is now the agent's p99 plus the browser's own spans p99 (keydown to send, plus echo arrival to paint), an upper bound. D296's ~25 ms was a different workload: the pooled shell echo on the perf VM. So the gate stays at the 25 ms baseline until a perf-VM run of this composed row reaches 8.3 ms. - The resumed-link reattach takes about one round trip (p99 319 ms against a 380 ms budget at 300 ± 30 ms RTT).
kayg commented 2026-11-09 08:29:07 +00:00
Author
Owner
Copy link

Progress: the profile run (run 1) at 0e7b61962, compute01, load 28–42 on 16 cores.

  • verify quick: PASS (QUICK_EXIT=0).
  • Native rows (the full profile log: compute01:~/.cache/codex-scratch/j407/evidence/verify-latency-20261109T073942Z.log):
    • own_overhead_agent p99 5.45 ms (p50 0.36);
    • idle RSS 10.02 MB (6.35 MB with no session, 9.79 MB with one quiet session);
    • resumed link 323 ms against 380;
    • flood effect +129 ms (same session) and +37 ms (other session). The other-session number was −22 ms in the quieter trial, so it looks like load noise. The same-session growth repeats.
  • Finding 3: verify-auth-browser failed on dev's own code with "initial public status". Since #252, /auth/status also returns setup_token_command, and scripts/browser-auth/check.mjs compared the whole body with {setup_required:true}. This branch has no Rust changes, so the failure is dev's. Fixed in 92233f3a2: the check now verifies setup_required and that no keys beyond the two public ones appear.
  • Change of plan: the profile's browser step runs the whole functional suite, 1283 tests on one worker. That took 19.5 min for 82 tests, so about 5 h in a shared slot, and it adds nothing to the D375 rows. I stopped it with SIGINT after 82 tests. The rows end-to-end report already worked on the partial log: it printed 3 rows and 7 MISSING, and exited 1. Run 2 (queued, position 14) runs the browser step on the two row-producing specs only, reports on the combined log, then runs verify fast.
  • Open question for the orchestrator/owner, not decided here: should verify latency run only the latency-producing specs instead of the full functional suite? The suite already runs elsewhere. The safe default is unchanged: this branch leaves the profile's step list as it was.
Progress: the profile run (run 1) at `0e7b61962`, compute01, load 28–42 on 16 cores. - `verify quick`: PASS (QUICK_EXIT=0). - Native rows (the full profile log: `compute01:~/.cache/codex-scratch/j407/evidence/verify-latency-20261109T073942Z.log`): - `own_overhead_agent` p99 5.45 ms (p50 0.36); - idle RSS 10.02 MB (6.35 MB with no session, 9.79 MB with one quiet session); - resumed link 323 ms against 380; - flood effect +129 ms (same session) and +37 ms (other session). The other-session number was −22 ms in the quieter trial, so it looks like load noise. The same-session growth repeats. - **Finding 3:** `verify-auth-browser` failed on dev's own code with "initial public status". Since #252, `/auth/status` also returns `setup_token_command`, and `scripts/browser-auth/check.mjs` compared the whole body with `{setup_required:true}`. This branch has no Rust changes, so the failure is dev's. Fixed in `92233f3a2`: the check now verifies `setup_required` and that no keys beyond the two public ones appear. - **Change of plan:** the profile's browser step runs the *whole* functional suite, 1283 tests on one worker. That took 19.5 min for 82 tests, so about 5 h in a shared slot, and it adds nothing to the D375 rows. I stopped it with SIGINT after 82 tests. The rows end-to-end report already worked on the partial log: it printed 3 rows and 7 MISSING, and exited 1. Run 2 (queued, position 14) runs the browser step on the two row-producing specs only, reports on the combined log, then runs `verify fast`. - Open question for the orchestrator/owner, not decided here: should `verify latency` run only the latency-producing specs instead of the full functional suite? The suite already runs elsewhere. The safe default is unchanged: this branch leaves the profile's step list as it was.
kayg commented 2026-11-09 11:04:22 +00:00
Author
Owner
Copy link

Run 2 (head 92233f3a2, compute01, load 31–40). Log: compute01:~/.cache/codex-scratch/j407/evidence/verify-latency-browser-rows-20261109T105844Z.log.

  • Rendering at 10 MB/s, Finding 4: the frame-limit flags worked (idle 269 fps uncapped). Under 10 MB/s the p99 frame interval is 29.1 ms, which is 34 fps at p99 against the 120 target. The mean is 234 fps, and the effective feed was 9.87 MB/s with 511 ms of write CPU. So most frames are cheap, but about 1 in 100 takes 29 ms: a parse or raster burst when a 100 KB chunk lands. This is the engine cost behind "120 fps" (a 120 Hz frame is 8.3 ms). Pacing is not the problem.
  • Two of my probes were wrong, fixed in 67cea945b:
    • last screen: the probe picked the hidden prediction-overlay canvas;
    • phone first paint: no first-contentful-paint entry was found. It now uses a buffered observer, with the visible-after-two-frames upper bound as a fallback, and counts the fallbacks.
  • verify-auth-browser stopped at its own relay self-test (configured 300 ms RTT did not arrive: p50=332.53 ms p99=371.59, load 37). That is host load, so the #252 fix has not been exercised yet.
  • I stopped run 2's verify fast after a minute, because it was gating a head I was about to change. Run 3, queued at position 10, runs every row on the final head in one log, then verify fast.
Run 2 (head `92233f3a2`, compute01, load 31–40). Log: `compute01:~/.cache/codex-scratch/j407/evidence/verify-latency-browser-rows-20261109T105844Z.log`. - **Rendering at 10 MB/s, Finding 4:** the frame-limit flags worked (idle 269 fps uncapped). Under 10 MB/s the **p99 frame interval is 29.1 ms, which is 34 fps at p99** against the 120 target. The mean is 234 fps, and the effective feed was 9.87 MB/s with 511 ms of write CPU. So most frames are cheap, but about 1 in 100 takes 29 ms: a parse or raster burst when a 100 KB chunk lands. This is the engine cost behind "120 fps" (a 120 Hz frame is 8.3 ms). Pacing is not the problem. - Two of my probes were wrong, fixed in `67cea945b`: - **last screen:** the probe picked the hidden prediction-overlay canvas; - **phone first paint:** no `first-contentful-paint` entry was found. It now uses a buffered observer, with the visible-after-two-frames upper bound as a fallback, and counts the fallbacks. - `verify-auth-browser` stopped at its own relay self-test (`configured 300 ms RTT did not arrive: p50=332.53 ms p99=371.59`, load 37). That is host load, so the #252 fix has not been exercised yet. - I stopped run 2's `verify fast` after a minute, because it was gating a head I was about to change. Run 3, queued at position 10, runs every row on the final head in one log, then `verify fast`.
kayg commented 2026-11-09 12:49:38 +00:00
Author
Owner
Copy link

Run 3 (head 67cea945b, compute01, load 32–39 on 16 cores). Log: compute01:~/.cache/codex-scratch/j407/evidence/verify-latency-rows-*.log (13:25 UTC).

  • The relay's own self-test failed under this load (configured 300 ms RTT did not arrive: p50=321.95 p99=425.45). Every round-trip row from this run is inflated by host load: baseline PTY echo p99 was 350 ms against 324–334 in the quieter runs.
  • Native: agent own overhead p99 4.37 ms, idle RSS 10.02 MB, resumed link 331 ms (budget 380), flood: same session +40.8 ms, other session −26.6 ms. Same-session growth was +148, +129 and +41 ms across the three runs, so it is real but noisy. Other-session growth is noise.
  • The browser rows did not run: tsc refused browserType (not part of this Playwright test API). Fixed in 94882075f with the playwright.chromium launcher, and the tracer tsc now exits 0 on compute01. I stopped run 3's verify fast again for the same reason as before.
  • Run 4 (final head 94882075f: all rows + verify fast) is queued at position 16.
Run 3 (head `67cea945b`, compute01, load 32–39 on 16 cores). Log: `compute01:~/.cache/codex-scratch/j407/evidence/verify-latency-rows-*.log` (13:25 UTC). - The relay's own self-test failed under this load (`configured 300 ms RTT did not arrive: p50=321.95 p99=425.45`). Every round-trip row from this run is inflated by host load: baseline PTY echo p99 was 350 ms against 324–334 in the quieter runs. - Native: agent own overhead p99 4.37 ms, idle RSS 10.02 MB, resumed link 331 ms (budget 380), flood: same session +40.8 ms, other session −26.6 ms. Same-session growth was +148, +129 and +41 ms across the three runs, so it is real but noisy. Other-session growth is noise. - The browser rows did not run: `tsc` refused `browserType` (not part of this Playwright test API). Fixed in `94882075f` with the `playwright.chromium` launcher, and the tracer `tsc` now exits 0 on compute01. I stopped run 3's `verify fast` again for the same reason as before. - Run 4 (final head `94882075f`: all rows + `verify fast`) is queued at position 16.
kayg commented 2026-11-09 13:06:55 +00:00
Author
Owner
Copy link

Finding 5: the browser rows on a quiet venue. dev-02 (aarch64, 10 cores, load 2.7–3.4), head 94882075f. Log: infraternal-dev-02-std:~/.cache/codex-scratch/j407/evidence/verify-latency-rows-aarch64-*.log. The relay self-test PASSED here: p50 303.6, p99 319.3 ms.

d375_predicted_glyph          n=20  value=15.00 ms  target=8.33   FAIL  p50=14.30 overlay_n=20 fallback_n=0
d375_real_echo                n=20  value=317.10    target=346.67 PASS  p50=299.30
own_overhead_browser          n=20  p50=3.70 p99=17.30
d375_reconnect_last_screen    n=20  value=13.30     target=8.33   FAIL  p50=2.90
d375_reconnect_network_change n=20  value=1469.60   target=1370   FAIL  p50=1425.80
d375_render_10mbps            n=2816 value=769 fps  target=120    PASS  p99_frame=1.30 ms, idle 3081 fps
d375_first_paint_phone        n=20  value=209.60 ms target=1000   PASS  fcp_missing_n=20 (upper bound used)
  • The frame-bound rows are capped by the venue. Headless Chromium paces paint at 60 Hz. The predicted glyph shows at p50 14.3 ms, about one 60 Hz frame. Every prediction was drawn (overlay 20/20, no fallback), so the time is the wait for the next 60 Hz frame. On this venue a frame-bound row cannot get under 8.3 ms whatever the code does. That applies to predicted glyph, last screen and the browser's own share. Judging them against D375 needs a real 120 Hz display (a ProMotion Mac or phone), or a browser run uncapped, which over-flatters. The gate reports what it measures. This is the owner's or orchestrator's call; I have not changed the profile's browser for it.
  • Reconnect after a network change takes 1470 ms p99, 100 ms over 4 round trips + 50. At p50 1426 it is already about 4.75 round trips, so the resume path costs more than D267 assumed even on the test double (no TLS).
  • Rendering at 10 MB/s: 769 fps p99 here against 34 on loaded compute01 (run 2). The engine fits 8.3 ms with a lot of room on a quiet machine. The compute01 number is CPU contention.
  • First paint: Chromium recorded no first-contentful-paint entry for any of the 20 phone pages. So the row uses the "visible after two frames" upper bound: 210 ms p99 at 4× CPU slowdown.
  • verify-auth-browser with the #252 fix: auth browser: PASS (4 isolated scenarios; Chromium 158.0.8127.18). The recovery check after it needs the tracer's node_modules. This fresh worktree had none yet because the profile installs them later. Rerunning.
Finding 5: the browser rows on a quiet venue. dev-02 (aarch64, 10 cores, load 2.7–3.4), head `94882075f`. Log: `infraternal-dev-02-std:~/.cache/codex-scratch/j407/evidence/verify-latency-rows-aarch64-*.log`. The relay self-test PASSED here: p50 303.6, p99 319.3 ms. ``` d375_predicted_glyph n=20 value=15.00 ms target=8.33 FAIL p50=14.30 overlay_n=20 fallback_n=0 d375_real_echo n=20 value=317.10 target=346.67 PASS p50=299.30 own_overhead_browser n=20 p50=3.70 p99=17.30 d375_reconnect_last_screen n=20 value=13.30 target=8.33 FAIL p50=2.90 d375_reconnect_network_change n=20 value=1469.60 target=1370 FAIL p50=1425.80 d375_render_10mbps n=2816 value=769 fps target=120 PASS p99_frame=1.30 ms, idle 3081 fps d375_first_paint_phone n=20 value=209.60 ms target=1000 PASS fcp_missing_n=20 (upper bound used) ``` - **The frame-bound rows are capped by the venue.** Headless Chromium paces paint at 60 Hz. The predicted glyph shows at p50 14.3 ms, about one 60 Hz frame. Every prediction was drawn (overlay 20/20, no fallback), so the time is the wait for the next 60 Hz frame. On this venue a frame-bound row cannot get under 8.3 ms whatever the code does. That applies to predicted glyph, last screen and the browser's own share. Judging them against D375 needs a real 120 Hz display (a ProMotion Mac or phone), or a browser run uncapped, which over-flatters. The gate reports what it measures. This is the owner's or orchestrator's call; I have not changed the profile's browser for it. - **Reconnect after a network change** takes 1470 ms p99, 100 ms over 4 round trips + 50. At p50 1426 it is already about 4.75 round trips, so the resume path costs more than D267 assumed even on the test double (no TLS). - **Rendering at 10 MB/s:** 769 fps p99 here against 34 on loaded compute01 (run 2). The engine fits 8.3 ms with a lot of room on a quiet machine. The compute01 number is CPU contention. - **First paint:** Chromium recorded no `first-contentful-paint` entry for any of the 20 phone pages. So the row uses the "visible after two frames" upper bound: 210 ms p99 at 4× CPU slowdown. - verify-auth-browser with the #252 fix: `auth browser: PASS (4 isolated scenarios; Chromium 158.0.8127.18)`. The recovery check after it needs the tracer's `node_modules`. This fresh worktree had none yet because the profile installs them later. Rerunning.
kayg commented 2026-11-09 14:11:13 +00:00
Author
Owner
Copy link

Finding 6: echo under a same-session flood stalls on aarch64 (dev-02). The stall predates this branch.

  • With this branch's probe on dev-02 (aarch64 static build of the same Rust as origin/dev; this branch changes no Rust): yes & cat runs in the PTY you type into. After the 64 KiB warm-up, the agent stops sending output on that stream. yes and cat are alive but blocked, the relay threads sit idle, and the key's echo never arrives. Before 174127529 the probe waited forever (15 min in a slot). Periodic stream-0 frames kept resetting the socket timeout. Snapshot: dev-02:~/.cache/codex-scratch/j407/evidence/same-session-stall-aarch64.txt.
  • With bounded waits (174127529) it reproduces as TimeoutError: PTY latency fixture did not reach the Ack and echo of a key on stream 3 within 30 s, with leftover_agent_processes=0. Log: dev-02:~/.cache/codex-scratch/j407/evidence/verify-latency-native2-aarch64-*.log.
  • Control: origin/dev's own scripts/verify-latency.py (c442500c2, no re-attach phase), on the same binary, under timeout 420. It completed baseline and other-session flood (growth 0.07 ms), then hung in the same-session phase: DEV_SCRIPT_EXIT=124. Log: dev-02:~/.cache/codex-scratch/j407/devcheck.log. So the stall is in dev's agent or fixture flow control on aarch64, not in this branch.
  • On x86 (compute01) the same phase completes, but grows echo p99 by 41–148 ms across three runs.
  • I have not debugged it. It is out of this job's scope and needs its own issue: either the agent or shim stops sending one stream's output while keys are pending under a flood (a typing freeze), or the fixture's credit refunds drift. In either case the D375 flood row reads FAIL or MISSING until it is fixed.
Finding 6: **echo under a same-session flood stalls on aarch64 (dev-02). The stall predates this branch.** - With this branch's probe on dev-02 (aarch64 static build of the same Rust as origin/dev; this branch changes no Rust): `yes & cat` runs in the PTY you type into. After the 64 KiB warm-up, the agent stops sending output on that stream. `yes` and `cat` are alive but blocked, the relay threads sit idle, and the key's echo never arrives. Before `174127529` the probe waited forever (15 min in a slot). Periodic stream-0 frames kept resetting the socket timeout. Snapshot: `dev-02:~/.cache/codex-scratch/j407/evidence/same-session-stall-aarch64.txt`. - With bounded waits (`174127529`) it reproduces as `TimeoutError: PTY latency fixture did not reach the Ack and echo of a key on stream 3 within 30 s`, with `leftover_agent_processes=0`. Log: `dev-02:~/.cache/codex-scratch/j407/evidence/verify-latency-native2-aarch64-*.log`. - **Control:** origin/dev's own `scripts/verify-latency.py` (`c442500c2`, no re-attach phase), on the same binary, under `timeout 420`. It completed baseline and other-session flood (growth 0.07 ms), then hung in the same-session phase: `DEV_SCRIPT_EXIT=124`. Log: `dev-02:~/.cache/codex-scratch/j407/devcheck.log`. So the stall is in dev's agent or fixture flow control on aarch64, not in this branch. - On x86 (compute01) the same phase completes, but grows echo p99 by 41–148 ms across three runs. - I have not debugged it. It is out of this job's scope and needs its own issue: either the agent or shim stops sending one stream's output while keys are pending under a flood (a typing freeze), or the fixture's credit refunds drift. In either case the D375 flood row reads FAIL or MISSING until it is fixed.
kayg commented 2026-11-09 16:04:25 +00:00
Author
Owner
Copy link

Finish: branch latency-targets-120hz, head 174127529e786eb6f90d8b799706dc691e45cb71 (git ls-remote origin latency-targets-120hz → 174127529e786eb6f90d8b799706dc691e45cb71 refs/heads/latency-targets-120hz). Base: origin/dev 80a09ad02. Not merged.

D375 rows measured against targets (dev, this branch)

The main venue is dev-02 (aarch64, 10 cores, load 2.7–3.4; relay self-test PASS, p50 303.6 ms). The x86 column is compute01, measured at load 32–50, where the relay itself missed its RTT, so read it as contention.

Measure Target, p99 dev-02 aarch64 Result compute01 x86 (loaded)
Key to letter, with prediction 8.3 ms 15.00 ms (p50 14.3, 20/20 predicted) FAIL 56.90 ms
Real echo round trip + 16.7 = 346.7 ms 317.1 ms (p50 299.3) PASS 384.6 ms
Flood effect on typing +8.3 ms MISSING: the same-session phase stalls (Finding 6, predates this branch) FAIL +129 / +148 / +41 ms over 3 runs
Own overhead, no network 8.3 ms; gate 25 ms until reached 17.97 ms = agent 0.67 + browser 17.30 PASS at the 25 ms baseline, target not met agent 4.37–5.45, browser 34.0
Reconnect, last screen 8.3 ms 13.30 ms (p50 2.9) FAIL 23.85 ms
Reconnect, typable, resumed link 1 round trip + 50 = 380 ms 328.8 ms (p50 300.7) PASS 323–331 ms
Reconnect, typable, after a network change 4 round trips + 50 = 1370 ms 1469.6 ms (p50 1425.8) FAIL 2432 ms
Rendering at 10 MB/s 120 fps 769 fps (p99 frame 1.30 ms) PASS 34 / 17 fps
First paint, mid-range phone < 1 s 209.6 ms (upper bound, no FCP entry recorded) PASS 2610 ms
Idle infraternal agent < 15 MB RSS 6.77 MB PASS 10.02 MB

What the misses mean:

  • Predicted glyph, last screen and the browser's own share are bounded by headless Chromium's 60 Hz paint pacing (p50 14.3 ms is one 60 Hz frame). This venue cannot show 8.3 ms. Judging these rows needs a real 120 Hz display.
  • Reconnect after a network change is genuinely about 0.75 round trip over budget, even with no TLS in the test double.
  • The flood row needs the stall in Finding 6 fixed first.

Evidence:

  • dev-02:~/.cache/codex-scratch/j407/evidence/verify-latency-rows-aarch64-20261109T130134Z.log (browser rows), …/verify-latency-native-aarch64-20261109T131159Z.log and …/verify-latency-native2-aarch64-*.log (native), …/verify-latency-aarch64-combined.log (report);
  • compute01:~/.cache/codex-scratch/j407/evidence/verify-latency-20261109T073942Z.log, …/verify-latency-rows-20261109T143255Z.log.

What changed (11 commits)

  • scripts/latency_targets.py: the D375 table, METRIC name=d375_<row> lines and report <log>, which exits 1 on any missing or failed row. Own overhead gates at OWN_OVERHEAD_BASELINE_MS = 25 until reached.
  • apps/tracer/test/latency-targets.ts: the TypeScript copy, checked against the Python one by scripts/test-latency-report.py (a new fast-gate row).
  • scripts/verify-latency.py: native rows (agent overhead at RTT 0, idle RSS, resumed-link re-attach, flood effect). Budgets moved to 120 Hz. Every wait now has a deadline, and SIGTERM still cleans up.
  • Tracer scenario: predicted glyph 8.3, real echo, last screen, network-change reconnect (4 RTT + 50 replaces RTT × 2 + 300), and the browser's own share. latency-profile.spec.ts adds the rendering (frame limit off) and phone first-paint rows.
  • scripts/verify latency: tees to an evidence log and ends with the D375 table.
  • 60 Hz test budgets moved to 120 Hz: held-output input and theme switch. The switch-spec comment now names the 8.3 ms target.
  • PROCESS.md §6: the D375 table, how rows gate, and the measurement limits.
  • browser-auth/check.mjs: accepts #252's setup_token_command (fixes a dev failure).

Gate

scripts/verify quick (compute01, 0e7b61962): QUICK_EXIT=0, all rows PASS, including tracer typecheck and doc comments on the diff.

scripts/verify fast (compute01, head 174127529), verbatim:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| latency report (D375)     | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | FAIL (101)   |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | FAIL (1)     |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| privileged integration    | SKIP         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
FAST_EXIT=1

Both failures are dev's, not this branch's. git diff origin/dev...HEAD -- crates Cargo.lock deny.toml is empty.

  • cargo-deny: RUSTSEC-2026-0459 (libsqlite3-sys, C SQLite CVE-2026-119375), a new advisory. The same failure is in j428/deny-dev.log (dev), j410, j416 and j417.
  • test: logins::tests::a_linger_request_landing_after_disable_never_survives_revocation failed on a systemctl call count (left 3, right 2) during the parallel workspace run at load 34–50. Rerun alone 3 times at load 50: test result: ok. 1 passed × 3 (code:~/.cache/codex-scratch/j407/linger.txt). This is a timing flake.
  • Static sizes (no Rust change, so the delta vs origin/dev is 0): x86 28,176,912 bytes, aarch64 25,306,336 bytes.

For the orchestrator / owner

  1. The same-session flood stall on aarch64 (Finding 6) needs its own issue.
  2. Should verify latency stop running the whole 1283-test functional suite, which takes about 5 h in one slot, and run only the row-producing specs? This branch keeps the existing step list.
  3. Frame-bound rows need a 120 Hz venue, a real display or the Mac VM. Headless Chromium caps them at about one 60 Hz frame.
  4. The own-overhead ratchet: the agent's share is already 0.67 ms, while the browser's 17.3 ms is mostly the wait for a 60 Hz frame. Lowering the gate from 25 to 8.3 ms waits for a run on a 120 Hz venue.
  5. verify-auth-browser's recovery check fails on dev with AuthError: The request could not be accepted (dev-02). Under load on compute01 it fails with "resident handle resolved wrong user". This branch doesn't touch either path.

Cleanup: target dirs deleted on compute01 (50G) and dev-02 (4.5G), and the leftover agent processes from the stalled run were killed by PID. VM worktrees ~/worktrees/j407 and the evidence are kept.

Finish: branch `latency-targets-120hz`, head **`174127529e786eb6f90d8b799706dc691e45cb71`** (`git ls-remote origin latency-targets-120hz` → `174127529e786eb6f90d8b799706dc691e45cb71 refs/heads/latency-targets-120hz`). Base: origin/dev `80a09ad02`. Not merged. ## D375 rows measured against targets (dev, this branch) The main venue is **dev-02** (aarch64, 10 cores, load 2.7–3.4; relay self-test PASS, p50 303.6 ms). The x86 column is **compute01**, measured at load 32–50, where the relay itself missed its RTT, so read it as contention. | Measure | Target, p99 | dev-02 aarch64 | Result | compute01 x86 (loaded) | |---|---|---|---|---| | Key to letter, with prediction | 8.3 ms | **15.00 ms** (p50 14.3, 20/20 predicted) | FAIL | 56.90 ms | | Real echo | round trip + 16.7 = 346.7 ms | **317.1 ms** (p50 299.3) | PASS | 384.6 ms | | Flood effect on typing | +8.3 ms | **MISSING**: the same-session phase stalls (Finding 6, predates this branch) | FAIL | +129 / +148 / +41 ms over 3 runs | | Own overhead, no network | 8.3 ms; gate 25 ms until reached | **17.97 ms** = agent 0.67 + browser 17.30 | PASS at the 25 ms baseline, target not met | agent 4.37–5.45, browser 34.0 | | Reconnect, last screen | 8.3 ms | **13.30 ms** (p50 2.9) | FAIL | 23.85 ms | | Reconnect, typable, resumed link | 1 round trip + 50 = 380 ms | **328.8 ms** (p50 300.7) | PASS | 323–331 ms | | Reconnect, typable, after a network change | 4 round trips + 50 = 1370 ms | **1469.6 ms** (p50 1425.8) | FAIL | 2432 ms | | Rendering at 10 MB/s | 120 fps | **769 fps** (p99 frame 1.30 ms) | PASS | 34 / 17 fps | | First paint, mid-range phone | < 1 s | **209.6 ms** (upper bound, no FCP entry recorded) | PASS | 2610 ms | | Idle infraternal agent | < 15 MB RSS | **6.77 MB** | PASS | 10.02 MB | What the misses mean: - **Predicted glyph, last screen and the browser's own share** are bounded by headless Chromium's 60 Hz paint pacing (p50 14.3 ms is one 60 Hz frame). This venue cannot show 8.3 ms. Judging these rows needs a real 120 Hz display. - **Reconnect after a network change** is genuinely about 0.75 round trip over budget, even with no TLS in the test double. - **The flood row** needs the stall in Finding 6 fixed first. Evidence: - `dev-02:~/.cache/codex-scratch/j407/evidence/verify-latency-rows-aarch64-20261109T130134Z.log` (browser rows), `…/verify-latency-native-aarch64-20261109T131159Z.log` and `…/verify-latency-native2-aarch64-*.log` (native), `…/verify-latency-aarch64-combined.log` (report); - `compute01:~/.cache/codex-scratch/j407/evidence/verify-latency-20261109T073942Z.log`, `…/verify-latency-rows-20261109T143255Z.log`. ## What changed (11 commits) - `scripts/latency_targets.py`: the D375 table, `METRIC name=d375_<row>` lines and `report <log>`, which exits 1 on any missing or failed row. Own overhead gates at `OWN_OVERHEAD_BASELINE_MS = 25` until reached. - `apps/tracer/test/latency-targets.ts`: the TypeScript copy, checked against the Python one by `scripts/test-latency-report.py` (a new fast-gate row). - `scripts/verify-latency.py`: native rows (agent overhead at RTT 0, idle RSS, resumed-link re-attach, flood effect). Budgets moved to 120 Hz. Every wait now has a deadline, and SIGTERM still cleans up. - Tracer scenario: predicted glyph 8.3, real echo, last screen, network-change reconnect (4 RTT + 50 replaces RTT × 2 + 300), and the browser's own share. `latency-profile.spec.ts` adds the rendering (frame limit off) and phone first-paint rows. - `scripts/verify latency`: tees to an evidence log and ends with the D375 table. - 60 Hz test budgets moved to 120 Hz: held-output input and theme switch. The switch-spec comment now names the 8.3 ms target. - PROCESS.md §6: the D375 table, how rows gate, and the measurement limits. - `browser-auth/check.mjs`: accepts #252's `setup_token_command` (fixes a dev failure). ## Gate `scripts/verify quick` (compute01, `0e7b61962`): `QUICK_EXIT=0`, all rows PASS, including tracer typecheck and doc comments on the diff. `scripts/verify fast` (compute01, head `174127529`), verbatim: ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | Nix Cargo vendoring | PASS | | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | latency report (D375) | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | FAIL (101) | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | FAIL (1) | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | SKIP | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | static builds | PASS | +---------------------------+--------------+ FAST_EXIT=1 ``` Both failures are dev's, not this branch's. `git diff origin/dev...HEAD -- crates Cargo.lock deny.toml` is empty. - **cargo-deny:** `RUSTSEC-2026-0459` (libsqlite3-sys, C SQLite CVE-2026-119375), a new advisory. The same failure is in `j428/deny-dev.log` (dev), `j410`, `j416` and `j417`. - **test:** `logins::tests::a_linger_request_landing_after_disable_never_survives_revocation` failed on a systemctl call count (left 3, right 2) during the parallel workspace run at load 34–50. Rerun alone 3 times at load 50: `test result: ok. 1 passed` × 3 (`code:~/.cache/codex-scratch/j407/linger.txt`). This is a timing flake. - Static sizes (no Rust change, so the delta vs origin/dev is 0): x86 28,176,912 bytes, aarch64 25,306,336 bytes. ## For the orchestrator / owner 1. The **same-session flood stall on aarch64** (Finding 6) needs its own issue. 2. Should `verify latency` stop running the whole 1283-test functional suite, which takes about 5 h in one slot, and run only the row-producing specs? This branch keeps the existing step list. 3. **Frame-bound rows need a 120 Hz venue**, a real display or the Mac VM. Headless Chromium caps them at about one 60 Hz frame. 4. The **own-overhead ratchet:** the agent's share is already 0.67 ms, while the browser's 17.3 ms is mostly the wait for a 60 Hz frame. Lowering the gate from 25 to 8.3 ms waits for a run on a 120 Hz venue. 5. `verify-auth-browser`'s recovery check fails on dev with `AuthError: The request could not be accepted` (dev-02). Under load on compute01 it fails with "resident handle resolved wrong user". This branch doesn't touch either path. Cleanup: target dirs deleted on compute01 (50G) and dev-02 (4.5G), and the leftover agent processes from the stalled run were killed by PID. VM worktrees `~/worktrees/j407` and the evidence are kept.
kayg commented 2026-11-09 16:09:07 +00:00
Author
Owner
Copy link

Round 2 start. Branch latency-targets-120hz, rebased on origin/dev 5eb968744; head 234fb403c.

Plan, following the orchestrator's decisions:

  1. Flood row stays MISSING/FAIL until J444 (#477) lands. No change.
  2. Done in 474a838da: the profile runs only test/terminal.spec.ts and test/latency-profile.spec.ts. The full suite stays in verify fast and the nightly.
  3. Done in 474a838da: the row specs run twice, unthrottled (--disable-gpu-vsync --disable-frame-rate-limit, which gates) and at 60 Hz (reported beside it). A frame_pacing idle_fps check fails the run if the unthrottled run did not get above 70 fps. 234fb403c builds the agent for the host's target, so the profile also runs on dev-02. Next: PROCESS §6 text and a "how to measure on your device" note.
  4. Own-overhead gate stays at 25 ms. I'll propose a value once the unthrottled numbers exist.
  5. Auth recovery check: diagnosing on a VM next (red first if it's the product).
    Then verify quick, one profile run, and push.
Round 2 start. Branch `latency-targets-120hz`, rebased on origin/dev `5eb968744`; head `234fb403c`. Plan, following the orchestrator's decisions: 1. Flood row stays MISSING/FAIL until J444 (#477) lands. No change. 2. Done in `474a838da`: the profile runs only `test/terminal.spec.ts` and `test/latency-profile.spec.ts`. The full suite stays in `verify fast` and the nightly. 3. Done in `474a838da`: the row specs run twice, unthrottled (`--disable-gpu-vsync --disable-frame-rate-limit`, which gates) and at 60 Hz (reported beside it). A `frame_pacing idle_fps` check fails the run if the unthrottled run did not get above 70 fps. `234fb403c` builds the agent for the host's target, so the profile also runs on dev-02. Next: PROCESS §6 text and a "how to measure on your device" note. 4. Own-overhead gate stays at 25 ms. I'll propose a value once the unthrottled numbers exist. 5. Auth recovery check: diagnosing on a VM next (red first if it's the product). Then `verify quick`, one profile run, and push.
kayg commented 2026-11-09 17:05:32 +00:00
Author
Owner
Copy link

Round 2 findings (dev-02 aarch64, load 3–7 during the measured phases, head 0b5bd0f01). Log: dev-02:~/.cache/codex-scratch/j407/evidence/verify-latency-20261109T162528Z.log. This is the real scripts/verify latency, end to end.

Unthrottled mode works: frame_pacing idle 357.0 fps unthrottled against 60.0 fps paced.

Measure Target, p99 Unthrottled (gates) 60 Hz display Result
Key to letter, with prediction 8.3 ms 0.90 ms (p50 0.40) 15.40 ms PASS
Real echo round trip + 16.7 (346.7 ms) 316.2 ms 296.6 ms PASS
Flood effect +8.3 ms MISSING (#477, J444) – FAIL
Own overhead, no network 8.3 ms; gate 25 8.56 ms (agent 0.56 + browser 8.00) 15.26 ms PASS at 25
Reconnect, last screen 8.3 ms 0.10 ms 12.50 ms PASS
Reconnect, resumed link 1 round trip + 50 (380 ms) 327.5 ms – PASS
Reconnect, network change 4 round trips + 50 (1370 ms) 1428.4 ms (p50 1404.7) 1448.3 ms FAIL
Rendering at 10 MB/s 120 fps 667 fps (p99 frame 1.5 ms) 59.5 fps PASS
First paint, phone < 1 s 195.6 ms 247.9 ms PASS
Idle agent < 15 MB 6.84 MB – PASS

D375 REPORT result=FAIL, which is expected: the flood row is MISSING and the network-change reconnect misses its target.

Reading it: once the 60 Hz wait is gone, our own work per key is under 1 ms at p50. The frame-bound rows at 60 Hz (15.4 / 12.5 / 15.3 ms) are almost all waiting for the display's frame.

Own-overhead gate proposal (point 4): lower the gate from 25 ms to 12 ms, and switch it to the 8.3 ms target after two consecutive perf-VM runs at or under 8.3. Evidence: the unthrottled own overhead is 8.56 ms. The agent's share is 0.56 ms (0.67 and 0.90 in earlier dev-02 runs). The browser's share has p99 8.00 ms but p50 0.90. With n=20, the p99 is simply the slowest of 20 keys, so one slow key sets it. 12 ms is the measured 8.56 plus about 40% headroom for that single-sample p99 on a shared VM. It is still half of today's 25 ms gate. I'd also raise the probe from 20 to 100 keys in the latency profile, so the p99 means something. That's a one-line change I have not made; it's the orchestrator's call. Until it's decided, OWN_OVERHEAD_BASELINE_MS stays at 25.

Auth recovery check (point 5): the check, not the product. Fixed on this branch in d224a9109 and the commit before it. A diagnostic run logging cookie names (never values):

DIAG /auth/registration-finish sent=…ceremony,…session-1-… status=200 set=__Host-infraternal-session-2-1796835034,__Host-infraternal-session-1-1796835034(expire)
DIAG /auth/wraps sent=__Host-infraternal-ceremony status=401 set=

Replacing a session correctly answers with two Set-Cookie headers. The checks' Playwright proxy rebuilt the headers with Object.fromEntries(pairs), which kept only the last one (the expiry), so the browser lost its session. The same loss explains check.mjs's "resident handle resolved wrong user". The new fulfillHeaders() joins Set-Cookie values with \n, which Playwright splits back into separate headers, and all 7 fixture proxies use it. Green on d224a9109:

(pass) keeps both cookies of a sign-in that replaces the session
 2 pass
 0 fail
auth browser: PASS (4 isolated scenarios; Chromium 158.0.8127.18)
native recovery same-account live session: PASS (original DEK preserved, capability consumed, replacement passkey signs in)
native recovery no session, enforced delay then elapsed: PASS (original DEK preserved, capability consumed, replacement passkey signs in)
native recovery: PASS (2 real-verifier scenarios; Chromium 158.0.8127.18)
AUTH_EXIT=0

Red, on 0b5bd0f01: DIAG /auth/wraps 401 {"error":"authorization failed"} → RECOVERY_EXIT=1. A fresh checkout also needed the tracer's dependencies before the recovery bundle; verify-auth-browser now installs them first.

One correction to the decisions: verify fast does not run the functional browser suite. Its TypeScript part is unit tests, typecheck, build and the bundle budget, so the whole suite runs only in the nightly (D373). PROCESS §6 says that.

Round 2 findings (dev-02 aarch64, load 3–7 during the measured phases, head `0b5bd0f01`). Log: `dev-02:~/.cache/codex-scratch/j407/evidence/verify-latency-20261109T162528Z.log`. This is the real `scripts/verify latency`, end to end. **Unthrottled mode works:** `frame_pacing` idle 357.0 fps unthrottled against 60.0 fps paced. | Measure | Target, p99 | Unthrottled (gates) | 60 Hz display | Result | |---|---|---|---|---| | Key to letter, with prediction | 8.3 ms | **0.90 ms** (p50 0.40) | 15.40 ms | PASS | | Real echo | round trip + 16.7 (346.7 ms) | 316.2 ms | 296.6 ms | PASS | | Flood effect | +8.3 ms | MISSING (#477, J444) | – | FAIL | | Own overhead, no network | 8.3 ms; gate 25 | **8.56 ms** (agent 0.56 + browser 8.00) | 15.26 ms | PASS at 25 | | Reconnect, last screen | 8.3 ms | **0.10 ms** | 12.50 ms | PASS | | Reconnect, resumed link | 1 round trip + 50 (380 ms) | 327.5 ms | – | PASS | | Reconnect, network change | 4 round trips + 50 (1370 ms) | **1428.4 ms** (p50 1404.7) | 1448.3 ms | FAIL | | Rendering at 10 MB/s | 120 fps | 667 fps (p99 frame 1.5 ms) | 59.5 fps | PASS | | First paint, phone | < 1 s | 195.6 ms | 247.9 ms | PASS | | Idle agent | < 15 MB | 6.84 MB | – | PASS | `D375 REPORT result=FAIL`, which is expected: the flood row is MISSING and the network-change reconnect misses its target. Reading it: once the 60 Hz wait is gone, our own work per key is under 1 ms at p50. The frame-bound rows at 60 Hz (15.4 / 12.5 / 15.3 ms) are almost all waiting for the display's frame. **Own-overhead gate proposal (point 4):** lower the gate from **25 ms to 12 ms**, and switch it to the 8.3 ms target after two consecutive perf-VM runs at or under 8.3. Evidence: the unthrottled own overhead is 8.56 ms. The agent's share is 0.56 ms (0.67 and 0.90 in earlier dev-02 runs). The browser's share has p99 8.00 ms but p50 0.90. With n=20, the p99 is simply the slowest of 20 keys, so one slow key sets it. 12 ms is the measured 8.56 plus about 40% headroom for that single-sample p99 on a shared VM. It is still half of today's 25 ms gate. I'd also raise the probe from 20 to 100 keys in the latency profile, so the p99 means something. That's a one-line change I have **not** made; it's the orchestrator's call. Until it's decided, `OWN_OVERHEAD_BASELINE_MS` stays at 25. **Auth recovery check (point 5): the check, not the product.** Fixed on this branch in `d224a9109` and the commit before it. A diagnostic run logging cookie names (never values): ``` DIAG /auth/registration-finish sent=…ceremony,…session-1-… status=200 set=__Host-infraternal-session-2-1796835034,__Host-infraternal-session-1-1796835034(expire) DIAG /auth/wraps sent=__Host-infraternal-ceremony status=401 set= ``` Replacing a session correctly answers with two `Set-Cookie` headers. The checks' Playwright proxy rebuilt the headers with `Object.fromEntries(pairs)`, which kept only the last one (the expiry), so the browser lost its session. The same loss explains `check.mjs`'s "resident handle resolved wrong user". The new `fulfillHeaders()` joins `Set-Cookie` values with `\n`, which Playwright splits back into separate headers, and all 7 fixture proxies use it. Green on `d224a9109`: ``` (pass) keeps both cookies of a sign-in that replaces the session 2 pass 0 fail auth browser: PASS (4 isolated scenarios; Chromium 158.0.8127.18) native recovery same-account live session: PASS (original DEK preserved, capability consumed, replacement passkey signs in) native recovery no session, enforced delay then elapsed: PASS (original DEK preserved, capability consumed, replacement passkey signs in) native recovery: PASS (2 real-verifier scenarios; Chromium 158.0.8127.18) AUTH_EXIT=0 ``` Red, on `0b5bd0f01`: `DIAG /auth/wraps 401 {"error":"authorization failed"}` → `RECOVERY_EXIT=1`. A fresh checkout also needed the tracer's dependencies before the recovery bundle; `verify-auth-browser` now installs them first. **One correction to the decisions:** `verify fast` does not run the functional browser suite. Its TypeScript part is unit tests, typecheck, build and the bundle budget, so the whole suite runs only in the nightly (D373). PROCESS §6 says that.
kayg commented 2026-11-09 17:13:59 +00:00
Author
Owner
Copy link

Round 2 finish. Branch latency-targets-120hz, head 1da4832c7884b62c1eb3f6b087aede49a459e2bb (git ls-remote gives the same SHA). Rebased on origin/dev 5eb968744. Not merged. No separate fix-auth-recovery-check branch, because the cause was the check, not the product.

Round 2 commits on top of round 1:

  • 474a838da latency profile: row specs only, unthrottled gate, 60 Hz reported beside it, frame_pacing guard
  • 234fb403c verify latency: build the agent for the host's own target
  • 0b5bd0f01 PROCESS §6: row specs only, frame pacing, measuring on your own device
  • d224a9109^ browser-auth: keep every Set-Cookie (fulfillHeaders, 7 proxies, unit test)
  • d224a9109 verify-auth-browser: run the header test, install tracer dependencies first
  • 1da4832c7 fixture-headers.mjs module doc comment

scripts/verify quick (dev-02, 1da4832c7; it compared against origin/dev bd551e549, which dev had moved to by then):

| scope guard               | PASS         |
| migration names           | PASS         |
| Nix Cargo vendoring       | PASS         |
| VT source recipe          | PASS         |
| fmt                       | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| ui literals               | PASS         |
| tracer typecheck          | PASS         |
QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done.
QUICK_EXIT=0

I ran verify fast in round 1 only (on 174127529); this round asked for quick. Linger flake evidence is on #475. dev-02 target dirs deleted after each run.

Round 2 finish. Branch `latency-targets-120hz`, head **`1da4832c7884b62c1eb3f6b087aede49a459e2bb`** (`git ls-remote` gives the same SHA). Rebased on origin/dev `5eb968744`. Not merged. No separate `fix-auth-recovery-check` branch, because the cause was the check, not the product. Round 2 commits on top of round 1: - `474a838da` latency profile: row specs only, unthrottled gate, 60 Hz reported beside it, `frame_pacing` guard - `234fb403c` verify latency: build the agent for the host's own target - `0b5bd0f01` PROCESS §6: row specs only, frame pacing, measuring on your own device - `d224a9109`^ browser-auth: keep every Set-Cookie (`fulfillHeaders`, 7 proxies, unit test) - `d224a9109` verify-auth-browser: run the header test, install tracer dependencies first - `1da4832c7` fixture-headers.mjs module doc comment `scripts/verify quick` (dev-02, `1da4832c7`; it compared against origin/dev `bd551e549`, which dev had moved to by then): ``` | scope guard | PASS | | migration names | PASS | | Nix Cargo vendoring | PASS | | VT source recipe | PASS | | fmt | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | ui literals | PASS | | tracer typecheck | PASS | QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done. QUICK_EXIT=0 ``` I ran `verify fast` in round 1 only (on `174127529`); this round asked for `quick`. Linger flake evidence is on #475. dev-02 target dirs deleted after each run.
kayg commented 2026-11-09 17:17:11 +00:00
Author
Owner
Copy link

Round 3 start. Head e44666d08:

  • 87d41bb3b lowers the own-overhead gate to 12 ms (OWN_OVERHEAD_GATE_MS) and puts the switch rule in PROCESS §6: after two consecutive perf-VM runs at or under 8.3 ms, the gate becomes 8.3 ms.
  • e44666d08 raises the probe to 100 keys in the latency profile and on the perf VM. The functional suite stays at 20 keys.

Now on dev-02, since compute01 has 9 queued: verify latency, then verify fast.

Round 3 start. Head `e44666d08`: - `87d41bb3b` lowers the own-overhead gate to 12 ms (`OWN_OVERHEAD_GATE_MS`) and puts the switch rule in PROCESS §6: after two consecutive perf-VM runs at or under 8.3 ms, the gate becomes 8.3 ms. - `e44666d08` raises the probe to 100 keys in the latency profile and on the perf VM. The functional suite stays at 20 keys. Now on dev-02, since compute01 has 9 queued: `verify latency`, then `verify fast`.
kayg commented 2026-11-09 17:53:47 +00:00
Author
Owner
Copy link

Round 3 progress, head 545bbe744. Two problems surfaced once the probe typed 100 keys, both fixed:

  • 96107b29a: the scenario read the probe results when p99: appeared, but the panel re-renders after every painted key, so it read 49 of 100 echoed keys. It now waits for Painted echoes: N / N.
  • 545bbe744 (product, the ?probe=1 panel): a run had a fixed deadline of max(15 s, 50 ms per key). Each key waits about one round trip, so at 300 ms a run stopped at 45 keys. Red on 96107b29a: Expected substring: "Painted echoes: 100 / 100", received "Painted echoes: 45 / 100". The deadline is now an idle timeout: 15 s with no painted echo. The owner's on-device check (n=100 over a WAN) would have hit the same cut-off.

I stopped the verify fast runs on the two superseded heads, by my own process group. The final run (profile, then fast) is queued on dev-02, since compute01 has 9 queued.

Round 3 progress, head `545bbe744`. Two problems surfaced once the probe typed 100 keys, both fixed: - `96107b29a`: the scenario read the probe results when `p99:` appeared, but the panel re-renders after every painted key, so it read 49 of 100 echoed keys. It now waits for `Painted echoes: N / N`. - `545bbe744` (product, the `?probe=1` panel): a run had a fixed deadline of max(15 s, 50 ms per key). Each key waits about one round trip, so at 300 ms a run stopped at 45 keys. Red on `96107b29a`: `Expected substring: "Painted echoes: 100 / 100"`, received `"Painted echoes: 45 / 100"`. The deadline is now an idle timeout: 15 s with no painted echo. The owner's on-device check (n=100 over a WAN) would have hit the same cut-off. I stopped the `verify fast` runs on the two superseded heads, by my own process group. The final run (profile, then fast) is queued on dev-02, since compute01 has 9 queued.
kayg commented 2026-11-09 18:52:22 +00:00
Author
Owner
Copy link

Round 3 finish. Branch latency-targets-120hz, head 545bbe744fb11161432a90ca71c1eca07d4039cb (confirmed with git ls-remote). Not merged.

Round 3 commits:

  • 87d41bb3b own-overhead gate 12 ms (OWN_OVERHEAD_GATE_MS), plus the switch rule in PROCESS §6: after two consecutive perf-VM runs at or under 8.3 ms, the gate becomes 8.3 ms. Shared-VM runs don't count.
  • e44666d08 100 probe keys in the latency profile and on the perf VM. The functional suite keeps 20.
  • 96107b29a the scenario waits for the whole probe run. It had read the results after the first key.
  • 545bbe744 the probe panel ends a run after 15 s with no echo, instead of at a fixed max(15 s, 50 ms per key) deadline. That deadline cut runs at 300 ms RTT off at 45 keys.

Re-measured with 100 keys (dev-02 aarch64, scripts/verify latency, log dev-02:~/.cache/codex-scratch/j407/evidence/verify-latency-20261109T180354Z.log; frame pacing: unthrottled 356.6 fps, 60 Hz 59.5):

Measure Target, p99 Unthrottled (gates) 60 Hz Result
Own overhead, no network 8.3 ms; gate 12 ms 5.54 ms (agent 0.84 + browser 4.70, n=100) 13.54 ms PASS, target met
Key to letter, with prediction 8.3 ms 1.10 ms (n=100, p50 0.40) 15.50 ms PASS
Real echo round trip + 16.7 ms (346.7) 307.7 ms (n=100) 318.5 ms PASS

With 100 keys the browser's p99 dropped from 8.00 ms (the slowest of 20 keys) to 4.70 ms. Own overhead is now under the 8.3 ms target and near the 5 ms stretch. This is a shared-VM run, so it doesn't count toward the two perf-VM runs the switch rule needs. The rest of the table: flood MISSING (#477), network-change reconnect 1455 ms against 1370 (FAIL), every other row PASS. Overall D375 REPORT result=FAIL, as expected from those two rows.

scripts/verify fast (dev-02, head 545bbe744; compute01 had 9 queued), verbatim:

| Rust gate                 | Result       |
+---------------------------+--------------+
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| latency report (D375)     | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| privileged integration    | SKIP         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| nightly browser suite reporter | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+

==> Compiler metadata generator checks
(fail) a closing notice cannot hide its replacement [1.21ms]

 655 pass
 1 fail
 24814 expect() calls
Ran 656 tests across 102 files. [13.51s]
FAST_EXIT=1

The only failure is apps/tracer/src/canvas/sessions/startup-toast.test.ts › "a closing notice cannot hide its replacement": TypeError: element.getAttribute is not a function in markMotion (packages/ui/src/input-source.ts:83, called from toast.ts under toast-fake-dom.ts). It is dev's, not this branch's. Rerun alone it fails 2 of 2, and input-source.ts, toast.ts, toast-fake-dom.ts and the test are byte-identical to origin/dev 547a48885. This branch changes nothing under packages/ or apps/tracer/src/canvas; its only change in apps/tracer/src is perf-probe.ts. It likely comes from #427's markMotion landing without the toast fake DOM gaining getAttribute. Every Rust row passes, including test and cargo-deny.

Disk: dev-02's j407 target (34G) was deleted when the run finished, and only logs and evidence (2.8M) are kept. df -h / on dev-02: /dev/vda3 158G 84G 68G 56% /.

Round 3 finish. Branch `latency-targets-120hz`, head **`545bbe744fb11161432a90ca71c1eca07d4039cb`** (confirmed with `git ls-remote`). Not merged. Round 3 commits: - `87d41bb3b` own-overhead gate 12 ms (`OWN_OVERHEAD_GATE_MS`), plus the switch rule in PROCESS §6: after two consecutive perf-VM runs at or under 8.3 ms, the gate becomes 8.3 ms. Shared-VM runs don't count. - `e44666d08` 100 probe keys in the latency profile and on the perf VM. The functional suite keeps 20. - `96107b29a` the scenario waits for the whole probe run. It had read the results after the first key. - `545bbe744` the probe panel ends a run after 15 s with no echo, instead of at a fixed max(15 s, 50 ms per key) deadline. That deadline cut runs at 300 ms RTT off at 45 keys. **Re-measured with 100 keys** (dev-02 aarch64, `scripts/verify latency`, log `dev-02:~/.cache/codex-scratch/j407/evidence/verify-latency-20261109T180354Z.log`; frame pacing: unthrottled 356.6 fps, 60 Hz 59.5): | Measure | Target, p99 | Unthrottled (gates) | 60 Hz | Result | |---|---|---|---|---| | Own overhead, no network | 8.3 ms; gate 12 ms | **5.54 ms** (agent 0.84 + browser 4.70, n=100) | 13.54 ms | PASS, target met | | Key to letter, with prediction | 8.3 ms | 1.10 ms (n=100, p50 0.40) | 15.50 ms | PASS | | Real echo | round trip + 16.7 ms (346.7) | 307.7 ms (n=100) | 318.5 ms | PASS | With 100 keys the browser's p99 dropped from 8.00 ms (the slowest of 20 keys) to 4.70 ms. Own overhead is now under the 8.3 ms target and near the 5 ms stretch. This is a shared-VM run, so it doesn't count toward the two perf-VM runs the switch rule needs. The rest of the table: flood MISSING (#477), network-change reconnect 1455 ms against 1370 (FAIL), every other row PASS. Overall `D375 REPORT result=FAIL`, as expected from those two rows. **`scripts/verify fast`** (dev-02, head `545bbe744`; compute01 had 9 queued), verbatim: ``` | Rust gate | Result | +---------------------------+--------------+ | Nix Cargo vendoring | PASS | | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | latency report (D375) | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | SKIP | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | nightly browser suite reporter | PASS | | static builds | PASS | +---------------------------+--------------+ ==> Compiler metadata generator checks (fail) a closing notice cannot hide its replacement [1.21ms] 655 pass 1 fail 24814 expect() calls Ran 656 tests across 102 files. [13.51s] FAST_EXIT=1 ``` The only failure is `apps/tracer/src/canvas/sessions/startup-toast.test.ts` › "a closing notice cannot hide its replacement": `TypeError: element.getAttribute is not a function` in `markMotion` (`packages/ui/src/input-source.ts:83`, called from `toast.ts` under `toast-fake-dom.ts`). It is dev's, not this branch's. Rerun alone it fails 2 of 2, and `input-source.ts`, `toast.ts`, `toast-fake-dom.ts` and the test are byte-identical to origin/dev `547a48885`. This branch changes nothing under `packages/` or `apps/tracer/src/canvas`; its only change in `apps/tracer/src` is `perf-probe.ts`. It likely comes from #427's `markMotion` landing without the toast fake DOM gaining `getAttribute`. Every Rust row passes, including `test` and `cargo-deny`. Disk: dev-02's j407 target (34G) was deleted when the run finished, and only logs and evidence (2.8M) are kept. `df -h /` on dev-02: `/dev/vda3 158G 84G 68G 56% /`.
kayg commented 2026-11-09 19:40:54 +00:00
Author
Owner
Copy link

Review of latency-targets-120hz @ 545bbe744fb11161432a90ca71c1eca07d4039cb (reviewer sub-agent, J459)

Verdict: REQUEST CHANGES. One P2 blocks: the 12 ms own-overhead gate failed in my re-run, with a number five times the author's. The rest is approvable, and the P3s below don't block on their own.

Re-run of scripts/verify latency (item 6)

dev-02 aarch64, through build-slot inside nix develop -c, head 545bbe744, load 1.8–2.3 during the browser phases. Log: dev-02:~/.cache/codex-scratch/latency/evidence/verify-latency-20261109T192313Z.log (table in dev-02:~/.cache/codex-scratch/review447/run2.log). Frame pacing: unthrottled 310.9 fps, 60 Hz 60.0.

Measure Target Mine (gated) Author's round 3 Result
Key to letter 8.3 2.80 (60 Hz 15.20) 1.10 (15.50) PASS
Real echo 346.7 341.80 (60 Hz 320.30) 307.7 (318.5) PASS
Flood effect +8.3 MISSING (TimeoutError … Ack and echo of a key on stream 3 within 30 s) MISSING FAIL, known #477
Own overhead 8.3, gate 12 27.03 (agent 0.73 + browser 26.30; 60 Hz 18.73) 5.54 (0.84 + 4.70) FAIL
Reconnect, last screen 8.3 0.30 0.10 PASS
Reconnect, resumed link 380 324.07 327.5 PASS
Reconnect, network change 1370 1457.50 1455 FAIL, same as the author
Render 10 MB/s 120 fps 714 fps 667 PASS
First paint, phone 1000 188.00 195.6 PASS
Idle agent RSS 15 MB 6.84 6.84 PASS

D375 REPORT result=FAIL. Every row agrees with the author except own overhead.

Findings

P2. The own-overhead gate looks unstable (27.03 vs 5.54 ms). The agent share matches (0.73 vs 0.84 ms). The unthrottled browser p99 is 26.30 ms with p50 0.90 ms, so a couple of slow keys set it (nearest-rank p99 of n=100 is the 99th-smallest sample). In the same run the S1 per-hop p99s are all small (the worst, term.write → write callback, is 2.8 ms, and the hops sum to well under 8 ms). So the browser own-overhead span captures something the per-hop table does not show, or one noisy stretch (the 60 Hz run's write callback p99 was 18.1 ms). Before the 12 ms gate becomes a hard row, please find what the outlier keys are. Also show the browser's slowest samples, not only p50 and p99. The switch rule to 8.3 ms (two perf-VM runs) should wait until the gate is stable on a shared VM.

P3. Items the owner listed, none blocking by itself:

  1. Drift gaps (item 5). The gate does go red when I break one value on a scratch copy. Breaking FIRST_PAINT_MS, RENDER_FPS, RECONNECT_ALLOWANCE_MS or the 30 jitter literal in the TypeScript file, or FIRST_PAINT_MS in the Python file, each turns scripts/test-latency-report.py red, and that test is the "latency report (D375)" row in verify fast. Gaps: PHONE_CPU_SLOWDOWN (TypeScript only), OWN_OVERHEAD_GATE_MS and IDLE_AGENT_RSS_MB (Python only) are not compared. The 30 literal is not checked against the real LatencyConfig jitter. The comparison logic of d375Row() is not checked against row_line().
  2. "Last METRIC wins" (item 4). parse() keeps the last line per row and display, so a rerun inside one log hides an earlier FAIL. By design, but surface a "rerun" count in the report.
  3. 70 fps floor (item 1). The check proves rAF is not tied to 60 Hz (I saw 310.9 fps unthrottled and 60.0 paced). It is measured idle on a mock page, not during the rows. A 120 Hz target would be better served by a floor near 120 plus a loaded measurement. Also, with vsync off the rows omit the 0–8.3 ms wait for the display's frame slot that a real 120 Hz screen adds. The unthrottled numbers show our own work, not display pacing, so the 60 Hz column beside it still matters.

The other checks

  • Report fails correctly (item 4): yes. A synthetic log with the flood row missing and reconnect-after-network-change at 1455 against 1370 prints MISSING and FAIL, then D375 REPORT result=FAIL; an empty log exits 1. verify_latency runs the report after the steps and sets status=1 on a non-zero exit, so a failure cannot be skipped. My real run showed the same: flood MISSING, reconnect FAIL, own overhead FAIL.
  • Probe in production (item 2): safe. The idle timer is re-armed on each painted echo (armIdleTimeout, clearing before arming) and cleared in dispose(), finishRun() and failRun(). No echo means the run ends after 15 s with "Timed out". A closed panel leaves no timer. A run is capped at 5,000 keys.
  • Set-Cookie (item 3): right. fulfillHeaders() joins repeated Set-Cookie with \n, which Playwright splits back into separate headers. The author's red/green runs show the fix, and verify-auth-browser passed in my run (auth browser: PASS (4 isolated scenarios)). No Object.fromEntries on a response-header path remains in scripts/browser-auth. I did not audit every route.fulfill call site elsewhere. Minor scope creep: check.mjs also changes its "public status" assertion.
  • Unthrottled venue (item 1): honest about our own work. See P3 3.
  • Known and not this branch: dev's red startup-toast.test.ts (J453), and the flood stall on aarch64 (#477).

I ran nothing beyond what's listed. Not merged, pushed or deployed.

## Review of `latency-targets-120hz` @ `545bbe744fb11161432a90ca71c1eca07d4039cb` (reviewer sub-agent, J459) **Verdict: REQUEST CHANGES.** One P2 blocks: the 12 ms own-overhead gate failed in my re-run, with a number five times the author's. The rest is approvable, and the P3s below don't block on their own. ### Re-run of `scripts/verify latency` (item 6) dev-02 aarch64, through build-slot inside `nix develop -c`, head `545bbe744`, load 1.8–2.3 during the browser phases. Log: `dev-02:~/.cache/codex-scratch/latency/evidence/verify-latency-20261109T192313Z.log` (table in `dev-02:~/.cache/codex-scratch/review447/run2.log`). Frame pacing: unthrottled 310.9 fps, 60 Hz 60.0. | Measure | Target | Mine (gated) | Author's round 3 | Result | |---|---|---|---|---| | Key to letter | 8.3 | 2.80 (60 Hz 15.20) | 1.10 (15.50) | PASS | | Real echo | 346.7 | 341.80 (60 Hz 320.30) | 307.7 (318.5) | PASS | | Flood effect | +8.3 | **MISSING** (`TimeoutError … Ack and echo of a key on stream 3 within 30 s`) | MISSING | FAIL, known #477 | | Own overhead | 8.3, gate 12 | **27.03** (agent 0.73 + browser 26.30; 60 Hz 18.73) | 5.54 (0.84 + 4.70) | **FAIL** | | Reconnect, last screen | 8.3 | 0.30 | 0.10 | PASS | | Reconnect, resumed link | 380 | 324.07 | 327.5 | PASS | | Reconnect, network change | 1370 | 1457.50 | 1455 | FAIL, same as the author | | Render 10 MB/s | 120 fps | 714 fps | 667 | PASS | | First paint, phone | 1000 | 188.00 | 195.6 | PASS | | Idle agent RSS | 15 MB | 6.84 | 6.84 | PASS | `D375 REPORT result=FAIL`. Every row agrees with the author except own overhead. ### Findings **P2. The own-overhead gate looks unstable (27.03 vs 5.54 ms).** The agent share matches (0.73 vs 0.84 ms). The unthrottled browser p99 is 26.30 ms with p50 0.90 ms, so a couple of slow keys set it (nearest-rank p99 of n=100 is the 99th-smallest sample). In the same run the S1 per-hop p99s are all small (the worst, `term.write → write callback`, is 2.8 ms, and the hops sum to well under 8 ms). So the browser own-overhead span captures something the per-hop table does not show, or one noisy stretch (the 60 Hz run's `write callback` p99 was 18.1 ms). Before the 12 ms gate becomes a hard row, please find what the outlier keys are. Also show the browser's slowest samples, not only p50 and p99. The switch rule to 8.3 ms (two perf-VM runs) should wait until the gate is stable on a shared VM. **P3. Items the owner listed, none blocking by itself:** 1. *Drift gaps (item 5).* The gate does go red when I break one value on a scratch copy. Breaking `FIRST_PAINT_MS`, `RENDER_FPS`, `RECONNECT_ALLOWANCE_MS` or the `30` jitter literal in the TypeScript file, or `FIRST_PAINT_MS` in the Python file, each turns `scripts/test-latency-report.py` red, and that test is the "latency report (D375)" row in `verify fast`. Gaps: `PHONE_CPU_SLOWDOWN` (TypeScript only), `OWN_OVERHEAD_GATE_MS` and `IDLE_AGENT_RSS_MB` (Python only) are not compared. The `30` literal is not checked against the real `LatencyConfig` jitter. The comparison logic of `d375Row()` is not checked against `row_line()`. 2. *"Last METRIC wins" (item 4).* `parse()` keeps the last line per row and display, so a rerun inside one log hides an earlier FAIL. By design, but surface a "rerun" count in the report. 3. *70 fps floor (item 1).* The check proves rAF is not tied to 60 Hz (I saw 310.9 fps unthrottled and 60.0 paced). It is measured idle on a mock page, not during the rows. A 120 Hz target would be better served by a floor near 120 plus a loaded measurement. Also, with vsync off the rows omit the 0–8.3 ms wait for the display's frame slot that a real 120 Hz screen adds. The unthrottled numbers show our own work, not display pacing, so the 60 Hz column beside it still matters. ### The other checks - **Report fails correctly (item 4): yes.** A synthetic log with the flood row missing and reconnect-after-network-change at 1455 against 1370 prints MISSING and FAIL, then `D375 REPORT result=FAIL`; an empty log exits 1. `verify_latency` runs the report after the steps and sets `status=1` on a non-zero exit, so a failure cannot be skipped. My real run showed the same: flood MISSING, reconnect FAIL, own overhead FAIL. - **Probe in production (item 2): safe.** The idle timer is re-armed on each painted echo (`armIdleTimeout`, clearing before arming) and cleared in `dispose()`, `finishRun()` and `failRun()`. No echo means the run ends after 15 s with "Timed out". A closed panel leaves no timer. A run is capped at 5,000 keys. - **Set-Cookie (item 3): right.** `fulfillHeaders()` joins repeated `Set-Cookie` with `\n`, which Playwright splits back into separate headers. The author's red/green runs show the fix, and `verify-auth-browser` passed in my run (`auth browser: PASS (4 isolated scenarios)`). No `Object.fromEntries` on a response-header path remains in `scripts/browser-auth`. I did not audit every `route.fulfill` call site elsewhere. Minor scope creep: `check.mjs` also changes its "public status" assertion. - **Unthrottled venue (item 1): honest about our own work.** See P3 3. - Known and not this branch: dev's red `startup-toast.test.ts` (J453), and the flood stall on aarch64 (#477). I ran nothing beyond what's listed. Not merged, pushed or deployed.
kayg commented 2026-11-09 23:59:09 +00:00
Author
Owner
Copy link

Round 4: where the own-overhead outliers come from (J459 P2). Head 6458fdcb5.

Shape: a tail, not a shifted median. Unthrottled browser share across 7 runs on dev-02: p99 1.6, 16.0, 1.3, 19.6, 1.2, 4.1, 1.8 ms. p50 stays 0.8–1.1 ms every time. J459's 26.3 ms is the same tail, a bit worse. Logs: dev-02:~/.cache/codex-scratch/j407/diag-oldhead.log, diag-run1.log, diag-trace.log, alloc.log.

Which keys: the new own_overhead_browser_slowest line (30f224f2c) shows:

  • Not warm-up: key 0 costs 1.2–1.3 ms in every run.
  • Slow keys come in neighbouring pairs, about 17–22 s into the prediction-off run: keys 64–65 (37.6, 19.6 ms), 71–72 (11.5, 4.1), 68–71 (11.3, 7.4, 6.8, 4.8), 55/61.
  • The time is almost all in one hop, termWrite > writeCallback (34.3, 17.2, 11.0 ms). That's the wait for the engine's next animation frame. Every other hop stays under 3 ms.

What overlapped them (Chromium trace, lean categories):

  • 10 main-thread MajorGC events of 5.2–5.5 ms in the 30 s run, plus heavy background marking.
  • The probe's key-sending timer (the synthetic keydown and the whole input path) inflated from about 0.2 ms to 6.2–7.1 ms twice.
  • An engine render-loop frame of 6.0 ms.
  • No single event accounts for a slow key: GC, key handling and a frame stack up.

Where the garbage comes from: a V8 sampling heap profile over the prediction-off run, including objects already collected:

Unthrottled (~350 fps) 60 Hz
Total 2708 KiB/s 582 KiB/s
TerminalCursorLayer.place 966 22.5
Set (vendored renderer, 2 per frame) 672 16.3
vendored render (closures, objects per frame) 343 13.2
syncCursorLayer 191 under 10

Allocation scales with frame rate. While output flows, the engine's render loop renders every animation frame, and our hook ran syncCursorLayer after every pass, rebuilding a placement object and a key string just to learn the cursor hadn't moved. That's real product cost on any display (2× at 120 Hz against 60 Hz). Uncapped, it's about 3× a 120 Hz screen, which is why it shows so strongly in this venue.

Fixes:

  • Product (44d01de9a): a render pass now skips the cursor-layer sync unless an input it uses changed. The inputs are parsed output, viewport, grid and canvas size, style or theme, and the layer's flags; a forced redraw always resyncs. The check is a few field reads with no allocation or layout read. place() compares fields instead of building a key string. No pacing change.
  • Not changed, proposal: the vendored renderer's per-frame new Set() ×2 and closures (about 1 MB/s uncapped, about 0.35 MB/s at 120 Hz) need a patch in vendor/ghostty-web and a pinned rebuild of dist. That's engine frame work, outside this job. Recommended as its own job with this profile as the gate.
  • Measurement (30f224f2c): the last-screen probe now drops only an attached terminal. One run had failed with {"linkDown":false,…} because the click came before the previous resume's attachment was confirmed.

P3s (6458fdcb5):

  • worst-of-reruns with a "worst of N runs" note;
  • the frame-rate floor is now 120 fps measured while the rows run;
  • drift checks for PHONE_CPU_SLOWDOWN, OWN_OVERHEAD_GATE_MS, IDLE_AGENT_RSS_MB, the jitter literal against the real LatencyConfig, and d375Row() against row_line() line for line under bun.
    scripts/test-latency-report.py: 19 tests OK.

Now running on dev-02: the allocation profile after the fix, the cursor-layer regression specs, verify quick, and the profile ×3.

Round 4: where the own-overhead outliers come from (J459 P2). Head `6458fdcb5`. **Shape: a tail, not a shifted median.** Unthrottled browser share across 7 runs on dev-02: p99 1.6, 16.0, 1.3, 19.6, 1.2, 4.1, 1.8 ms. p50 stays 0.8–1.1 ms every time. J459's 26.3 ms is the same tail, a bit worse. Logs: `dev-02:~/.cache/codex-scratch/j407/diag-oldhead.log`, `diag-run1.log`, `diag-trace.log`, `alloc.log`. **Which keys:** the new `own_overhead_browser_slowest` line (`30f224f2c`) shows: - **Not warm-up:** key 0 costs 1.2–1.3 ms in every run. - **Slow keys come in neighbouring pairs, about 17–22 s into the prediction-off run:** keys 64–65 (37.6, 19.6 ms), 71–72 (11.5, 4.1), 68–71 (11.3, 7.4, 6.8, 4.8), 55/61. - **The time is almost all in one hop, `termWrite > writeCallback`** (34.3, 17.2, 11.0 ms). That's the wait for the engine's next animation frame. Every other hop stays under 3 ms. **What overlapped them (Chromium trace, lean categories):** - 10 main-thread `MajorGC` events of 5.2–5.5 ms in the 30 s run, plus heavy background marking. - The probe's key-sending timer (the synthetic keydown and the whole input path) inflated from about 0.2 ms to 6.2–7.1 ms twice. - An engine render-loop frame of 6.0 ms. - No single event accounts for a slow key: GC, key handling and a frame stack up. **Where the garbage comes from:** a V8 sampling heap profile over the prediction-off run, including objects already collected: | | Unthrottled (~350 fps) | 60 Hz | |---|---|---| | Total | **2708 KiB/s** | **582 KiB/s** | | `TerminalCursorLayer.place` | 966 | 22.5 | | `Set` (vendored renderer, 2 per frame) | 672 | 16.3 | | vendored `render` (closures, objects per frame) | 343 | 13.2 | | `syncCursorLayer` | 191 | under 10 | Allocation scales with frame rate. While output flows, the engine's render loop renders every animation frame, and **our** hook ran `syncCursorLayer` after every pass, rebuilding a placement object and a key string just to learn the cursor hadn't moved. That's real product cost on any display (2× at 120 Hz against 60 Hz). Uncapped, it's about 3× a 120 Hz screen, which is why it shows so strongly in this venue. **Fixes:** - **Product** (`44d01de9a`): a render pass now skips the cursor-layer sync unless an input it uses changed. The inputs are parsed output, viewport, grid and canvas size, style or theme, and the layer's flags; a forced redraw always resyncs. The check is a few field reads with no allocation or layout read. `place()` compares fields instead of building a key string. No pacing change. - **Not changed, proposal:** the vendored renderer's per-frame `new Set()` ×2 and closures (about 1 MB/s uncapped, about 0.35 MB/s at 120 Hz) need a patch in `vendor/ghostty-web` and a pinned rebuild of `dist`. That's engine frame work, outside this job. Recommended as its own job with this profile as the gate. - **Measurement** (`30f224f2c`): the last-screen probe now drops only an attached terminal. One run had failed with `{"linkDown":false,…}` because the click came before the previous resume's attachment was confirmed. **P3s** (`6458fdcb5`): - worst-of-reruns with a "worst of N runs" note; - the frame-rate floor is now 120 fps measured while the rows run; - drift checks for `PHONE_CPU_SLOWDOWN`, `OWN_OVERHEAD_GATE_MS`, `IDLE_AGENT_RSS_MB`, the jitter literal against the real `LatencyConfig`, and `d375Row()` against `row_line()` line for line under bun. `scripts/test-latency-report.py`: 19 tests OK. Now running on dev-02: the allocation profile after the fix, the cursor-layer regression specs, `verify quick`, and the profile ×3.
kayg commented 2026-11-10 04:42:26 +00:00
Author
Owner
Copy link

Round 4 finish. Branch latency-targets-120hz, head 6458fdcb5e3a8da83f3f6f75052c83998d486129 (confirmed with git ls-remote). Not merged.

Three profile runs on dev-02 (aarch64, load 3–7, real scripts/verify latency; logs dev-02:~/.cache/codex-scratch/j407/evidence/run{1,2,3}/ and ~/.cache/codex-scratch/j407/r4.log):

Row Target Run 1 Run 2 Run 3
Own overhead (gate 12) 8.3 5.93 (agent 0.83 + browser 5.10) 3.78 (1.78 + 2.00) 2.49 (0.89 + 1.60)
Key to letter 8.3 2.40 3.10 1.90
Real echo 346.7 291.3 298.8 323.2
Reconnect, last screen 8.3 0.20 0.70 0.10
Reconnect, resumed link 380 325.1 326.9 327.5
Reconnect, network change 1370 1495.1 FAIL 1441.4 FAIL 1447.5 FAIL
Rendering 10 MB/s 120 fps 500 625 455
First paint, phone 1000 346.0 164.5 421.8
Idle agent 15 MB 6.84 6.84 6.84
Flood +8.3 MISSING (#477) MISSING MISSING
Frame rate while rows ran (unthrottled, floor 120) – 2438 3469 2817

The 12 ms gate held in all three runs, with a spread of 2.49–5.93 ms; every run was also under the 8.3 ms target. The slowest browser keys are now 1.7–2.8 ms, against 11–38 ms before the fix. The 60 Hz column (own overhead 7.6–19.5 ms) is still dominated by the wait for a 60 Hz frame. Each run's D375 REPORT result=FAIL comes only from the two known rows.

Allocation after the fix: 1703 KiB/s, from 2708 (−37%). place and syncCursorLayer are gone from the top list. What remains is the vendored renderer: Set 711 and render 381 KiB/s at 3280 frames/s, about 220 bytes per frame, about 26 KiB/s on a 120 Hz screen. That remainder is an artefact of the uncapped venue, not a 120 Hz cost. It is a candidate for its own vendor-patch job, not this one.

Other checks at 6458fdcb5:

  • verify quick: QUICK_EXIT=0.
  • Cursor-layer regression specs (cursor-blink, keyboard-instant, keyboard-sweep, synchronized-output): 45 passed (22.1 m), CURSOR_SPECS_EXIT=0.
  • scripts/test-latency-report.py: 19 tests OK, including the bun-run d375Row against row_line comparison.

Disk: the one dev-02 target was deleted at the end of the run; df -h / shows /dev/vda3 158G 64G 88G 42% /.

Round 4 finish. Branch `latency-targets-120hz`, head **`6458fdcb5e3a8da83f3f6f75052c83998d486129`** (confirmed with `git ls-remote`). Not merged. **Three profile runs on dev-02** (aarch64, load 3–7, real `scripts/verify latency`; logs `dev-02:~/.cache/codex-scratch/j407/evidence/run{1,2,3}/` and `~/.cache/codex-scratch/j407/r4.log`): | Row | Target | Run 1 | Run 2 | Run 3 | |---|---|---|---|---| | **Own overhead** (gate 12) | 8.3 | **5.93** (agent 0.83 + browser 5.10) | **3.78** (1.78 + 2.00) | **2.49** (0.89 + 1.60) | | Key to letter | 8.3 | 2.40 | 3.10 | 1.90 | | Real echo | 346.7 | 291.3 | 298.8 | 323.2 | | Reconnect, last screen | 8.3 | 0.20 | 0.70 | 0.10 | | Reconnect, resumed link | 380 | 325.1 | 326.9 | 327.5 | | Reconnect, network change | 1370 | 1495.1 FAIL | 1441.4 FAIL | 1447.5 FAIL | | Rendering 10 MB/s | 120 fps | 500 | 625 | 455 | | First paint, phone | 1000 | 346.0 | 164.5 | 421.8 | | Idle agent | 15 MB | 6.84 | 6.84 | 6.84 | | Flood | +8.3 | MISSING (#477) | MISSING | MISSING | | Frame rate while rows ran (unthrottled, floor 120) | – | 2438 | 3469 | 2817 | The 12 ms gate held in all three runs, with a spread of 2.49–5.93 ms; every run was also under the 8.3 ms target. The slowest browser keys are now 1.7–2.8 ms, against 11–38 ms before the fix. The 60 Hz column (own overhead 7.6–19.5 ms) is still dominated by the wait for a 60 Hz frame. Each run's `D375 REPORT result=FAIL` comes only from the two known rows. **Allocation after the fix:** 1703 KiB/s, from 2708 (−37%). `place` and `syncCursorLayer` are gone from the top list. What remains is the vendored renderer: `Set` 711 and `render` 381 KiB/s at 3280 frames/s, about 220 bytes per frame, about 26 KiB/s on a 120 Hz screen. That remainder is an artefact of the uncapped venue, not a 120 Hz cost. It is a candidate for its own vendor-patch job, not this one. **Other checks at `6458fdcb5`:** - `verify quick`: `QUICK_EXIT=0`. - Cursor-layer regression specs (`cursor-blink`, `keyboard-instant`, `keyboard-sweep`, `synchronized-output`): **45 passed** (22.1 m), `CURSOR_SPECS_EXIT=0`. - `scripts/test-latency-report.py`: 19 tests OK, including the bun-run `d375Row` against `row_line` comparison. Disk: the one dev-02 target was deleted at the end of the run; `df -h /` shows `/dev/vda3 158G 64G 88G 42% /`.
kayg commented 2026-11-10 09:41:54 +00:00
Author
Owner
Copy link

J468 re-review of latency-targets-120hz @ 6458fdcb5 — verdict: REQUEST CHANGES (one P2, three P3)

Evidence is on code in ~/.cache/codex-scratch/j468/evidence/: verify-latency-20261110T045648Z.log, run.log, spec5.log, spec4.log, cursor-layer-skip.spec.ts, spec2.sh.

1. Cursor-layer skip

I read every input of syncCursorLayer. Cursor position and visibility come from parsed output (parsedOutputVersion). Scrollback offset (viewportY), cols/rows, canvas size, cursor style, theme identity and the six cursorLayerWanted flags are compared. Font, theme and resize call render(..., true), which always resyncs. Focus and blink don't feed the sync. Selection and IME are not inputs.

I wrote cursor-layer-skip.spec.ts. It checks the layer in the frame after each renderer pass, not only after the renderer parks. A parked-renderer check can't see a stale cursor, because pauseTerminal always resyncs after 700 ms. The spec covers wire output (hide/show/move, alternate screen), scrollback up and back, font size, resize, theme, typing with prediction, and selection.

  • Branch: 6 of 7 pass. The one failure is the P2 below.
  • Sensitivity, with inputs removed one at a time:
    • viewportY removed: the scroll test goes red.
    • Output version removed, or all inputs removed: 6 of 7 go red.
    • cols/rows/canvas removed: stays green, because resize is forced anyway.
  • cursor-blink.spec.ts: 6 of 6 pass on the branch.

P2 — local writes bypass the output version, so the layer stays stale (apps/tracer/src/terminal.ts, the terminal.write wrapper near line 432).

  • writeTextForTest, seedPromptForTest and benchmarkOutputAt10MBps call terminal.write directly.
  • Nothing bumps parsedOutputVersion, so every pass skips the sync.
  • After writeText("local one\r\nlocal two") the cursor was at (9,1). The layer stayed at the old cell for about 36 consecutive frames, until the renderer parked.
  • Production output goes through the pump, which does bump the version, so real sessions are fine.
  • This hits the mock and mockup pages the owner reviews.
  • It also makes the render_10mbps row optimistic. The benchmark runs with the cursor-layer sync skipped on every frame, while a real 10 MB/s flood syncs every frame (getCursor() and cellMetrics() still allocate).
  • Fix: bump parsedOutputVersion inside the existing terminal.write wrapper. That covers all three callers. Then add my spec, or a trimmed copy, to the suite.

2. Worst of repeats

  • parse() keeps the worse line per row and display. A FAIL beats a PASS, a lower fps counts as worse, and runs is shown. The unit tests cover this.

P3 — compose_own_overhead keeps the last agent and browser line, not the worst. It's equivalent today, because each is printed once per display per run. Make it worst-wins too.

3. Vacuous passes

  • Missing or skipped rows print MISSING and fail. A 60 Hz-only row can't gate. result defaults to FAIL. n=0 or a non-finite value fails. The unthrottled floor fails when it wasn't measured.
  • The log showed the real case: flood_effect is MISSING and the report FAILs.

P3 — test_typescript_rows_decide_like_python_rows calls skipTest when bun is missing, so the TS-vs-Python check can pass without running.
P3 — the report doesn't enforce a minimum n. A p99 from 5 samples is just the maximum.
P3 — fcp_missing_n=20, so first_paint_phone always uses the two-frame upper bound, never the FCP entry. That's honest, but the row says "first paint".
P3 (not new) — predicted_glyph p50 was −1.20 ms: the prediction stamp precedes the keydown stamp. The formula is unchanged from dev, but the row can under-report.

4. Latency profile re-run on dev-02 (build-slot, aarch64, unthrottled gating)

Unthrottled, from verify-latency-20261110T045648Z.log:

Row Value Gate
predicted_glyph 0.80 ms 8.33
real_echo 327.10 ms 346.67
flood_effect MISSING —
own_overhead 4.09 ms (agent 0.79 + browser 3.30) 12
reconnect_last_screen 0.80 ms 8.33
reconnect_resumed_link 319.41 ms 380
reconnect_network_change 1435.80 ms FAIL 1370
render_10mbps 769 fps 120
first_paint_phone 167.6 ms 1000
idle_agent_rss 6.84 MB 15
  • Frame rate while the rows ran: 4370 fps unthrottled. Idle: 341 fps unthrottled and 60 fps at 60 Hz.
  • The report is FAIL only because of the two known rows. Flood is owned by #477 and reconnect by J448.
  • Own overhead is in line with the author's 5.93, 3.78 and 2.49 ms. The outliers are gone.
  • No animations were touched, so the 300 ms ease-out rule doesn't apply.
  • I didn't push to the branch. The dev-02 worktree and target are deleted.
**J468 re-review of `latency-targets-120hz` @ 6458fdcb5 — verdict: REQUEST CHANGES (one P2, three P3)** Evidence is on `code` in `~/.cache/codex-scratch/j468/evidence/`: `verify-latency-20261110T045648Z.log`, `run.log`, `spec5.log`, `spec4.log`, `cursor-layer-skip.spec.ts`, `spec2.sh`. ### 1. Cursor-layer skip I read every input of `syncCursorLayer`. Cursor position and visibility come from parsed output (`parsedOutputVersion`). Scrollback offset (`viewportY`), cols/rows, canvas size, cursor style, theme identity and the six `cursorLayerWanted` flags are compared. Font, theme and resize call `render(..., true)`, which always resyncs. Focus and blink don't feed the sync. Selection and IME are not inputs. I wrote `cursor-layer-skip.spec.ts`. It checks the layer in the frame after each renderer pass, not only after the renderer parks. A parked-renderer check can't see a stale cursor, because `pauseTerminal` always resyncs after 700 ms. The spec covers wire output (hide/show/move, alternate screen), scrollback up and back, font size, resize, theme, typing with prediction, and selection. - **Branch: 6 of 7 pass.** The one failure is the P2 below. - **Sensitivity, with inputs removed one at a time:** - viewportY removed: the scroll test goes red. - Output version removed, or all inputs removed: 6 of 7 go red. - cols/rows/canvas removed: stays green, because resize is forced anyway. - **`cursor-blink.spec.ts`:** 6 of 6 pass on the branch. **P2 — local writes bypass the output version, so the layer stays stale (`apps/tracer/src/terminal.ts`, the `terminal.write` wrapper near line 432).** - `writeTextForTest`, `seedPromptForTest` and `benchmarkOutputAt10MBps` call `terminal.write` directly. - Nothing bumps `parsedOutputVersion`, so every pass skips the sync. - After `writeText("local one\r\nlocal two")` the cursor was at (9,1). The layer stayed at the old cell for about 36 consecutive frames, until the renderer parked. - Production output goes through the pump, which does bump the version, so real sessions are fine. - This hits the mock and mockup pages the owner reviews. - It also makes the `render_10mbps` row optimistic. The benchmark runs with the cursor-layer sync skipped on every frame, while a real 10 MB/s flood syncs every frame (`getCursor()` and `cellMetrics()` still allocate). - Fix: bump `parsedOutputVersion` inside the existing `terminal.write` wrapper. That covers all three callers. Then add my spec, or a trimmed copy, to the suite. ### 2. Worst of repeats - `parse()` keeps the worse line per row and display. A FAIL beats a PASS, a lower fps counts as worse, and `runs` is shown. The unit tests cover this. **P3** — `compose_own_overhead` keeps the last agent and browser line, not the worst. It's equivalent today, because each is printed once per display per run. Make it worst-wins too. ### 3. Vacuous passes - Missing or skipped rows print MISSING and fail. A 60 Hz-only row can't gate. `result` defaults to FAIL. n=0 or a non-finite value fails. The unthrottled floor fails when it wasn't measured. - The log showed the real case: `flood_effect` is MISSING and the report FAILs. **P3** — `test_typescript_rows_decide_like_python_rows` calls `skipTest` when bun is missing, so the TS-vs-Python check can pass without running. **P3** — the report doesn't enforce a minimum n. A p99 from 5 samples is just the maximum. **P3** — `fcp_missing_n=20`, so `first_paint_phone` always uses the two-frame upper bound, never the FCP entry. That's honest, but the row says "first paint". **P3 (not new)** — `predicted_glyph` p50 was −1.20 ms: the prediction stamp precedes the keydown stamp. The formula is unchanged from dev, but the row can under-report. ### 4. Latency profile re-run on dev-02 (build-slot, aarch64, unthrottled gating) Unthrottled, from `verify-latency-20261110T045648Z.log`: | Row | Value | Gate | |---|---|---| | predicted_glyph | 0.80 ms | 8.33 | | real_echo | 327.10 ms | 346.67 | | flood_effect | MISSING | — | | own_overhead | **4.09 ms** (agent 0.79 + browser 3.30) | 12 | | reconnect_last_screen | 0.80 ms | 8.33 | | reconnect_resumed_link | 319.41 ms | 380 | | reconnect_network_change | **1435.80 ms FAIL** | 1370 | | render_10mbps | 769 fps | 120 | | first_paint_phone | 167.6 ms | 1000 | | idle_agent_rss | 6.84 MB | 15 | - Frame rate while the rows ran: 4370 fps unthrottled. Idle: 341 fps unthrottled and 60 fps at 60 Hz. - The report is FAIL only because of the two known rows. Flood is owned by #477 and reconnect by J448. - Own overhead is in line with the author's 5.93, 3.78 and 2.49 ms. The outliers are gone. - No animations were touched, so the 300 ms ease-out rule doesn't apply. - I didn't push to the branch. The dev-02 worktree and target are deleted.
kayg commented 2026-11-10 09:57:52 +00:00
Author
Owner
Copy link

Round 5 (J468's re-review), head af543ecd6.

P2, local writes bypass the output version: fixed in 066b9f0bc. The terminal.write wrapper bumps parsedOutputVersion, which covers writeTextForTest, seedPromptForTest and benchmarkOutputAt10MBps. The pump's own bump stays; every reader compares with !==. J468's cursor-layer-skip.spec.ts is in the branch; af543ecd6 adds a theme-lookup guard so it passes the tracer typecheck.

  • Red on 6458fdcb5 (compute01): ✘ a local write (mock helper, no wire sequence) moves the layer, Received + 562.
  • Green on af543ecd6 (dev-02): 7 passed (1.9 m).

render_10mbps, re-measured (dev-02, load 0.1–2.8, two runs per head):

Head Unthrottled p99 fps (p99 frame) 60 Hz
6458fdcb5 (sync skipped on local writes) 417 (2.4 ms), 588 (1.7 ms) 59.5, 59.5
af543ecd6 (fixed) 526 (1.9 ms), 556 (1.8 ms) 59.5, 59.5

The difference is inside run-to-run noise. The benchmark writes 100 chunks a second, so the fix adds about 100 syncs a second, not one per frame. The optimism was real in principle and too small to measure. A compute01 run at load 19–27 gave 16 fps on the old head, so render numbers from that host aren't usable.

P3s, fixed (066b9f0bc):

  • compose_own_overhead keeps the slower share per display;
  • each row has a min_n (100 keys or frames or flood samples, 20 reconnects or pages, 2 memory readings) and fails as "FAIL (n X < min)" below it, which PROCESS §6 now states;
  • the TS-vs-Python row test fails, rather than skips, without bun.
    scripts/test-latency-report.py: 21 OK.

Follow-ups, noted and not fixed:

  • first_paint_phone never uses the FCP entry. Every run shows fcp_missing_n=20, so the row is always the "visible after two frames" upper bound. Something about the CPU-throttled mobile context (or the route-fulfilled page) keeps Chromium from emitting first-contentful-paint, or the buffered observer can't see it. Needs its own look.
  • predicted_glyph shows a negative p50 in some runs (J468). The overlay's paint stage can be stamped before keydown (likely the rAF timestamp, which is the frame's start time, against performance.now() at keydown), so predicted - started goes below zero. The row should clamp at 0, or stamp the paint with performance.now(). Not changed here.

verify fast is now running on compute01; its output follows.

Round 5 (J468's re-review), head `af543ecd6`. **P2, local writes bypass the output version: fixed** in `066b9f0bc`. The `terminal.write` wrapper bumps `parsedOutputVersion`, which covers `writeTextForTest`, `seedPromptForTest` and `benchmarkOutputAt10MBps`. The pump's own bump stays; every reader compares with `!==`. J468's `cursor-layer-skip.spec.ts` is in the branch; `af543ecd6` adds a theme-lookup guard so it passes the tracer typecheck. - **Red** on `6458fdcb5` (compute01): `✘ a local write (mock helper, no wire sequence) moves the layer`, `Received + 562`. - **Green** on `af543ecd6` (dev-02): **7 passed** (1.9 m). **render_10mbps, re-measured** (dev-02, load 0.1–2.8, two runs per head): | Head | Unthrottled p99 fps (p99 frame) | 60 Hz | |---|---|---| | `6458fdcb5` (sync skipped on local writes) | 417 (2.4 ms), 588 (1.7 ms) | 59.5, 59.5 | | `af543ecd6` (fixed) | 526 (1.9 ms), 556 (1.8 ms) | 59.5, 59.5 | The difference is inside run-to-run noise. The benchmark writes 100 chunks a second, so the fix adds about 100 syncs a second, not one per frame. The optimism was real in principle and too small to measure. A compute01 run at load 19–27 gave 16 fps on the old head, so render numbers from that host aren't usable. **P3s, fixed** (`066b9f0bc`): - `compose_own_overhead` keeps the slower share per display; - each row has a `min_n` (100 keys or frames or flood samples, 20 reconnects or pages, 2 memory readings) and fails as "FAIL (n X < min)" below it, which PROCESS §6 now states; - the TS-vs-Python row test fails, rather than skips, without bun. `scripts/test-latency-report.py`: 21 OK. **Follow-ups, noted and not fixed:** - `first_paint_phone` never uses the FCP entry. Every run shows `fcp_missing_n=20`, so the row is always the "visible after two frames" upper bound. Something about the CPU-throttled mobile context (or the route-fulfilled page) keeps Chromium from emitting `first-contentful-paint`, or the buffered observer can't see it. Needs its own look. - `predicted_glyph` shows a negative p50 in some runs (J468). The overlay's paint stage can be stamped before `keydown` (likely the rAF timestamp, which is the frame's start time, against `performance.now()` at keydown), so `predicted - started` goes below zero. The row should clamp at 0, or stamp the paint with `performance.now()`. Not changed here. `verify fast` is now running on compute01; its output follows.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
j470b-server-mesh-rejoin
mesh-compat-382
j444b-d381-key-wins-flood
j461b-macos-utun-reader-thread
j475b-remove-host-motion
fix-reconnect-new-link
j481-d380
dev
j479-mesh-soak
fix/j476-host-facts
j475-motion-pill-split
j470-mac-node-join
root-upgrade-d376
j474-review-merge
latency-targets-120hz
nightly-suite-shards
gotatun-pin-j469
macos-utun-reader-thread
fix-flood-echo-stall
wip/j416
motion-fast-ease-out
mesh-addhost-j259
takeover-close-terminals-d368
kbd-instant-followups
fix-trust-key-double-click
nightly-suite-partial-report
appwin-icons-j207
rusqlite-slice1
swipe-content-first-d374
mesh-mtu-381
addhost-instant-open
glass-contrast-d365-mock
glass-contrast-d365
signout-followups-442
fix-startup-toast-red
research-gotatun-matrix-j317
themes-sync-j304
fix-load-flakes
fix-nix-zig-patch
fix-compute01-browser-baseline
ratchet-size-baseline-419
predict-auto-d130
revoke-coverage-d367
nightly-suite-queue-timeout
nightly-browser-suite
fix-rustsec-2026-0459
fonts-ioskeley-obel
log-retention-d375
fix-kbd-no-anim-427
fix-dev-red-specs-2
takeover-proof-305-j205
wip/j410-server-pump
ui-consistency-d372
fix-osc52-clipboard
libghostty-vt-1.0
forget-retired-host-d369
j404-merge-426
fix-setup-lock-race
fix-touch-select-426
revoke-access-report-ui
fix-predictive-echo-specs
fix-selinux-agent-exec
wip/j395-checkpoint2
wip/j395
j391-review-scratch
ui-consistency-j195-deferred-rebased
fix-rustc-paths-421
decisions-wording-341
fix-install-suite-red
fix-dev-rtt-specs
fix-pending-removal-align
fix-privileged-skip
fix-install-suite-404
review-j368
mesh-dns-j254
revoke-access-report-d356b
revoke-privilege-round5
fix-removal-confirm
review-j360
gotatun-rotation-367
deps-refresh-340
review-j354
review-J351
review-j350
addhost-takeover-d359
revoke-report-items-347
revoke-end-sessions-d362
review-j348
size-mesh-ssh-346
size-trim-337
revoke-access-diff-d356
j259-pre-rebase-b06c5a6
j305-ended-at-start-spec
addhost-sudo-consent
fix-layout-real-specs-411
fix-shell-open-411
fix-console-hosts-373
fix-shell-exit
fuzz-host-name-353
fix-host-details-410
macos-filter-j197
research-gotatun-tricks
revoke-p3-j216l
stale-state-pill
revoke-followups-j216k
revoke-progress-j216e
backup/macos01-20261106/lume/j197
backup/macos01-20261106/lume/appwin-icons-j207-stash0
backup/macos01-20261106/claude/stash-0
backup/macos01-20261106/claude/stash-1
backup/macos01-20261106/claude/J80-macos-s1
backup/macos01-20261106/claude/J80
backup/macos01-20261106/claude/J80-slice1-20261102
backup/macos01-20261106/claude/J16b-appwin-on-dev
backup/macos01-20261106/claude/appwin-launch-idle-146
backup/macos01-20261106/claude/J80-s1-run
wip/j259
trust-285-p3c
runtime-lock-j216f
mesh-ssh-p3-j175
tun-proof-p3b
trust-285-p3
terminal-polish-j271
tun-proof-p3
ssh-install-flakes-j216i
fuzz-fixes-j264
trust-285-a
trust-285-b
restart-oversized-fixture-j198
backup/macos01-20261106/lume/appwin-icons-j207
restart-bound-j216h
up-preflight-p3-j198
backup/macos01-20261106/lume/j251
up-preflight-j198
wip/up-preflight-j198-j245-p2
wip/up-preflight-j198-j245-p1
agent-fsync-offloop-j196
wip/up-preflight-j198-j245
settings-offscreen-j231
wip/up-preflight-j198-j226-inspection
wip/j103f
themes-j206
wip/up-preflight-j198-j226
flakes-j216c
backup/macos01-20261106/claude/j197
flakes-j216b
backup/macos01-20261106/claude/j197-red
wip/up-preflight-j198-review
flakes-j216
backup/macos01-20261106/claude/appwin-icons-j207
engineering-j208b
appwin-mock-j209
toast-stack-j165
engineering-j208
decisions-d351
decisions-d350
decisions-d349
appearance-presets-mock-j129
decisions-d348
docs-audit-spec
doc-coverage-baseline-raise
decisions-d347
decisions-d346b
decisions-d346
j88h-profile
ui-consistency-j195-deferred
ui-consistency-j195
engineering-practices-j194
preferred-login-ui-j166
signin-pools-263-j174
macos-node-265-s1
decisions-d345
decisions-d344
proxy-cgroup-j164
verify-quick-fuzz-329
proxy-start-io-j168
timing-flakes-j163
up-263-s1
mcp-events-j147
verify-quick-fuzz-j158
proxy-p3-j159
agent-resource-j157-detect
wip/j103d-final
remote-cancel-hang-j155
proxy-traefik-281-j62d
j149-verify-quick
proxy-traefik-281-j62b
j150-drop-unnamed
wip/j79d-red-r4
wip/j103d-d332
rootless-toast-j142
wip/j103d-head
wip/j103d
j125c-mcp
macos-build-j136
backup/macos01-20261106/claude/j136
wip/j79d-red-cli
wip/j79d-red-server
j121-link-liveness
j118c-tracer-failures
wip/j79c-head
release-desktop-j131
j124c-signed-macos
wip/j79c-red-p3
wip/j79c-red-p23
wip/j79c-red-p22
merge-j128-check
perf-deferred-j128
sidebar-toggle-260
j127-shortcut-changed-dot
j125-mcp
j118-tracer-failures
j126-account-summary
settings-sections-mock-j92
j120-pinch-lock
sidebar-292
backup/macos01-20261106/claude/j120
taste-2026-11-03
row-actions-242-215
backup/macos01-20261106/claude/stash-0-untracked
backup/macos01-20261106/claude/J80-265-s1-run
backup/macos01-20261106/claude/J80-macos-s1-20261102
j112-canvas-commands
ui-no-overshoot-scroll
fix-recovery-restart
backup/macos01-20261106/claude/stash-1-untracked
taste-2026-11-02
size-baseline-ratchet
machine-cards-structure-j108
j109-boot-copy-mockups
fix-dev-red
wip/up-263-s1
j95b-settings-rows
proxy-traefik-281
j95-option-rows
docs-human-j91
docs-keys-palette
fix-flood-1013-270-j38d
mesh-ssh-s1
mesh-ssh-s0
addhost-d291-tunnel
test-stability-load
mesh-relay-220
repro-296
shim-thp-off-24
mesh-pins-258-v2
mesh-dns
signin-d311-merged
docs-pages
ui-size-283
fix-ui-batch-specs
wip/j12b-edge-update
gate-on-arm
fix-rename-sync-293
trust-285-j60b
fix-focus-ring-clip
integ-ui-1101
edge-update-v2
signin-d311
taste-2026-11-01
layout-per-tab-250
ui-approved-splitopt-224
ui-approved-splitopt-224-on-dev
icon-text-centre-282
ui-approved-189-212-216
research-reverse-proxy
transfer-budget-320
review-mesh-pins-258-r2
sidebar-pill-259
review-mesh-relay-220
wip/j12-edge-update-v2
nix-package
status-crash-loop-237
fix-dev-test-failures
fix-fixture-socket-path
wip/mesh-relay-220-pre-dev-advance-rebase
fix-canvas-side-clip
research-head-to-head
frosted-real-256
review-mesh-pins-258
wip/mesh-relay-220-pre-final-rebase
review-layout-per-tab-250-r2
broad-suite-triage
nix-devshell
review-status-crash-loop-237-r3
wip/nix-devshell-pre-p3
taste-frosted-real
wip/code-serve-demo-2026-10-30
wip/code-serve-glass-2026-10-30
wip/code-serve-clusters-2026-10-30
wip/code-merge-release94-dev-2026-10-30
wip/rproxy-delta-scratch-2026-10-30
wip/stale-note-257-staged-2026-10-30
wip/d241-size-staged-2026-10-30
review-broad-suite-207
wip/mesh-pins-258-pre-split
wip/mesh-relay-220-pre-rebase
mesh-measure-220
plan-211-update
review-edge-autoupdate-r3
perf24-reconnect
review-stale-note-r2
release-prod-live
d241-shell-marker
review-mesh-pins-server-258
review-revocation-r3
review-setup-token-r2
review-sync-journal-r3
wip/review-mesh-pins-258-prune-review-1995fee46
frosted-scroll-256
sync-journal-r3
revocation-per-account-wip
review-tokio-asyncfd-0386
review-sync-p3s
wip/stash-8by64-stash0-2026-10-30
mesh-drag-r2
setup-token-data-wip
release-9-4
auth-busy-1013
mesh-view-overlap
sync-p3s
review-auth-busy-wake
review-readd-r8-wip
mesh-drag-bounds
audit-auth-93
app-windows-stream-start
review-authority-cap-r3
app-windows-r7-measure
readd-minimal-r6
readd-minimal-r5
app-windows-r7-soak
readd-minimal-r4
app-windows-r7
app-windows-r6
readd-minimal-r3
readd-minimal-r2
readd-minimal
merge-release8-dev
review-busy-page-liveness-r2
review-busy-page-liveness
mesh-view-trim
review-authority-read-retry-r2
review-remove-uninstall-r2
release-retry-reattach
release-retry
review-remove-uninstall
review-authority-read-retry
app-windows-ship
mesh-autoswitch-p2
mesh-ssh
mesh-relay
mesh-macos-node
screen-and-control
session-labels
review-real-logins-r5
review-real-logins-r4
review-real-logins-r3
shim-memory-r2
review-real-logins-r2
shim-memory
shadow-addhost-instant
shadow-real-logins
shadow-same-session-flood
shadow-prod-perf-fix
hosts-from-server
app-windows-real
app-windows-transport
release-prod-6w
mesh-policy-verify
quiet-close-toast
dev-merge-r4
review-mesh-agent-transport
review-app-windows-transport
settings-roundtrip
review-release-pipeline-r4
mesh-agent-transport
ended-strip-fix
review-audit-ca-update-d244
settings-rows
phone-width
signout-clear
fix-restart-tests
dev-merge-rp
signer-hardening
review-audit-ca-update
terminal-fidelity
review-audit-ca-update-wip
mesh-integrate
review-mesh-agent-transport-515d3712
review-mesh-dataplane-feec1988
mesh-dataplane
signout
research-canvas-node
wip/review-renderer-cost
wip/review-terminal-fidelity
padlock-glyph
review-release-pipeline-2e95b689
phone-polish
sidebar-polish
inbox-generic
review-mesh-dataplane-378d4055
prod-batch
ui-inbox
ui-ca-unlock
ui-machine-sections
review-mesh-dataplane-a5f744fb
tls-session-prod
session-bounds-prod
fix-dead-session
session-bounds
tls-session
palette-polish
shortcuts-d222
research-appmeta
phone-gutter
ui-centre-button
wip/ui-centre-button
wip/tls-session
review-addhost-breakit
ui-platform
renderer-cost
remote-agents
mesh-api
ssh-bootstrap
prod-hotfix2
prod-fixes
review-remote-agents
review-mesh-dataplane
session-life
research-browser
review-mesh-api
research-capture
review-mesh-api-round3
review-mesh-api-round1
gate-stability
review-gate-stability
review-relay-direct
review-mesh-api-findings
proto-sessions
font-picker
proto-addhost
proto-windows
mesh-core
latency-gate
research-bsd
research-discovery
keytray-mods
mock-core
ui-tweaks
fix-motion-sel
fix-motion
integrate2
fix-robust
t-web
t-server
vt-ghostty
fix-snap-echo
t-web-auth
review-vt2
t-agent
lane-protocol
lane-shim
lane-auth
s1-base
s1-contracts
s1-workspace
No results found.
Labels
Clear labels   
launch-qa
Findings from launch QA.

No labels
launch-qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/infraternal#447
No description provided.