Tests: high latency by default + input-to-photon gate (D130) #55

Closed
opened 2026-10-07 07:35:56 +00:00 by kayg · 17 comments
Owner

D130: the default test environment runs with high latency (a 300 ms+ round trip via netem or CDP emulation) for the browser suite and the native end-to-end checks, so every run exposes slow protocol and app paths. Add an input-to-photon gate (typed character visible within 1 frame with prediction; echo within RTT + 1 frame; a flood must not delay echo). Regressions block merges.

D130: the default test environment runs with high latency (a 300 ms+ round trip via netem or CDP emulation) for the browser suite and the native end-to-end checks, so every run exposes slow protocol and app paths. Add an input-to-photon gate (typed character visible within 1 frame with prediction; echo within RTT + 1 frame; a flood must not delay echo). Regressions block merges.
Author
Owner

Starting on branch latency-gate from dev base cecc7ecd7b. Issue #55 has no comments; parent S1 issue #5 has one comment recording round 1 and DECISIONS 48–53.

Starting on branch latency-gate from dev base cecc7ecd7bb452229fcf292bebde0917c0568522. Issue #55 has no comments; parent S1 issue #5 has one comment recording round 1 and DECISIONS 48–53.
Author
Owner

Correction to my start note: the related parent is #52 (11 comments, all read), not #5. Its prior measurement found CDP did not shape established WebSocket traffic (100 ms configured, 2–10 ms Hello/Welcome and Attach/Output); the follow-up used a test-only bidirectional WebSocket bridge and measured ~206–245 ms at 100 ms each way. I will use a stream proxy for the browser leg.

Correction to my start note: the related parent is #52 (11 comments, all read), not #5. Its prior measurement found CDP did not shape established WebSocket traffic (100 ms configured, 2–10 ms Hello/Welcome and Attach/Output); the follow-up used a test-only bidirectional WebSocket bridge and measured ~206–245 ms at 100 ms each way. I will use a stream proxy for the browser leg.
Author
Owner

Finding: reconnect-to-resumed is 1038.70 ms at configured RTT 300 ms (jitter ±30), exceeding the 900 ms RTT×2+300 budget. Production tracer test output: METRIC name=reconnect_resumed source=tracer-production-build n=1 p50_ms=1038.70 p99_ms=1038.70 budget_ms=900 result=FAIL. Filed as #61; product changes are out of scope for this lane.

Finding: reconnect-to-resumed is 1038.70 ms at configured RTT 300 ms (jitter ±30), exceeding the 900 ms RTT×2+300 budget. Production tracer test output: METRIC name=reconnect_resumed source=tracer-production-build n=1 p50_ms=1038.70 p99_ms=1038.70 budget_ms=900 result=FAIL. Filed as #61; product changes are out of scope for this lane.
Author
Owner

Finding: at 300 ms RTT, cold setup-screen render p99 was 1664.10 ms and first-contentful-paint p99 was 1582.30 ms on infraternal-dev (8 cores, load 6.29/7.97/7.09), exceeding the existing 1000 ms local gate. Filed separately as #62; the run was desktop Chromium, not a mid-range phone.

Finding: at 300 ms RTT, cold setup-screen render p99 was 1664.10 ms and first-contentful-paint p99 was 1582.30 ms on infraternal-dev (8 cores, load 6.29/7.97/7.09), exceeding the existing 1000 ms local gate. Filed separately as #62; the run was desktop Chromium, not a mid-range phone.
Author
Owner

Finding: production tracer keystroke→echo confirmation measured p50 323.90 ms / p99 338.00 ms (n=20) against 334 ms (300 RTT + 2×17 ms). Glyph visible measured p50 330.80 / p99 357.20 ms and is report-only until #50 lands. The host was loaded at 10.19/10.22/8.16 on 8 cores. Filed as #63.

Finding: production tracer keystroke→echo confirmation measured p50 323.90 ms / p99 338.00 ms (n=20) against 334 ms (300 RTT + 2×17 ms). Glyph visible measured p50 330.80 / p99 357.20 ms and is report-only until #50 lands. The host was loaded at 10.19/10.22/8.16 on 8 cores. Filed as #63.
Author
Owner

Second corrected-proxy run confirmed reconnect→resumed at 1041.40 ms and echo-confirmed p99 at 338.00 ms (budget 334). The first run over-estimated WebSocket frame delay because jitter was resampled per TCP recv chunk; the relay now samples once per TCP flow, then preserves that delay across stream chunks.

Second corrected-proxy run confirmed reconnect→resumed at 1041.40 ms and echo-confirmed p99 at 338.00 ms (budget 334). The first run over-estimated WebSocket frame delay because jitter was resampled per TCP recv chunk; the relay now samples once per TCP flow, then preserves that delay across stream chunks.
Author
Owner

The relay now samples jitter once per direction per 200 ms quiet-to-active burst, reusing it across arbitrary TCP chunks in one WebSocket frame. Its 31-sample self-test at RTT 300/jitter 30 measured p50 308.80 ms, p99 324.03 ms. With that model, the latest production run measured reconnect 996.70 ms (budget 900), echo confirmation p99 359.90 ms (budget 334), and glyph visible p99 369.90 ms (report-only); host load was 7.00/7.52/7.43 on 8 cores.

The relay now samples jitter once per direction per 200 ms quiet-to-active burst, reusing it across arbitrary TCP chunks in one WebSocket frame. Its 31-sample self-test at RTT 300/jitter 30 measured p50 308.80 ms, p99 324.03 ms. With that model, the latest production run measured reconnect 996.70 ms (budget 900), echo confirmation p99 359.90 ms (budget 334), and glyph visible p99 369.90 ms (report-only); host load was 7.00/7.52/7.43 on 8 cores.
Author
Owner

Final relay access-screen measurement: setup render p50 1442.90 / p99 1955.50 ms; first-contentful-paint p50 1221.40 / p99 1733.80 ms (n=30), against the existing 1000 ms render gate. Host load 19.18/11.15/8.71 on 8 cores. The test timeout is increased so the sign-in group can finish too.

Final relay access-screen measurement: setup render p50 1442.90 / p99 1955.50 ms; first-contentful-paint p50 1221.40 / p99 1733.80 ms (n=30), against the existing 1000 ms render gate. Host load 19.18/11.15/8.71 on 8 cores. The test timeout is increased so the sign-in group can finish too.
Author
Owner

Harness finding: the first native latency stage failed before measurement because the scratch state path produced a 114-byte AF_UNIX session socket path, beyond sun_path; the agent returned terminal-open-refused. Reproduction with a short per-run state root produced a 100-byte socket path and opened 3/3 real PTY sessions at INFRATERNAL_TEST_RTT=0. I’m routing native fixture state under the short per-job scratch root and will rerun the native profile.

Harness finding: the first native latency stage failed before measurement because the scratch state path produced a 114-byte AF_UNIX session socket path, beyond sun_path; the agent returned terminal-open-refused. Reproduction with a short per-run state root produced a 100-byte socket path and opened 3/3 real PTY sessions at INFRATERNAL_TEST_RTT=0. I’m routing native fixture state under the short per-job scratch root and will rerun the native profile.
Author
Owner

Full scripts/verify latency run on infraternal-dev (x86_64, 8 cores; profile loads: setup 3.39 6.72 10.07, native 4.57 6.87 10.08, auth-browser 14.80 9.22 10.58, tracer-browser 17.73 10.48 10.96; TRACER_TEST_PORT=4621). Proxy self-test n=31: RTT p50=305.20 ms, p99=331.02 ms, PASS; auth-browser passed 4 Chromium scenarios. Native n=100 baseline PTY echo p99=329.47 ms, other-session flood p99=302.45 ms (growth −27.02 ms, PASS); same-session flood p99=449.58 ms (growth +120.11 ms, FAIL; new finding #73). Production browser: sign-in usable p99=4987.10 ms (PASS); reconnect p99=1012.70 ms/900 ms (FAIL); confirmed echo p99=371.40 ms/334 ms (FAIL); glyph p99=381.00 ms (report); setup/sign-in render p99=1509.70/1839.10 ms (FAIL). Full Playwright suite: 29 passed, 3 failed in 9.2 min; failures were the motion audit (two 62–63 ms main-thread long tasks), reconnect/echo budget assertions, and access-render budget assertion. The earlier setup-token socket hang-up did not reproduce. Full log: $HOME/.cache/codex-scratch/latency-gate/latency-final.log.

Full `scripts/verify latency` run on infraternal-dev (x86_64, 8 cores; profile loads: setup 3.39 6.72 10.07, native 4.57 6.87 10.08, auth-browser 14.80 9.22 10.58, tracer-browser 17.73 10.48 10.96; TRACER_TEST_PORT=4621). Proxy self-test n=31: RTT p50=305.20 ms, p99=331.02 ms, PASS; auth-browser passed 4 Chromium scenarios. Native n=100 baseline PTY echo p99=329.47 ms, other-session flood p99=302.45 ms (growth −27.02 ms, PASS); same-session flood p99=449.58 ms (growth +120.11 ms, FAIL; new finding #73). Production browser: sign-in usable p99=4987.10 ms (PASS); reconnect p99=1012.70 ms/900 ms (FAIL); confirmed echo p99=371.40 ms/334 ms (FAIL); glyph p99=381.00 ms (report); setup/sign-in render p99=1509.70/1839.10 ms (FAIL). Full Playwright suite: 29 passed, 3 failed in 9.2 min; failures were the motion audit (two 62–63 ms main-thread long tasks), reconnect/echo budget assertions, and access-render budget assertion. The earlier setup-token socket hang-up did not reproduce. Full log: `$HOME/.cache/codex-scratch/latency-gate/latency-final.log`.
Author
Owner

Completed the scratch-target support step at 70fa661. The default native fixture root exposed a test-path bug: .sock was 105 bytes but AckBarrier’s .upstream sibling was 109 bytes, beyond Linux sun_path; the real binary then emitted terminal shim disconnected. fixture_state now accounts for both names and uses a shorter default root. Reproduction after the fix: all 24 abrupt-stop positions pass at RTT=300. Fast gate passes; relay smoke n=3 p50=296.82 ms, p99=302.52 ms, and Rust workspace tests plus TypeScript protocol tests are green.

Completed the scratch-target support step at 70fa661. The default native fixture root exposed a test-path bug: `.sock` was 105 bytes but AckBarrier’s `.upstream` sibling was 109 bytes, beyond Linux `sun_path`; the real binary then emitted `terminal shim disconnected`. `fixture_state` now accounts for both names and uses a shorter default root. Reproduction after the fix: all 24 abrupt-stop positions pass at RTT=300. Fast gate passes; relay smoke n=3 p50=296.82 ms, p99=302.52 ms, and Rust workspace tests plus TypeScript protocol tests are green.
Author
Owner

Full latency profile finished on infraternal-dev (x86_64, 8 cores), 300 ms RTT ±30 ms, TRACER_TEST_PORT=4621. Native phase load was 14.01/9.92/7.08; tracer-browser load was 8.84/9.29/7.19. Full log during this run: $HOME/.cache/codex-scratch/latency-gate/latency-final.log. Current code head before the docs-only commit: a8bd3f2.

METRIC name=network_rtt_selftest configured_rtt_ms=300 jitter_ms=30 loss=unsupported transport=tcp-stream n=31 p50_ms=308.19 p99_ms=329.64 result=PASS
METRIC name=keystroke_echo_confirmed_baseline source=real-agent-pty n=100 p50_ms=300.54 p99_ms=329.01 budget_ms=334.00 result=PASS
METRIC name=keystroke_pty_echo_baseline source=real-agent-pty n=100 p50_ms=300.58 p99_ms=329.06 budget_ms=334.00 result=PASS
METRIC name=keystroke_echo_confirmed_flood_other_session source=real-agent-pty n=100 p50_ms=317.67 p99_ms=323.25 budget_ms=334.00 result=PASS
METRIC name=keystroke_pty_echo_flood_other_session source=real-agent-pty n=100 p50_ms=318.03 p99_ms=326.25 budget_ms=334.00 result=PASS
METRIC name=flood_isolation_other_session source=real-agent-pty n=100 baseline_p99_ms=329.06 flood_p99_ms=326.25 growth_ms=-2.81 budget_ms=16.67 result=PASS
METRIC name=keystroke_echo_confirmed_flood_same_session source=real-agent-pty n=100 p50_ms=319.19 p99_ms=333.73 budget_ms=334.00 result=PASS
METRIC name=keystroke_pty_echo_flood_same_session source=real-agent-pty n=100 p50_ms=339.07 p99_ms=382.24 budget_ms=334.00 result=FAIL
METRIC name=flood_isolation_same_session source=real-agent-pty n=100 baseline_p99_ms=329.06 flood_p99_ms=382.24 growth_ms=53.18 budget_ms=16.67 result=FAIL
METRIC name=returning_signin_terminal_usable source=tracer-production-build n=10 p50_ms=3956.40 p99_ms=5024.70 budget_ms=5000 result=FAIL
METRIC name=reconnect_resumed source=tracer-production-build n=20 p50_ms=808.00 p99_ms=1009.00 budget_ms=900 result=FAIL
METRIC name=keystroke_echo_confirmed source=tracer-production-build n=20 p50_ms=337.60 p99_ms=364.70 budget_ms=334 result=FAIL
METRIC name=keystroke_glyph_visible source=tracer-production-build n=20 p50_ms=352.10 p99_ms=372.40 budget_ms=17 result=REPORT
METRIC name=access_screen_render_upper_bound_setup source=tracer-production-build n=30 configured_rtt_ms=300 p50_ms=1341.60 p99_ms=1861.70 budget_ms=1000 result=FAIL
METRIC name=access_screen_first_contentful_paint_setup source=tracer-production-build n=30 configured_rtt_ms=300 p50_ms=1172.20 p99_ms=1614.20 budget_ms=1000 result=REPORT
METRIC name=access_screen_render_upper_bound_signin source=tracer-production-build n=30 configured_rtt_ms=300 p50_ms=1621.60 p99_ms=2054.40 budget_ms=1000 result=FAIL
METRIC name=access_screen_first_contentful_paint_signin source=tracer-production-build n=30 configured_rtt_ms=300 p50_ms=1454.40 p99_ms=1964.30 budget_ms=1000 result=REPORT

Browser suite: 28 passed, 4 failed. Failures: the 4x CPU motion audit sampled pointer checkbox draw offset 0 and missed an intermediate frame; onboarding timed out waiting 5 s for the test double to connect after returning sign-in; the terminal probe assertion included sign-in/echo/reconnect budget failures above; access render exceeded budget. Findings are tracked in #52, #61, #62, #63, #73 and #74. No product changes were made.

Full latency profile finished on infraternal-dev (x86_64, 8 cores), 300 ms RTT ±30 ms, TRACER_TEST_PORT=4621. Native phase load was 14.01/9.92/7.08; tracer-browser load was 8.84/9.29/7.19. Full log during this run: $HOME/.cache/codex-scratch/latency-gate/latency-final.log. Current code head before the docs-only commit: a8bd3f2. METRIC name=network_rtt_selftest configured_rtt_ms=300 jitter_ms=30 loss=unsupported transport=tcp-stream n=31 p50_ms=308.19 p99_ms=329.64 result=PASS METRIC name=keystroke_echo_confirmed_baseline source=real-agent-pty n=100 p50_ms=300.54 p99_ms=329.01 budget_ms=334.00 result=PASS METRIC name=keystroke_pty_echo_baseline source=real-agent-pty n=100 p50_ms=300.58 p99_ms=329.06 budget_ms=334.00 result=PASS METRIC name=keystroke_echo_confirmed_flood_other_session source=real-agent-pty n=100 p50_ms=317.67 p99_ms=323.25 budget_ms=334.00 result=PASS METRIC name=keystroke_pty_echo_flood_other_session source=real-agent-pty n=100 p50_ms=318.03 p99_ms=326.25 budget_ms=334.00 result=PASS METRIC name=flood_isolation_other_session source=real-agent-pty n=100 baseline_p99_ms=329.06 flood_p99_ms=326.25 growth_ms=-2.81 budget_ms=16.67 result=PASS METRIC name=keystroke_echo_confirmed_flood_same_session source=real-agent-pty n=100 p50_ms=319.19 p99_ms=333.73 budget_ms=334.00 result=PASS METRIC name=keystroke_pty_echo_flood_same_session source=real-agent-pty n=100 p50_ms=339.07 p99_ms=382.24 budget_ms=334.00 result=FAIL METRIC name=flood_isolation_same_session source=real-agent-pty n=100 baseline_p99_ms=329.06 flood_p99_ms=382.24 growth_ms=53.18 budget_ms=16.67 result=FAIL METRIC name=returning_signin_terminal_usable source=tracer-production-build n=10 p50_ms=3956.40 p99_ms=5024.70 budget_ms=5000 result=FAIL METRIC name=reconnect_resumed source=tracer-production-build n=20 p50_ms=808.00 p99_ms=1009.00 budget_ms=900 result=FAIL METRIC name=keystroke_echo_confirmed source=tracer-production-build n=20 p50_ms=337.60 p99_ms=364.70 budget_ms=334 result=FAIL METRIC name=keystroke_glyph_visible source=tracer-production-build n=20 p50_ms=352.10 p99_ms=372.40 budget_ms=17 result=REPORT METRIC name=access_screen_render_upper_bound_setup source=tracer-production-build n=30 configured_rtt_ms=300 p50_ms=1341.60 p99_ms=1861.70 budget_ms=1000 result=FAIL METRIC name=access_screen_first_contentful_paint_setup source=tracer-production-build n=30 configured_rtt_ms=300 p50_ms=1172.20 p99_ms=1614.20 budget_ms=1000 result=REPORT METRIC name=access_screen_render_upper_bound_signin source=tracer-production-build n=30 configured_rtt_ms=300 p50_ms=1621.60 p99_ms=2054.40 budget_ms=1000 result=FAIL METRIC name=access_screen_first_contentful_paint_signin source=tracer-production-build n=30 configured_rtt_ms=300 p50_ms=1454.40 p99_ms=1964.30 budget_ms=1000 result=REPORT Browser suite: 28 passed, 4 failed. Failures: the 4x CPU motion audit sampled pointer checkbox draw offset 0 and missed an intermediate frame; onboarding timed out waiting 5 s for the test double to connect after returning sign-in; the terminal probe assertion included sign-in/echo/reconnect budget failures above; access render exceeded budget. Findings are tracked in #52, #61, #62, #63, #73 and #74. No product changes were made.
Author
Owner

Documentation step complete: 6c4edf7 updates docs/PROCESS.md and apps/tracer/README.md with the default 300 ms RTT and jitter, rootless relay boundaries, loss limitation, metrics, and fast/full profiles. Fast gate follows on this exact head.

Documentation step complete: 6c4edf7 updates docs/PROCESS.md and apps/tracer/README.md with the default 300 ms RTT and jitter, rootless relay boundaries, loss limitation, metrics, and fast/full profiles. Fast gate follows on this exact head.
Author
Owner

Finished on branch latency-gate, HEAD 6c4edf78081f79787bb3ac14408983389f5d3b8a. Final required gate command:
nix shell nixpkgs#binutils nixpkgs#file nixpkgs#gnumake nixpkgs#perl nixpkgs#zig_0_15 -c scripts/verify fast

Fast gate output (verbatim summary from the run):

METRIC name=network_rtt_selftest configured_rtt_ms=300 jitter_ms=30 loss=unsupported transport=tcp-stream n=3 p50_ms=316.63 p99_ms=322.37 result=PASS

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+

dist/ghostty-web.js: OK
dist/ghostty-vt.wasm: OK
dist/index.d.ts: OK

stream differential: 134 streams, 11439 schedules, zero divergences

 7 pass
 0 fail
 34390 expect() calls
Ran 7 tests across 2 files. [4.35s]

Fast gate exited 0. Full latency profile findings and associated issues are recorded in the earlier comments; no issue was closed. No push or deploy performed.

Finished on branch `latency-gate`, HEAD `6c4edf78081f79787bb3ac14408983389f5d3b8a`. Final required gate command: `nix shell nixpkgs#binutils nixpkgs#file nixpkgs#gnumake nixpkgs#perl nixpkgs#zig_0_15 -c scripts/verify fast` Fast gate output (verbatim summary from the run): ``` METRIC name=network_rtt_selftest configured_rtt_ms=300 jitter_ms=30 loss=unsupported transport=tcp-stream n=3 p50_ms=316.63 p99_ms=322.37 result=PASS +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | PASS | +---------------------------+--------------+ dist/ghostty-web.js: OK dist/ghostty-vt.wasm: OK dist/index.d.ts: OK stream differential: 134 streams, 11439 schedules, zero divergences 7 pass 0 fail 34390 expect() calls Ran 7 tests across 2 files. [4.35s] ``` Fast gate exited 0. Full latency profile findings and associated issues are recorded in the earlier comments; no issue was closed. No push or deploy performed.
Author
Owner

Reproducibility note: the exact final invocation was CARGO_TARGET_DIR="$HOME/.cache/codex-scratch/latency-gate/target" TRACER_TEST_PORT=4621 nix shell nixpkgs#binutils nixpkgs#file nixpkgs#gnumake nixpkgs#perl nixpkgs#zig_0_15 -c scripts/verify fast. The temporary target and log directory were removed after the run as required; all profile metrics are recorded in these issue comments. Final self-review found a clean diff check and no product-code or dependency changes.

Reproducibility note: the exact final invocation was `CARGO_TARGET_DIR="$HOME/.cache/codex-scratch/latency-gate/target" TRACER_TEST_PORT=4621 nix shell nixpkgs#binutils nixpkgs#file nixpkgs#gnumake nixpkgs#perl nixpkgs#zig_0_15 -c scripts/verify fast`. The temporary target and log directory were removed after the run as required; all profile metrics are recorded in these issue comments. Final self-review found a clean diff check and no product-code or dependency changes.
Author
Owner

Merged latency-gate into dev at ac03b19. Orchestrator ran scripts/verify fast on dev after the merge (exit 0):

METRIC name=network_rtt_selftest configured_rtt_ms=300 jitter_ms=30 loss=unsupported transport=tcp-stream n=3 p50_ms=307.20 p99_ms=312.12 result=PASS
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | PASS         |
 2 pass
 0 fail
 7 pass
 0 fail
exit=0

Budget misses from the full profile stay open in #52 #61 #62 #63 #73 #74.

Merged latency-gate into dev at ac03b19. Orchestrator ran `scripts/verify fast` on dev after the merge (exit 0): ``` METRIC name=network_rtt_selftest configured_rtt_ms=300 jitter_ms=30 loss=unsupported transport=tcp-stream n=3 p50_ms=307.20 p99_ms=312.12 result=PASS | fmt | PASS | | VT native fmt | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | PASS | 2 pass 0 fail 7 pass 0 fail exit=0 ``` Budget misses from the full profile stay open in #52 #61 #62 #63 #73 #74.
Author
Owner

Closed by the orchestrator during the 2026-10-28 handover (#254), on the owner's request to close finished issues. Evidence: Gate merged as ac03b1913 (verified in dev and prod): 300 ms RTT default profile and input-to-photon gate; remaining budget misses were split into #52 #61 #62 #63 #73 #74. Reopen if anything is still missing.

Closed by the orchestrator during the 2026-10-28 handover (#254), on the owner's request to close finished issues. Evidence: Gate merged as ac03b1913 (verified in dev and prod): 300 ms RTT default profile and input-to-photon gate; remaining budget misses were split into #52 #61 #62 #63 #73 #74. Reopen if anything is still missing.
kayg closed this issue 2026-10-28 17:52:29 +00:00
Sign in to join this conversation.
No labels
launch-qa
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/infraternal#55
No description provided.