J394: two tracer specs fail on dev (terminal.spec.ts:5 prediction, predictive-echo.spec.ts:530) #438
Open
opened 2026-11-08 20:27:49 +00:00 by kayg
·
7 comments
No Branch/Tag specified
main
j470b-server-mesh-rejoin
mesh-compat-382
j444b-d381-key-wins-flood
j461b-macos-utun-reader-thread
j475b-remove-host-motion
fix-reconnect-new-link
j481-d380
dev
j479-mesh-soak
fix/j476-host-facts
j475-motion-pill-split
j470-mac-node-join
root-upgrade-d376
j474-review-merge
latency-targets-120hz
nightly-suite-shards
gotatun-pin-j469
macos-utun-reader-thread
fix-flood-echo-stall
wip/j416
motion-fast-ease-out
mesh-addhost-j259
takeover-close-terminals-d368
kbd-instant-followups
fix-trust-key-double-click
nightly-suite-partial-report
appwin-icons-j207
rusqlite-slice1
swipe-content-first-d374
mesh-mtu-381
addhost-instant-open
glass-contrast-d365-mock
glass-contrast-d365
signout-followups-442
fix-startup-toast-red
research-gotatun-matrix-j317
themes-sync-j304
fix-load-flakes
fix-nix-zig-patch
fix-compute01-browser-baseline
ratchet-size-baseline-419
predict-auto-d130
revoke-coverage-d367
nightly-suite-queue-timeout
nightly-browser-suite
fix-rustsec-2026-0459
fonts-ioskeley-obel
log-retention-d375
fix-kbd-no-anim-427
fix-dev-red-specs-2
takeover-proof-305-j205
wip/j410-server-pump
ui-consistency-d372
fix-osc52-clipboard
libghostty-vt-1.0
forget-retired-host-d369
j404-merge-426
fix-setup-lock-race
fix-touch-select-426
revoke-access-report-ui
fix-predictive-echo-specs
fix-selinux-agent-exec
wip/j395-checkpoint2
wip/j395
j391-review-scratch
ui-consistency-j195-deferred-rebased
fix-rustc-paths-421
decisions-wording-341
fix-install-suite-red
fix-dev-rtt-specs
fix-pending-removal-align
fix-privileged-skip
fix-install-suite-404
review-j368
mesh-dns-j254
revoke-access-report-d356b
revoke-privilege-round5
fix-removal-confirm
review-j360
gotatun-rotation-367
deps-refresh-340
review-j354
review-J351
review-j350
addhost-takeover-d359
revoke-report-items-347
revoke-end-sessions-d362
review-j348
size-mesh-ssh-346
size-trim-337
revoke-access-diff-d356
j259-pre-rebase-b06c5a6
j305-ended-at-start-spec
addhost-sudo-consent
fix-layout-real-specs-411
fix-shell-open-411
fix-console-hosts-373
fix-shell-exit
fuzz-host-name-353
fix-host-details-410
macos-filter-j197
research-gotatun-tricks
revoke-p3-j216l
stale-state-pill
revoke-followups-j216k
revoke-progress-j216e
backup/macos01-20261106/lume/j197
backup/macos01-20261106/lume/appwin-icons-j207-stash0
backup/macos01-20261106/claude/stash-0
backup/macos01-20261106/claude/stash-1
backup/macos01-20261106/claude/J80-macos-s1
backup/macos01-20261106/claude/J80
backup/macos01-20261106/claude/J80-slice1-20261102
backup/macos01-20261106/claude/J16b-appwin-on-dev
backup/macos01-20261106/claude/appwin-launch-idle-146
backup/macos01-20261106/claude/J80-s1-run
wip/j259
trust-285-p3c
runtime-lock-j216f
mesh-ssh-p3-j175
tun-proof-p3b
trust-285-p3
terminal-polish-j271
tun-proof-p3
ssh-install-flakes-j216i
fuzz-fixes-j264
trust-285-a
trust-285-b
restart-oversized-fixture-j198
backup/macos01-20261106/lume/appwin-icons-j207
restart-bound-j216h
up-preflight-p3-j198
backup/macos01-20261106/lume/j251
up-preflight-j198
wip/up-preflight-j198-j245-p2
wip/up-preflight-j198-j245-p1
agent-fsync-offloop-j196
wip/up-preflight-j198-j245
settings-offscreen-j231
wip/up-preflight-j198-j226-inspection
wip/j103f
themes-j206
wip/up-preflight-j198-j226
flakes-j216c
backup/macos01-20261106/claude/j197
flakes-j216b
backup/macos01-20261106/claude/j197-red
wip/up-preflight-j198-review
flakes-j216
backup/macos01-20261106/claude/appwin-icons-j207
engineering-j208b
appwin-mock-j209
toast-stack-j165
engineering-j208
decisions-d351
decisions-d350
decisions-d349
appearance-presets-mock-j129
decisions-d348
docs-audit-spec
doc-coverage-baseline-raise
decisions-d347
decisions-d346b
decisions-d346
j88h-profile
ui-consistency-j195-deferred
ui-consistency-j195
engineering-practices-j194
preferred-login-ui-j166
signin-pools-263-j174
macos-node-265-s1
decisions-d345
decisions-d344
proxy-cgroup-j164
verify-quick-fuzz-329
proxy-start-io-j168
timing-flakes-j163
up-263-s1
mcp-events-j147
verify-quick-fuzz-j158
proxy-p3-j159
agent-resource-j157-detect
wip/j103d-final
remote-cancel-hang-j155
proxy-traefik-281-j62d
j149-verify-quick
proxy-traefik-281-j62b
j150-drop-unnamed
wip/j79d-red-r4
wip/j103d-d332
rootless-toast-j142
wip/j103d-head
wip/j103d
j125c-mcp
macos-build-j136
backup/macos01-20261106/claude/j136
wip/j79d-red-cli
wip/j79d-red-server
j121-link-liveness
j118c-tracer-failures
wip/j79c-head
release-desktop-j131
j124c-signed-macos
wip/j79c-red-p3
wip/j79c-red-p23
wip/j79c-red-p22
merge-j128-check
perf-deferred-j128
sidebar-toggle-260
j127-shortcut-changed-dot
j125-mcp
j118-tracer-failures
j126-account-summary
settings-sections-mock-j92
j120-pinch-lock
sidebar-292
backup/macos01-20261106/claude/j120
taste-2026-11-03
row-actions-242-215
backup/macos01-20261106/claude/stash-0-untracked
backup/macos01-20261106/claude/J80-265-s1-run
backup/macos01-20261106/claude/J80-macos-s1-20261102
j112-canvas-commands
ui-no-overshoot-scroll
fix-recovery-restart
backup/macos01-20261106/claude/stash-1-untracked
taste-2026-11-02
size-baseline-ratchet
machine-cards-structure-j108
j109-boot-copy-mockups
fix-dev-red
wip/up-263-s1
j95b-settings-rows
proxy-traefik-281
j95-option-rows
docs-human-j91
docs-keys-palette
fix-flood-1013-270-j38d
mesh-ssh-s1
mesh-ssh-s0
addhost-d291-tunnel
test-stability-load
mesh-relay-220
repro-296
shim-thp-off-24
mesh-pins-258-v2
mesh-dns
signin-d311-merged
docs-pages
ui-size-283
fix-ui-batch-specs
wip/j12b-edge-update
gate-on-arm
fix-rename-sync-293
trust-285-j60b
fix-focus-ring-clip
integ-ui-1101
edge-update-v2
signin-d311
taste-2026-11-01
layout-per-tab-250
ui-approved-splitopt-224
ui-approved-splitopt-224-on-dev
icon-text-centre-282
ui-approved-189-212-216
research-reverse-proxy
transfer-budget-320
review-mesh-pins-258-r2
sidebar-pill-259
review-mesh-relay-220
wip/j12-edge-update-v2
nix-package
status-crash-loop-237
fix-dev-test-failures
fix-fixture-socket-path
wip/mesh-relay-220-pre-dev-advance-rebase
fix-canvas-side-clip
research-head-to-head
frosted-real-256
review-mesh-pins-258
wip/mesh-relay-220-pre-final-rebase
review-layout-per-tab-250-r2
broad-suite-triage
nix-devshell
review-status-crash-loop-237-r3
wip/nix-devshell-pre-p3
taste-frosted-real
wip/code-serve-demo-2026-10-30
wip/code-serve-glass-2026-10-30
wip/code-serve-clusters-2026-10-30
wip/code-merge-release94-dev-2026-10-30
wip/rproxy-delta-scratch-2026-10-30
wip/stale-note-257-staged-2026-10-30
wip/d241-size-staged-2026-10-30
review-broad-suite-207
wip/mesh-pins-258-pre-split
wip/mesh-relay-220-pre-rebase
mesh-measure-220
plan-211-update
review-edge-autoupdate-r3
perf24-reconnect
review-stale-note-r2
release-prod-live
d241-shell-marker
review-mesh-pins-server-258
review-revocation-r3
review-setup-token-r2
review-sync-journal-r3
wip/review-mesh-pins-258-prune-review-1995fee46
frosted-scroll-256
sync-journal-r3
revocation-per-account-wip
review-tokio-asyncfd-0386
review-sync-p3s
wip/stash-8by64-stash0-2026-10-30
mesh-drag-r2
setup-token-data-wip
release-9-4
auth-busy-1013
mesh-view-overlap
sync-p3s
review-auth-busy-wake
review-readd-r8-wip
mesh-drag-bounds
audit-auth-93
app-windows-stream-start
review-authority-cap-r3
app-windows-r7-measure
readd-minimal-r6
readd-minimal-r5
app-windows-r7-soak
readd-minimal-r4
app-windows-r7
app-windows-r6
readd-minimal-r3
readd-minimal-r2
readd-minimal
merge-release8-dev
review-busy-page-liveness-r2
review-busy-page-liveness
mesh-view-trim
review-authority-read-retry-r2
review-remove-uninstall-r2
release-retry-reattach
release-retry
review-remove-uninstall
review-authority-read-retry
app-windows-ship
mesh-autoswitch-p2
mesh-ssh
mesh-relay
mesh-macos-node
screen-and-control
session-labels
review-real-logins-r5
review-real-logins-r4
review-real-logins-r3
shim-memory-r2
review-real-logins-r2
shim-memory
shadow-addhost-instant
shadow-real-logins
shadow-same-session-flood
shadow-prod-perf-fix
hosts-from-server
app-windows-real
app-windows-transport
release-prod-6w
mesh-policy-verify
quiet-close-toast
dev-merge-r4
review-mesh-agent-transport
review-app-windows-transport
settings-roundtrip
review-release-pipeline-r4
mesh-agent-transport
ended-strip-fix
review-audit-ca-update-d244
settings-rows
phone-width
signout-clear
fix-restart-tests
dev-merge-rp
signer-hardening
review-audit-ca-update
terminal-fidelity
review-audit-ca-update-wip
mesh-integrate
review-mesh-agent-transport-515d3712
review-mesh-dataplane-feec1988
mesh-dataplane
signout
research-canvas-node
wip/review-renderer-cost
wip/review-terminal-fidelity
padlock-glyph
review-release-pipeline-2e95b689
phone-polish
sidebar-polish
inbox-generic
review-mesh-dataplane-378d4055
prod-batch
ui-inbox
ui-ca-unlock
ui-machine-sections
review-mesh-dataplane-a5f744fb
tls-session-prod
session-bounds-prod
fix-dead-session
session-bounds
tls-session
palette-polish
shortcuts-d222
research-appmeta
phone-gutter
ui-centre-button
wip/ui-centre-button
wip/tls-session
review-addhost-breakit
ui-platform
renderer-cost
remote-agents
mesh-api
ssh-bootstrap
prod-hotfix2
prod-fixes
review-remote-agents
review-mesh-dataplane
session-life
research-browser
review-mesh-api
research-capture
review-mesh-api-round3
review-mesh-api-round1
gate-stability
review-gate-stability
review-relay-direct
review-mesh-api-findings
proto-sessions
font-picker
proto-addhost
proto-windows
mesh-core
latency-gate
research-bsd
research-discovery
keytray-mods
mock-core
ui-tweaks
fix-motion-sel
fix-motion
integrate2
fix-robust
t-web
t-server
vt-ghostty
fix-snap-echo
t-web-auth
review-vt2
t-agent
lane-protocol
lane-shim
lane-auth
s1-base
s1-contracts
s1-workspace
No results found.
No labels
launch-qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set
Reference
kayg/infraternal#438
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
J394: two tracer specs fail on dev (terminal.spec.ts:5 prediction, predictive-echo.spec.ts:530)
Background: J374 (#430) fixed the RTT readout and found two more failures on unmodified dev that it did not investigate:
apps/tracer/test/terminal.spec.ts:5fails interminal-scenario.ts(waitForPendingText): the predicted"p"never appears.apps/tracer/test/predictive-echo.spec.ts:530("captures pending and confirmed glyphs across production viewports and themes") fails; J374 saw a proxy "socket hang up".Questions: is production affected (prod
f6f62a875, test server same client)? Is the product or the spec wrong? Plan: bisect each over dev merges with a production build (compute01, build-slot), red test first if product bug, measure prediction latency before/after, atomic commits onfix-predictive-echo-specs(base origin/dev753fcd657). No merge, no deploy.Estimate: ~3 h (bisects are minutes per step; queue waits dominate).
Start. Branch
fix-predictive-echo-specs, base origin/dev753fcd657. Plan: (1) run both specs on compute01 at dev head and at prodf6f62a875to see if they fail there; (2) bisect each; (3) decide product vs spec; red test first if product. ETA ~3 h.Finding 1 (dev head
753fcd657, both specs reproduce; log ~/j394-evidence/dev-two.log on compute01): terminal.spec: at the no-echo step the predicted "p" is not drawn because the prediction state has no round trip. Diagnostic: before the keyreason=rtt-low, round trip undefined; after the keyrtt-unmeasured. So after the drop-ack/resume the round trip is blanked (TerminalClient.startPings on attach) and not refilled in time. Finding is the symptom; cause (does the probe fail to go out, or is the answer dropped?) in progress, with an RTT timeline.Finding 2 (hypothesis, to be proven): predictive-echo.spec:530 "socket hang up" is the first context.request.get(/__mock/state) on a kept-alive connection through the 300 ms test proxy; the mock server is a node:http server with the default 5 s keep-alive timeout, so a request issued ~5 s after the previous one races the server closing the idle socket.
Finding 2 proven (no product code involved).
predictive-echo.spec.ts:530"socket hang up" = HTTP keep-alive race in the test double.context.request.get("/__mock/state")issued 5757 ms after the previous request of the same context (/__mock/open-terminal); the only other calls were reset/open-terminal.node:httpserver (default keep-alive timeout) behind scripts/latency_proxy.py at the spec's 300 ms RTT, driven by Playwright's own APIRequestContext (node and bun runtimes): gap 5.0/5.2/5.4/5.6 s ok, 5.7 s and 5.8 s ->socket hang up (ECONNRESET), 6.0/6.2 ok. With the server'skeepAliveTimeout = 0all gaps pass.keepAliveTimeout = 0(it is a per-run test double), plus a spec that requests, waits 5.8 s and requests again (red before, green after).Finding 1 (update, diag2.log): the RTT timeline around the failure is
100ms: rtt 302.8, connectedthen200ms: rtt undefined, DISCONNECTEDfor the rest of the window. The mock closes the socket (code 4001) when it sees the dropped-ack key, the page'sisConnected()poll happened to see the old socket still open for ~100 ms, so the scenario goes on while the client is reconnecting; the reattach blanks the round trip (startPings), and "p" is typed while it is unmeasured (rtt-unmeasured), so it is not predicted. Still to establish: how long after a real reconnect the first key is predicted again (production effect) and which commit moved the timing. Bisect batch queued next.Bisect + production exposure (logs
~/j394-evidence/bis-<sha>.log, tablebisect-results-1.txton compute01; each commit built as production, #430's two-line source fix applied on top because without it the RTT-readout step masks later steps):3cc63cb09(11-03 15:46)fa7ec56f6(11-03 20:39)1633ffeea(11-04)a6d0223bf(11-04 19:31)eb0684cbf,442ac0f99,9630c2bd9,af033d0b4,5d34c133a753fcd657(head)a6d0223bf, head) without any change to the prediction code: the request lands in the 5.65-5.8 s reset window or not, depending on how long the earlier steps took. That is the keep-alive race from finding 2, so a commit bisect cannot name a culprit. Fix is committed (8f0ae27db, red testcfb258b4c).e81eb5bb4#327,5ed27aa0ej150,fa7ec56f6#330). Narrowing now.Production exposure: the live bundles at https://ssh.kayg.org/main.js and https://test.ssh.kayg.org/main.js contain exactly one gate for prediction:
ce=c.get("predict")==="1"andpredictiveEcho:ce. Predictive local echo is off for every user unless the page URL has?predict=1; nothing turns it on from the measured round trip (D130 says it should switch on automatically above ~60 ms; that wiring does not exist yet). So the failing specs describe an opt-in path and cannot affect users today. Question for kayg (not blocking, no change made): is the?predict=1-only state intended until the D130 auto-switch is built?Finding 1 cause (terminal.spec.ts:5, "p" never predicted): a spec race, not a prediction bug. Evidence:
~/j394-evidence/run3-diag.log(page-side timeline, 100 ms samples, round trip : connected : prediction reason):The scenario types
pat about 100-500 ms, inside the reconnect, so the key correctly gets no prediction (rtt-unmeasured) and the poll forpendingText === "p"can never succeed. The mock closes the socket (4001) while acceptingb; the scenario waited for "ab" at the double andisConnected(), butisConnected()is still true for the old socket until the close arrives one link delay later, so it moved on early.Bisect (
bisect-results-1.txt,bisect-results.txt): first-parent merges3cc63cb09pass;e81eb5bb4(#327),5ed27aa0e,fa7ec56f6and everything after fail at "p".e81eb5bb4changes nothing in terminal.ts, connection.ts, mock-server.ts or terminal-scenario.ts (only the rootless refusal toast and the real-server harness), so it moved the timing; it did not break prediction.Product vs spec: spec. Prediction needs a measured round trip and a reattach discards the old one on purpose (TerminalClient.startPings); after a resume the first key is unpredicted for about one round trip (here 300 ms), then prediction is back. That is by design, no change made. Fix: commit
62a775aa4makes the scenario wait for the double'sattach_countto go up (the resume happened) before it types again.Branch so far (
fix-predictive-echo-specs):bf6b7502c(red keep-alive spec),8251033e8(mock-server keepAliveTimeout=0),62a775aa4(scenario waits for the resume). Gate batch (red, 3x both specs, sibling specs) is queued.Gate evidence on compute01 (production tracer build, test double behind the 300 ms link). Logs:
~/j394-evidence/run4-*.log.Sibling run = predictive-echo, predictive-backspace, terminal, terminal-* (links, query-replay, query-viewers, states, theme, cell-position), snapshot-resume, transient-session-input: 66 passed, 1 failed (terminal-links.spec.ts:476, a sub-pixel tooltip width 306.78 vs 308.00; no relation to this change). Checking that it also fails on unmodified origin/dev, and running
scripts/verify quick, in the next slot run.Finish (J394). Branch
fix-predictive-echo-specs, head16ca32dc67e3a4b8e98bd31530e4fd888f494e87, base origin/dev753fcd657. Pushed; not merged, not deployed.Commits:
6e53ad7c6test(tracer): failing spec for a request after an idle pause on the test double (red:run4-red.log, 13 staggered pauses, fails on the unfixed double)96ea2ac9ctest(tracer): the test double keeps idle connections open (server.keepAliveTimeout = 0in test/mock-server.ts)16ca32dc6test(tracer): the terminal scenario waits for the resume before it types again (waits for the double'sattach_count; doc comment onterminalScenario)Causes:
a6d0223bfand head fail, the rest pass), so a bisect names no culprit. Spec and product were right.pduring the reconnect the double forces (the reattach clears the round trip by design; it is measured again about one RTT later), sopwas correctly unpredicted. Bisect: green3cc63cb09, red frome81eb5bb4on, which changes none of the terminal, connection, mock or scenario code, so it only moved the timing.Production: not affected. Predictive local echo is only enabled by
?predict=1in the page URL (live bundles of ssh.kayg.org and test.ssh.kayg.org contain that one gate); no user has it by default, and the D130 automatic switch-on from the measured round trip is not wired. No prediction latency number changed because no product code changed. Question for kayg (not blocking): is?predict=1-only intended until the D130 auto-switch exists?Gates (compute01, production build, all logs in
~/j394-evidence/):run7-green3.log: 9 passed (6.2m).scripts/verify quickon the head (run7-quick.log): all rows PASS (scope guard, migration names, Nix vendoring, fmt, doc coverage, shape lints, doc comments on diff, fixed test waits, ui literals, tracer typecheck),QUICK_EXIT=0.verify fastwas not run (the task asked for quick).run7-siblings.log: 65 passed, 2 failed, both unrelated to this change:terminal-links.spec.ts:476(tooltip width 306.78 vs 308.00): also fails 2 of 2 on unmodified origin/dev (run5-links-dev.log) and in the first sibling run on the branch.terminal-theme.spec.ts:206(one 96 ms long task under load): passed in the first sibling run (run4-siblings.log, 66 passed) and 4 of 4 on both dev and head (run8-theme-*.log).Real-environment proof: not applicable to a test-only change (no user-visible or root-only path touched); the production claim was checked against the live bundles instead. Cleanup: my VM worktrees and cargo target removed.