Investigate intermittent browser disconnect/replay during three-viewer terminal query corpus #277

New issue

Open

opened 2026-10-31 19:27:27 +00:00 by kayg · 0 comments

kayg commented

2026-10-31 19:27:27 +00:00

Owner

Split from #207: determine why a live browser terminal connection intermittently disconnects and replays during the three-viewer query corpus property.

Observed on aarch64 infraternal-dev-02-std, production head 284384981e10ebbe90149f4c11635ed64576bdfb, Nix Chromium, Playwright 1.61.1, two workers, retries zero:

Broad D: corpus 34 (cancelled OSC followed by CPR), browser marked-reply delta 0, protocol viewers 1/1.
After disk repair, broad F: same missing-browser-reply symptom at corpus 32 (DEL inside CSI), with about 74GB free. Result 114 passed/15 skipped/1 failed. Disk exhaustion is not sufficient to explain it.
Unchanged focused repetitions: 3/3 passed. Diagnostic-only dist/test copy: 4/6 passed; both failures occurred at 33. Both failing metadata receipts show old bridge closed (server 1005) after 27 replies, replacement bridge replayed once with 0 new marked replies. Old-bridge write/reply events end before the failed query.
Further close-metadata repetitions: 4/4 passed. Every diagnostic pass retained its original bridge through corpus 36 with 31 replies. The accepted corpus contains 30 answered forms plus the initial query.
Broad H/I later passed consecutively: 115 passed/15 skipped each, 9.3m/9.2m; no production artifact changes, workers 2/retries 0, one attempt/test.

This is a real intermittent disconnect/replay associated with the missing attempt. The disconnect cause is unproved; a deterministic parser defect at corpus 34 is not established. The exact cause of D's uninstrumented failure is not known. Historical replies are intentionally suppressed during replay, so the replacement viewer cannot satisfy the original per-viewer attempt assertion retrospectively. Do not solve by relaxing the corpus/assertions, adding retries, or replaying historical terminal replies.

Reproduce with the production apps/tracer/real/query-replies.spec.ts property three real viewers produce one PTY reply and refused marked input cannot stall typing, and with the full playwright.real.config.ts suite under two workers/retries0. Use build-slot and Nix; the diagnostic instrumentation logs only response-path metadata, not authentication or terminal bytes. Capture both transport close directions/reasons, native close classification, replay fences, and browser response-path metadata at the disconnect. Distinguish a server-initiated close, proxy behavior, and browser liveness retirement before changing code.

Evidence and complete accounting: #207 (comment)
Receipt bundle (includes query-triage-metadata.json and broad F/H/I receipts): https://git.kayg.org/attachments/aece72ff-91b6-4b54-9ec0-0390832f5b3f

Raw private local receipts remain under ~/.cache/codex-scratch/broad-suite-207/resume on control and dev-02-std. No diagnostic code was committed. Follow-up is separate from the #207 fixes now rebased at a5a2ccd77f581bdf0130a1c3b0d6886dbb475eee; the H/I acceptance evidence predates that rebase.

Split from #207: determine why a live browser terminal connection intermittently disconnects and replays during the three-viewer query corpus property. Observed on aarch64 infraternal-dev-02-std, production head `284384981e10ebbe90149f4c11635ed64576bdfb`, Nix Chromium, Playwright 1.61.1, two workers, retries zero: - Broad D: corpus 34 (cancelled OSC followed by CPR), browser marked-reply delta 0, protocol viewers 1/1. - After disk repair, broad F: same missing-browser-reply symptom at corpus 32 (DEL inside CSI), with about 74GB free. Result 114 passed/15 skipped/1 failed. Disk exhaustion is not sufficient to explain it. - Unchanged focused repetitions: 3/3 passed. Diagnostic-only dist/test copy: 4/6 passed; both failures occurred at 33. Both failing metadata receipts show old bridge closed (`server 1005`) after 27 replies, replacement bridge replayed once with 0 new marked replies. Old-bridge write/reply events end before the failed query. - Further close-metadata repetitions: 4/4 passed. Every diagnostic pass retained its original bridge through corpus 36 with 31 replies. The accepted corpus contains 30 answered forms plus the initial query. - Broad H/I later passed consecutively: 115 passed/15 skipped each, 9.3m/9.2m; no production artifact changes, workers 2/retries 0, one attempt/test. This is a real intermittent disconnect/replay associated with the missing attempt. The disconnect cause is unproved; a deterministic parser defect at corpus 34 is not established. The exact cause of D's uninstrumented failure is not known. Historical replies are intentionally suppressed during replay, so the replacement viewer cannot satisfy the original per-viewer attempt assertion retrospectively. Do not solve by relaxing the corpus/assertions, adding retries, or replaying historical terminal replies. Reproduce with the production `apps/tracer/real/query-replies.spec.ts` property `three real viewers produce one PTY reply and refused marked input cannot stall typing`, and with the full `playwright.real.config.ts` suite under two workers/retries0. Use build-slot and Nix; the diagnostic instrumentation logs only response-path metadata, not authentication or terminal bytes. Capture both transport close directions/reasons, native close classification, replay fences, and browser response-path metadata at the disconnect. Distinguish a server-initiated close, proxy behavior, and browser liveness retirement before changing code. Evidence and complete accounting: https://git.kayg.org/kayg/infraternal/issues/207#issuecomment-71376 Receipt bundle (includes query-triage-metadata.json and broad F/H/I receipts): https://git.kayg.org/attachments/aece72ff-91b6-4b54-9ec0-0390832f5b3f Raw private local receipts remain under `~/.cache/codex-scratch/broad-suite-207/resume` on control and dev-02-std. No diagnostic code was committed. Follow-up is separate from the #207 fixes now rebased at `a5a2ccd77f581bdf0130a1c3b0d6886dbb475eee`; the H/I acceptance evidence predates that rebase.

kayg referenced this issue

2026-10-31 19:27:29 +00:00

dev gate: restart tests depend on CARGO_TARGET_DIR and wall-clock timeouts #207

kayg referenced this issue

2026-10-31 19:31:06 +00:00

dev gate: restart tests depend on CARGO_TARGET_DIR and wall-clock timeouts #207

kayg referenced this issue

2026-10-31 19:45:06 +00:00

dev gate: restart tests depend on CARGO_TARGET_DIR and wall-clock timeouts #207

kayg referenced this issue

2026-10-31 19:57:16 +00:00

dev gate: restart tests depend on CARGO_TARGET_DIR and wall-clock timeouts #207