Bug: echo after a Snapshot is received but not painted in the browser #39

Closed
opened 2026-10-06 23:29:24 +00:00 by kayg · 12 comments
Owner

Found by spike-mem (#38, branch spike-mem a82523c, apps/tracer/MEMORY-SPIKE.md): in Snapshot mode, a keystroke echo that arrives after a Snapshot reaches the browser but is not painted. Production uses Snapshots on resume after the ring wraps, so deploy 1 (dev 0c47cd4) may show a frozen-looking terminal after a long disconnect. High priority: reproduce on dev, then fix.

Second, separate finding: streaming the raw behemoth fixture into ghostty-web loses emoji modifiers and grapheme clusters that the native Snapshot preserves (a browser renderer fidelity issue).

Found by spike-mem (#38, branch spike-mem a82523c, apps/tracer/MEMORY-SPIKE.md): in Snapshot mode, a keystroke echo that arrives after a Snapshot **reaches the browser but is not painted**. Production uses Snapshots on resume after the ring wraps, so deploy 1 (dev 0c47cd4) may show a frozen-looking terminal after a long disconnect. High priority: reproduce on dev, then fix. Second, separate finding: streaming the raw behemoth fixture into ghostty-web loses emoji modifiers and grapheme clusters that the native Snapshot preserves (a browser renderer fidelity issue).
Author
Owner

Starting reproduction and fix on branch fix-snap-echo at base SHA 0c47cd42e6. Reviewed #39 and its spike parent #38 comments; continuing from the recorded 5,000,000-byte reproduction.

Starting reproduction and fix on branch fix-snap-echo at base SHA 0c47cd42e6afeec7ca9d6d430bc11b319eb677c8. Reviewed #39 and its spike parent #38 comments; continuing from the recorded 5,000,000-byte reproduction.
Author
Owner

Finding: reproduced on deployed base 0c47cd4. The production tracer page resumes after 8 MiB + 64 detached bytes; the test double sends two contract-valid Snapshot frames, then the server accepts/echoes the queued x. Browser receivedSequence reaches 8,388,673 while painted outputSequence stays 8,388,672. Red test: apps/tracer/test/snapshot-resume.spec.ts (TRACER_TEST_PORT=4431, failure at outputSequence assertion).

Finding: reproduced on deployed base 0c47cd4. The production tracer page resumes after 8 MiB + 64 detached bytes; the test double sends two contract-valid Snapshot frames, then the server accepts/echoes the queued x. Browser receivedSequence reaches 8,388,673 while painted outputSequence stays 8,388,672. Red test: apps/tracer/test/snapshot-resume.spec.ts (TRACER_TEST_PORT=4431, failure at outputSequence assertion).
Author
Owner

Snapshot fix committed: ff6a8c9. Removed the stale-output queue purge from older Snapshot callbacks; wrapped-ring Playwright test covers 8 MiB + 64 B, two Snapshots, post-resume echo, resize, scrollback, and theme replay. bun run check passed; TRACER_TEST_PORT=4431 bun x playwright test test/snapshot-resume.spec.ts: 1 passed.

Snapshot fix committed: ff6a8c9. Removed the stale-output queue purge from older Snapshot callbacks; wrapped-ring Playwright test covers 8 MiB + 64 B, two Snapshots, post-resume echo, resize, scrollback, and theme replay. bun run check passed; TRACER_TEST_PORT=4431 bun x playwright test test/snapshot-resume.spec.ts: 1 passed.
Author
Owner

Unicode finding: the 77 B raw and Snapshot controls and 16,454 B / 16,457 B split streams retained all five behemoth clusters in Ghostty's grapheme buffer; the splits were inside a UTF-8 scalar and between base/modifier codepoints. The 50,000,000 B behemoth raw stream at 205x56 also retained all five (291 scrollback rows, WASM 1,053,818,880 B, 21.2 s). No lossy client chunking reproduced. The minimal Snapshot control uses the same VT bytes, not an independently exported native state, so the original #38 comparison remains unconfirmed.

Unicode finding: the 77 B raw and Snapshot controls and 16,454 B / 16,457 B split streams retained all five behemoth clusters in Ghostty's grapheme buffer; the splits were inside a UTF-8 scalar and between base/modifier codepoints. The 50,000,000 B behemoth raw stream at 205x56 also retained all five (291 scrollback rows, WASM 1,053,818,880 B, 21.2 s). No lossy client chunking reproduced. The minimal Snapshot control uses the same VT bytes, not an independently exported native state, so the original #38 comparison remains unconfirmed.
Author
Owner

Unicode investigation committed: 06ea200. Added grapheme-buffer probe and tests for raw Output splits plus Snapshot control, and documented the 50 MB result in apps/tracer/RENDERER-VERIFICATION.md. bun run check passed; Unicode matrix: 1 passed (77 B control, 16,454 B UTF-8 split, 16,457 B grapheme split, 77 B Snapshot); behemoth regression: 1 passed, 205x56, 291 retained rows, 1,053,818,880 B WASM, 21.2 s.

Unicode investigation committed: 06ea200. Added grapheme-buffer probe and tests for raw Output splits plus Snapshot control, and documented the 50 MB result in apps/tracer/RENDERER-VERIFICATION.md. bun run check passed; Unicode matrix: 1 passed (77 B control, 16,454 B UTF-8 split, 16,457 B grapheme split, 77 B Snapshot); behemoth regression: 1 passed, 205x56, 291 retained rows, 1,053,818,880 B WASM, 21.2 s.
Author
Owner

Fast-gate finding: the requested nix shell ... scripts/verify fast returned 1. Exact Rust summary: | fmt | PASS |; | clippy | PASS |; | test | FAIL (101) |; | cargo-deny | PASS |; | static builds | PASS |. The sole failed test was auth::tests::review3_backend_reply_has_a_deadline at crates/infraternal-server/src/auth/review3_tests.rs:619, where the expected Result { call_id: 78, ok: false } did not arrive. Tracer changes do not touch this auth code; I will rerun it separately and run the browser gate.

Fast-gate finding: the requested nix shell ... scripts/verify fast returned 1. Exact Rust summary: | fmt | PASS |; | clippy | PASS |; | test | FAIL (101) |; | cargo-deny | PASS |; | static builds | PASS |. The sole failed test was auth::tests::review3_backend_reply_has_a_deadline at crates/infraternal-server/src/auth/review3_tests.rs:619, where the expected Result { call_id: 78, ok: false } did not arrive. Tracer changes do not touch this auth code; I will rerun it separately and run the browser gate.
Author
Owner

Browser-gate finding: TRACER_TEST_PORT=4431 bun x playwright test finished 17 passed / 1 failed. The failure is test/onboarding.spec.ts layout-shift assertion (first run measured 0.002634910300925926); the isolated rerun also failed its same no-non-input-shifts assertion with 0.0000031190723058127575. Snapshot, Unicode, 50 MB behemoth, and other 14 cases passed. No onboarding UI code is changed on this branch; this existing gate issue is outside the terminal fix.

Browser-gate finding: TRACER_TEST_PORT=4431 bun x playwright test finished 17 passed / 1 failed. The failure is test/onboarding.spec.ts layout-shift assertion (first run measured 0.002634910300925926); the isolated rerun also failed its same no-non-input-shifts assertion with 0.0000031190723058127575. Snapshot, Unicode, 50 MB behemoth, and other 14 cases passed. No onboarding UI code is changed on this branch; this existing gate issue is outside the terminal fix.
Author
Owner

Fast-gate follow-up: the previously failing auth deadline test passed in isolation after waiting its full deadline: 'backend reply deadline: failure at 60003ms; late success ignored; next Ping succeeds; terminal rows=0' and 1 passed in 61.71 s. The full-workspace failure remains a load-sensitive result; no auth code changed.

Fast-gate follow-up: the previously failing auth deadline test passed in isolation after waiting its full deadline: 'backend reply deadline: failure at 60003ms; late success ignored; next Ping succeeds; terminal rows=0' and 1 passed in 61.71 s. The full-workspace failure remains a load-sensitive result; no auth code changed.
Author
Owner

Second full Playwright run: 17 passed / 1 failed at test/terminal.spec.ts:157, local p99 threshold <20 ms received 31.4 ms. The prior full run passed the same 500-key probe at p99 17.7 ms (500/500 painted). During this rerun host load was 10.08/10.25/12.86 with concurrent Rust suites in other worktrees; Snapshot and Unicode tests passed. I will check the probe once more under lower load; no production rendering slowdown has been reproduced.

Second full Playwright run: 17 passed / 1 failed at test/terminal.spec.ts:157, local p99 threshold <20 ms received 31.4 ms. The prior full run passed the same 500-key probe at p99 17.7 ms (500/500 painted). During this rerun host load was 10.08/10.25/12.86 with concurrent Rust suites in other worktrees; Snapshot and Unicode tests passed. I will check the probe once more under lower load; no production rendering slowdown has been reproduced.
Author
Owner

Finished at 06ea200. The Snapshot fix and Unicode investigation are committed; no deploy or push.

The final nix shell nixpkgs#binutils nixpkgs#file nixpkgs#gnumake nixpkgs#perl -c scripts/verify fast exited 0. Verbatim gate summary:
+---------------------------+--------------+
| Rust gate | Result |
+---------------------------+--------------+
| fmt | PASS |
| clippy | PASS |
| test | PASS |
| cargo-deny | PASS |
| static builds | PASS |
+---------------------------+--------------+
7 pass
0 fail
34390 expect() calls
Ran 7 tests across 2 files. [2.87s]

The full TRACER_TEST_PORT=4431 bun x playwright test ran twice. Run 1 ended 1 failed (test/onboarding.spec.ts) and 17 passed (4.5m); run 2 ended 1 failed (test/terminal.spec.ts, p99 31.4 ms vs <20 ms) and 17 passed (3.4m). The 500-key probe passed on run 1 at p99 17.7 ms. Snapshot and Unicode regressions passed in both runs. The browser failures varied with shared host load; no single full Playwright run was all-green. Focused Snapshot, Unicode and 50 MB behemoth tests passed. The final browser build emitted mobile/desktop screenshots under apps/tracer/test-results/ for external review.

Finished at 06ea200. The Snapshot fix and Unicode investigation are committed; no deploy or push. The final `nix shell nixpkgs#binutils nixpkgs#file nixpkgs#gnumake nixpkgs#perl -c scripts/verify fast` exited 0. Verbatim gate summary: +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | PASS | +---------------------------+--------------+ 7 pass 0 fail 34390 expect() calls Ran 7 tests across 2 files. [2.87s] The full `TRACER_TEST_PORT=4431 bun x playwright test` ran twice. Run 1 ended `1 failed` (`test/onboarding.spec.ts`) and `17 passed (4.5m)`; run 2 ended `1 failed` (`test/terminal.spec.ts`, p99 31.4 ms vs <20 ms) and `17 passed (3.4m)`. The 500-key probe passed on run 1 at p99 17.7 ms. Snapshot and Unicode regressions passed in both runs. The browser failures varied with shared host load; no single full Playwright run was all-green. Focused Snapshot, Unicode and 50 MB behemoth tests passed. The final browser build emitted mobile/desktop screenshots under `apps/tracer/test-results/` for external review.
Author
Owner

Fixed and merged into dev (7d17212): an older Snapshot write callback cleared output queued after a newer Snapshot. Regression snapshot-resume.spec.ts passes on dev, and so does unicode-fidelity (no grapheme loss reproduced). It ships with the next redeploy.

Fixed and merged into dev (7d17212): an older Snapshot write callback cleared output queued after a newer Snapshot. Regression snapshot-resume.spec.ts passes on dev, and so does unicode-fidelity (no grapheme loss reproduced). It ships with the next redeploy.
Author
Owner

Closed by the orchestrator during the 2026-10-28 handover (#254), on the owner's request to close finished issues. Evidence: Snapshot paint bug fixed and merged as 7d17212 (in dev), regression snapshot-resume.spec.ts passes. Reopen if anything is still missing.

Closed by the orchestrator during the 2026-10-28 handover (#254), on the owner's request to close finished issues. Evidence: Snapshot paint bug fixed and merged as 7d17212 (in dev), regression snapshot-resume.spec.ts passes. Reopen if anything is still missing.
kayg closed this issue 2026-10-28 17:52:26 +00:00
Sign in to join this conversation.
No labels
launch-qa
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/infraternal#39
No description provided.