Perf: browser renderer keeps ~812 MiB per session after a flood (WASM memory never shrinks)

kayg commented

2026-10-06 20:05:06 +00:00

Owner

Found by ghostty-pin (#36, t-web 845c293, apps/tracer/RENDERER-VERIFICATION.md): after a 50 MB burst (behemoth), the browser renderer holds ~812 MiB per session, 6.8 GB for 8 sessions. The scrollback cap barely matters (a 1-byte cap gives similar RSS). The driver is WASM linear-memory growth during the write. WASM memory never shrinks, so a tab that has seen one big flood keeps the memory until the terminal instance is dropped.

The server already holds the authoritative VT state natively (vt-ghostty), so the browser does not need to parse every byte. Options to measure (spike, no decision):

(a) Coalesce at the server: when a client falls behind (credit exhausted, or bytes/frame above a threshold), skip the raw bytes and send a Snapshot at display rate. The browser then parses at most one screen per frame. This also lowers flood latency and bandwidth (contracts/shim.md already has the Snapshot frame; check whether the leg-C wire needs anything additive).
(b) Recycle the renderer: when WASM memory passes a threshold, rebuild the terminal from a Snapshot (as on a theme change) and drop the old instance. Measure the visible hitch.
(c) Both.

Measure: peak and steady renderer RSS after behemoth, mixed and binary floods; flood wall time to the final frame; keystroke latency during a flood; and correctness (the final screen equals the native VT).
Budgets: D41/D81 memory per session × expected sessions.

Found by ghostty-pin (#36, t-web 845c293, apps/tracer/RENDERER-VERIFICATION.md): after a 50 MB burst (behemoth), the browser renderer holds **~812 MiB per session**, 6.8 GB for 8 sessions. The scrollback cap barely matters (a 1-byte cap gives similar RSS). The driver is WASM linear-memory growth during the write. WASM memory never shrinks, so a tab that has seen one big flood keeps the memory until the terminal instance is dropped. The server already holds the authoritative VT state natively (vt-ghostty), so the browser does not need to parse every byte. Options to measure (spike, no decision): - **(a) Coalesce at the server:** when a client falls behind (credit exhausted, or bytes/frame above a threshold), skip the raw bytes and send a Snapshot at display rate. The browser then parses at most one screen per frame. This also lowers flood latency and bandwidth (contracts/shim.md already has the Snapshot frame; check whether the leg-C wire needs anything additive). - **(b) Recycle the renderer:** when WASM memory passes a threshold, rebuild the terminal from a Snapshot (as on a theme change) and drop the old instance. Measure the visible hitch. - (c) Both. Measure: peak and steady renderer RSS after behemoth, mixed and binary floods; flood wall time to the final frame; keystroke latency during a flood; and correctness (the final screen equals the native VT). Budgets: D41/D81 memory per session × expected sessions.

kayg referenced this issue

2026-10-06 20:05:28 +00:00

Status board: work in flight (kept current) #24

kayg referenced this issue

2026-10-06 20:34:55 +00:00

Status board: work in flight (kept current) #24

kayg commented

2026-10-06 20:35:41 +00:00

Author

Owner

Starting spike on branch spike-mem at base SHA 845c293a70. Read #38 (no comments) and all 18 comments on parent #36; continuing its pinned production Chromium corpus and memory probe. Measuring raw, Snapshot coalescing, renderer recycle, and combined at 1/8 sessions.

Starting spike on branch spike-mem at base SHA 845c293a70d6f86e05a6f46794462009793df973. Read #38 (no comments) and all 18 comments on parent #36; continuing its pinned production Chromium corpus and memory probe. Measuring raw, Snapshot coalescing, renderer recycle, and combined at 1/8 sessions.

kayg commented

2026-10-06 20:52:31 +00:00

Author

Owner

Native Snapshot finding (vt-ghostty copy at 544a2e1; binaries and targets stay under ~/.cache/codex-scratch/spike-mem): 205x56 checkpoints are faithful for behemoth 50/50 (final 84,556 B), mixed 20/20 (4,131 B), log 5/5 (42,564 B), and yes 50/50 (11,929 B). Binary 15/20 are faithful; native bounded painter rejects 5/20 states, so the spike uses the native approximate repaint at those checkpoints. Final binary checkpoint is faithful (11,343 B). No source changes made in vt-ghostty; this limitation will be called out in MEMORY-SPIKE.md.

Native Snapshot finding (vt-ghostty copy at 544a2e1; binaries and targets stay under ~/.cache/codex-scratch/spike-mem): 205x56 checkpoints are faithful for behemoth 50/50 (final 84,556 B), mixed 20/20 (4,131 B), log 5/5 (42,564 B), and yes 50/50 (11,929 B). Binary 15/20 are faithful; native bounded painter rejects 5/20 states, so the spike uses the native approximate repaint at those checkpoints. Final binary checkpoint is faithful (11,343 B). No source changes made in vt-ghostty; this limitation will be called out in MEMORY-SPIKE.md.

kayg commented

2026-10-06 23:12:33 +00:00

Author

Owner

Probe finding: rapid scenario resets left prior test-double sockets in the fixture target set, causing HTTP 409; reset now terminates and clears those sockets in apps/tracer/test/mock-server.ts. A Snapshot-fed 5 MB combined probe accepted the 2-byte echo (received sequence 5,000,002) but the renderer paint sequence stayed at 5,000,000; latency is unmeasured for that sample. Raw/recycled 5 MB one-key probes painted in 1,169/1,141 ms (apps/tracer/test/renderer-memory-spike.ts; Chromium 149, loaded host).

kayg commented

2026-10-06 23:16:34 +00:00

Author

Owner

Option (a) measured in apps/tracer/MEMORY-SPIKE.md: behemoth steady renderer RSS fell from 814.9 to 202.0 MiB/session at 1 session and 807.6 to 186.5 at 8; final paint fell from 35.503/34.570 s to 2.864/2.816 s. Native coalescing sent 50 Snapshots and 4,245,070 payload bytes/session; Snapshot-fed final grid text matched native VT, while raw behemoth lost grapheme modifiers.

kayg commented

2026-10-06 23:16:35 +00:00

Author

Owner

Option (b) measured in apps/tracer/MEMORY-SPIKE.md at a 256 MiB WASM threshold: behemoth steady RSS was 378.1 MiB/session at 1 and 339.9 at 8; mixed was 381.0/336.8. Swap paint was 143–318 ms (83–200 ms max frame gap). Old WebAssembly.Memory WeakRefs stayed live after forced GC; RSS fell about 1.4 MiB/session.

kayg commented

2026-10-06 23:16:35 +00:00

Author

Owner

Option (c) measured in apps/tracer/MEMORY-SPIKE.md: behemoth combined steady RSS was 197.4 MiB/session at 1 and 185.7 at 8, with final paint at 2.381/2.607 s. It was close to coalescing alone; Snapshot-fed streams did not cross the recycle threshold. A post-Snapshot echo ordering issue blocks a valid combined-mode latency claim.

kayg commented

2026-10-06 23:16:42 +00:00

Author

Owner

Finding: the behemoth raw renderer text diverges from the native Snapshot on its Unicode sample line: raw omits skin-tone modifiers, ZWJ emoji and regional-indicator pairs; Snapshot/coalesced, recycled and combined modes retain the full clusters for sessions 1 and 8. Evidence and production screenshots are in apps/tracer/MEMORY-SPIKE.md and the attached run captures.

kayg commented

2026-10-06 23:16:49 +00:00

Author

Owner

Finding: during a combined 5 MB log probe, terminal diagnostics showed receivedSequence=5,000,002, outputQueueBytes=0, no output write in flight, but painted outputSequence stayed at 5,000,000. This followed a post-Snapshot echoed key and blocked latency measurement. Evidence: apps/tracer/src/terminal.ts diagnostics and apps/tracer/MEMORY-SPIKE.md. The snapshot/next-output ordering needs review before production coalescing.

kayg commented

2026-10-06 23:17:22 +00:00

Author

Owner

Step complete: added the 5,000,000-byte structured log and 50,000,000-byte yes fixture plus manifest entries. Commit 78cf54c. SHA-256 values are 8d9a88f3e33be1c7de3deecd2589a2e6f52f1d42ef101073bd65dda68c8c89ca and bf44164239a7e217299f833d1c6f6b50c9814e0ae1c752d1cf78b151de60a009. bun run check passed.

Step complete: added the 5,000,000-byte structured log and 50,000,000-byte yes fixture plus manifest entries. Commit 78cf54c. SHA-256 values are 8d9a88f3e33be1c7de3deecd2589a2e6f52f1d42ef101073bd65dda68c8c89ca and bf44164239a7e217299f833d1c6f6b50c9814e0ae1c752d1cf78b151de60a009. bun run check passed.

kayg commented

2026-10-06 23:17:43 +00:00

Author

Owner

Step complete: mock server can validate and stream native Snapshot checkpoints at 60 Hz, provide a one-shot final Snapshot for recycling, report delivery counters, and isolate fixture runs. Spike-only echo filtering stopped binary CSI reply loops (13,781-byte feedback observed before filtering). Commit 1603ed8; bun run check passed. Native payload totals were behemoth 4,245,070 B, mixed 1,398,631 B, binary 667,041 B, log 212,830 B, yes 596,450 B.

Step complete: mock server can validate and stream native Snapshot checkpoints at 60 Hz, provide a one-shot final Snapshot for recycling, report delivery counters, and isolate fixture runs. Spike-only echo filtering stopped binary CSI reply loops (13,781-byte feedback observed before filtering). Commit 1603ed8; bun run check passed. Native payload totals were behemoth 4,245,070 B, mixed 1,398,631 B, binary 667,041 B, log 212,830 B, yes 596,450 B.

kayg commented

2026-10-06 23:18:00 +00:00

Author

Owner

Step complete: terminal creation now shares one factory, and the mock-only hook can recycle at a Snapshot boundary while measuring old WASM liveness and frame gap. Commit a64acbc; tracer typecheck and production build passed. Recycle triggered at 256 MiB; behemoth swap paint was 143 ms (1 session) and 193/218 ms (sessions 4/6 of 8), with old WebAssembly.Memory still live after forced GC.

Step complete: terminal creation now shares one factory, and the mock-only hook can recycle at a Snapshot boundary while measuring old WASM liveness and frame gap. Commit a64acbc; tracer typecheck and production build passed. Recycle triggered at 256 MiB; behemoth swap paint was 143 ms (1 session) and 193/218 ms (sessions 4/6 of 8), with old WebAssembly.Memory still live after forced GC.

kayg commented

2026-10-06 23:18:30 +00:00

Author

Owner

Step complete: production Chromium Playwright matrix and optional screenshots are in the harness. Commit b10a9ce; bun run check passed. The no-probe matrix completed 40 scenarios with zero page errors; behemoth raw/coal/recycle/combined at 1 session measured 814.9/202.0/378.1/197.4 MiB steady per session and 35.503/2.864/8.941/2.381 s to final paint.

Step complete: production Chromium Playwright matrix and optional screenshots are in the harness. Commit b10a9ce; bun run check passed. The no-probe matrix completed 40 scenarios with zero page errors; behemoth raw/coal/recycle/combined at 1 session measured 814.9/202.0/378.1/197.4 MiB steady per session and 35.503/2.864/8.941/2.381 s to final paint.

kayg commented

2026-10-06 23:19:03 +00:00

Author

Owner

Report committed at a82523c: apps/tracer/MEMORY-SPIKE.md contains all 40 no-probe RSS/final-frame/screen rows, native Snapshot payloads, recycle/GC measurements, latency probe limits, wire sketch, Safari plan, and owner questions. Behemoth coalescing reduced steady RSS 75.2%/76.9% at 1/8 sessions; Snapshot-fed screens matched native text and raw behemoth did not. TypeScript check and production build passed; the required repository fast gate remains to run.

Report committed at a82523c: apps/tracer/MEMORY-SPIKE.md contains all 40 no-probe RSS/final-frame/screen rows, native Snapshot payloads, recycle/GC measurements, latency probe limits, wire sketch, Safari plan, and owner questions. Behemoth coalescing reduced steady RSS 75.2%/76.9% at 1/8 sessions; Snapshot-fed screens matched native text and raw behemoth did not. TypeScript check and production build passed; the required repository fast gate remains to run.

kayg commented

2026-10-06 23:28:07 +00:00

Author

Owner

Finished on spike-mem at a82523c4c4; worktree is clean. The first fast-gate attempt exposed the repo-relative target-path assumption; reran with target symlinked to the required scratch build directory, then removed the symlink and scratch artifacts. scripts/verify fast passed. Verbatim gate summary:

==> Vendored Ghostty artifact checksums
dist/ghostty-web.js: OK
dist/ghostty-vt.wasm: OK
dist/index.d.ts: OK

7 pass
0 fail
34390 expect() calls
Ran 7 tests across 2 files. [1.80s]

Finished on spike-mem at a82523c4c423c79abbda106536214d419b39489e; worktree is clean. The first fast-gate attempt exposed the repo-relative target-path assumption; reran with target symlinked to the required scratch build directory, then removed the symlink and scratch artifacts. scripts/verify fast passed. Verbatim gate summary: +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | PASS | +---------------------------+--------------+ ==> Vendored Ghostty artifact checksums dist/ghostty-web.js: OK dist/ghostty-vt.wasm: OK dist/index.d.ts: OK 7 pass 0 fail 34390 expect() calls Ran 7 tests across 2 files. [1.80s]

kayg referenced this issue

2026-10-06 23:29:24 +00:00

Bug: echo after a Snapshot is received but not painted in the browser #39

kayg commented

2026-10-06 23:29:31 +00:00

Author

Owner

Owner question for the morning (from spike-mem a82523c): adopt server-side Snapshot coalescing for clients that fall behind? Behemoth 50 MB: renderer RSS 815 → 202 MiB per session (8 sessions: 808 → 187), final paint 35.5 s → 2.9 s. Renderer recycle was rejected by the evidence: the old WASM memory stays live after GC, with a 143–318 ms swap hitch. It needs an additive wire change on leg B/C. The post-Snapshot paint bug it exposed is #39 (fix job running).

Owner question for the morning (from spike-mem a82523c): adopt **server-side Snapshot coalescing** for clients that fall behind? Behemoth 50 MB: renderer RSS 815 → 202 MiB per session (8 sessions: 808 → 187), final paint 35.5 s → 2.9 s. Renderer recycle was rejected by the evidence: the old WASM memory stays live after GC, with a 143–318 ms swap hitch. It needs an additive wire change on leg B/C. The post-Snapshot paint bug it exposed is #39 (fix job running).

kayg referenced this issue

2026-10-06 23:29:32 +00:00

Status board: work in flight (kept current) #24

kayg referenced this issue

2026-10-06 23:29:51 +00:00

Bug: echo after a Snapshot is received but not painted in the browser #39

kayg referenced this issue

2026-10-06 23:51:37 +00:00

Bug: echo after a Snapshot is received but not painted in the browser #39

kayg referenced this issue from a commit

2026-10-10 09:21:10 +00:00

Add deterministic log and yes flood fixtures for #38