Renderer: pin ghostty-web to a source build with the crash fix (D109)

kayg commented

2026-10-06 18:25:38 +00:00

Owner

DECISIONS 109. Replace the npm ghostty-web@0.4.0 in apps/tracer with a source build pinned to an upstream commit that contains the crash fix (bec9e162 / PR #182 line). The spike evidence is in spike-renderer 846c188, spikes/stress/crash/RESULTS.md.

Plan:

A reproducible build script (pinned SHA, pinned toolchain), and a vendored artifact with a SHA256 checksum checked in CI. The licence notice stays with the artifact.
Every crash reproducer and stress fixture becomes a browser regression test in apps/tracer.
Investigate the renderer RSS (861 MiB on behemoth in 0.5.0-rc) and the scrollback configuration before setting a memory budget.
Switch back to the npm release when one contains the fix.

Blocked by: fix-auth2, which is writing t-web (one writer per worktree).

DECISIONS 109. Replace the npm `ghostty-web@0.4.0` in apps/tracer with a source build pinned to an upstream commit that contains the crash fix (bec9e162 / PR #182 line). The spike evidence is in spike-renderer 846c188, `spikes/stress/crash/RESULTS.md`. Plan: - A reproducible build script (pinned SHA, pinned toolchain), and a vendored artifact with a SHA256 checksum checked in CI. The licence notice stays with the artifact. - Every crash reproducer and stress fixture becomes a browser regression test in apps/tracer. - Investigate the renderer RSS (861 MiB on behemoth in 0.5.0-rc) and the scrollback configuration before setting a memory budget. - Switch back to the npm release when one contains the fix. **Blocked by:** fix-auth2, which is writing t-web (one writer per worktree).

kayg referenced this issue

2026-10-06 18:25:45 +00:00

Status board: work in flight (kept current) #24

kayg referenced this issue

2026-10-06 18:26:56 +00:00

Renderer: ghostty-web crash on behemoth/emoji/binary input, upstream fix #31

kayg commented

2026-10-06 19:02:32 +00:00

Author

Owner

Starting #36 on branch t-web at base SHA faa31544c6. Read #36 (no comments), parent renderer history #31 and related #18/#13. Carrying forward spike-renderer 846c188 evidence: PR #182 head 5f0c6b7 passes all 7 reproducers and 3 fixtures; #180 bec9e162 is the crash fix.

Starting #36 on branch t-web at base SHA faa31544c6a26f2aec743e526fb78c1499eaee28. Read #36 (no comments), parent renderer history #31 and related #18/#13. Carrying forward spike-renderer 846c188 evidence: PR #182 head 5f0c6b7 passes all 7 reproducers and 3 fixtures; #180 bec9e162 is the crash fix.

kayg commented

2026-10-06 19:03:36 +00:00

Author

Owner

Baseline finding: tracer production build with ghostty-web 0.4.0 emits 662,832 B deferred Ghostty JS (191,545 B gzip); all JS totals 739,372 B / 220,423 B gzip. Runtime setting is scrollback=10,000 in apps/tracer/src/terminal.ts. Host toolchain: Bun 1.4.2, Node 22.23.3; Zig is not on PATH.

kayg commented

2026-10-06 19:05:36 +00:00

Author

Owner

Baseline browser finding on 0.4.0: all 500 echoes and resume completed, but the existing latency budget failed at p50 19.80 ms / p99 49.40 ms and resume 103.00 ms (apps/tracer/test-results/terminal-first-run-passkey-29742-esume-and-the-latency-probe/error-context.md). Access setup also hit p99 1,870.5 ms. Host load average was 28.04 with active rustc/cc1 jobs, so these are loaded-host samples, not clean comparisons.

kayg referenced this issue

2026-10-06 19:06:15 +00:00

Status board: work in flight (kept current) #24

kayg commented

2026-10-06 19:12:27 +00:00

Author

Owner

Source finding: checkout 5f0c6b7720e5ac241208afe7a2335981ded1b486 has ghostty submodule 5714ed07a1012573261b7b7e3ed2add9c1504496. Build succeeded with Bun 1.4.2 + Zig 0.15.2; ESM is 702,145 B (SHA-256 457cb34280030cac3eb177707ccf838dc74577ba02d106f957f656df33ae8547), WASM 423,939 B (SHA-256 794191ccd9f469ed0d49ebcb3e6b326d64279b91752b71ad80a3d89ff1f26416). Both ghostty-web and its Ghostty submodule are MIT; the exact source SHA is the tested PR #182 head containing bec9e162.

kayg commented

2026-10-06 19:29:08 +00:00

Author

Owner

Step 1 committed as 172c29c. Added the 5f0c6b7 source build, pinned Ghostty submodule 5714ed0, Bun/Zig checks, MIT notices, three built outputs (ESM 702,145 B; WASM 423,939 B; declarations 62,126 B), and fast-gate SHA-256 verification. A fresh checkout reproduced all hashes. Fast gate passed: 5/5 Rust rows and 7/7 protocol tests; checksum rows all OK.

Step 1 committed as 172c29c. Added the 5f0c6b7 source build, pinned Ghostty submodule 5714ed0, Bun/Zig checks, MIT notices, three built outputs (ESM 702,145 B; WASM 423,939 B; declarations 62,126 B), and fast-gate SHA-256 verification. A fresh checkout reproduced all hashes. Fast gate passed: 5/5 Rust rows and 7/7 protocol tests; checksum rows all OK.

kayg commented

2026-10-06 19:35:31 +00:00

Author

Owner

Finding: the first production-page attempt at the 0.4.0 baseline passed the 5,679-byte fixture at Playwright's default 1280x720 viewport (2.8 s). The spike reproducer uses an 80x24 terminal resized to 205x56 before writing. I am aligning the production-page viewport/grid to that trigger before treating this as a regression result.

kayg commented

2026-10-06 19:38:46 +00:00

Author

Owner

Finding: with the tracer grid set to the spike's 205x56 trigger, npm ghostty-web@0.4.0 crashed Chromium on the exact 5,679-byte fixture (SHA256 0047de6d657e0a1f69e16b9cdf8c6dc9343c98086c8b44df9276223f4bf3428f). Playwright error context: apps/tracer/test-results/renderer-regression-render-6b19f-679-bin-and-paints-a-canary/error-context.md reports . The default 1280x720 grid passed; reproducing the recorded geometry is required.

kayg commented

2026-10-06 19:38:52 +00:00

Author

Owner

Finding: with the tracer grid set to the spike 205x56 trigger, npm ghostty-web@0.4.0 crashed Chromium on the exact 5,679-byte fixture (SHA256 0047de6d657e0a1f69e16b9cdf8c6dc9343c98086c8b44df9276223f4bf3428f). Playwright error context: apps/tracer/test-results/renderer-regression-render-6b19f-679-bin-and-paints-a-canary/error-context.md reports Target crashed. The default 1280x720 grid passed; reproducing the recorded geometry is required.

kayg commented

2026-10-06 19:39:28 +00:00

Author

Owner

Reproducer test is now a clean baseline failure before the pin change: bun x playwright test test/renderer-regression.spec.ts --grep minimal-combining-5679.bin failed in 720 ms at 205x56 because Chromium 0.4.0 could not paint the post-fixture canary. Error context: apps/tracer/test-results/renderer-regression-render-6b19f-679-bin-and-paints-a-canary/error-context.md. This records the required failing test before the fix.

kayg commented

2026-10-06 19:41:46 +00:00

Author

Owner

Step 2 committed as af4721d. The tracer now resolves the vendored build and no longer lists ghostty-web@0.4.0; main.js retains its dynamic import. Deferred renderer chunk: 662,832 B / 191,545 B gzip baseline; 675,877 B / 194,425 B after (+13,045 B / +2,880 B gzip). Total JS: 739,372 B / 220,423 B gzip baseline; 752,576 B / 223,651 B after. The minimal reproducer passes at 205x56 after the switch.

Step 2 committed as af4721d. The tracer now resolves the vendored build and no longer lists ghostty-web@0.4.0; main.js retains its dynamic import. Deferred renderer chunk: 662,832 B / 191,545 B gzip baseline; 675,877 B / 194,425 B after (+13,045 B / +2,880 B gzip). Total JS: 739,372 B / 220,423 B gzip baseline; 752,576 B / 223,651 B after. The minimal reproducer passes at 205x56 after the switch.

kayg commented

2026-10-06 19:46:41 +00:00

Author

Owner

Finding: the production-page behemoth.txt test passed after 50,000,000 B at 205x56 with the tracer default scrollback cap of 10,000. Before the canary reset, the renderer reported 291 scrollback rows and 1,053,818,880 B WASM memory (1004.9 MiB), matching the spike despite the much lower cap. This points to retained WASM page or write allocations rather than scrollback rows; I am collecting RSS and a zero-scrollback control.

kayg commented

2026-10-06 19:52:06 +00:00

Author

Owner

Memory probe on Chromium 149.0.7827.55, i9-9900K/8 CPUs, at 205x56: the 10,000-row app setting used 851,144,704 B peak renderer RSS (811.7 MiB) for one session and 6,823,071,744 B total for eight (8 renderer processes; 813.4 MiB/session average). Each WASM memory was 1,053,818,880 B (1005 MiB); reported scrollback was 291 rows. The scrollback=0 probe retained 229,674 rows, so zero means unbounded here and is not a valid no-history control. I am replacing that control with a one-row cap.

kayg commented

2026-10-06 19:55:02 +00:00

Author

Owner

Correction and driver evidence: the pinned ghostty-wasm-api.patch passes scrollback_limit unchanged to Ghostty Screen.max_scrollback; Ghostty Screen.zig documents that limit in bytes and rounds it up to a page. The wrapper calls it lines, so tracer scrollback=10000 is effectively a 10,000-byte cap, retained as one page (291 rows). A one-byte cap also retained 291 rows. Both runs grew WASM memory to 1,053,818,880 B (1005 MiB); renderer peak RSS was 851,156,992 B at 10,000 and 847,368,192 B at 1. The page allocator/high-water growth drives the 1 GiB WASM buffer, not retained history. Latest 8-session run: 6,818,861,056 B aggregate peak RSS across 8 renderer processes (812.9 MiB/session average). No budgets changed.

kayg commented

2026-10-06 19:55:50 +00:00

Author

Owner

Step 3 committed as 78d2d5f. Test-double-only diagnostics now expose configured scrollback, retained rows, output sequence, grid, and WASM memory; only mock mode can change the cap. The normal tracer remains at 10,000. bun run check and bun run build passed; main.js still dynamically imports the deferred renderer chunk.

Step 3 committed as 78d2d5f. Test-double-only diagnostics now expose configured scrollback, retained rows, output sequence, grid, and WASM memory; only mock mode can change the cap. The normal tracer remains at 10,000. bun run check and bun run build passed; main.js still dynamically imports the deferred renderer chunk.

kayg commented

2026-10-06 20:00:45 +00:00

Author

Owner

Step 4 committed as 9551130. Gate output: cd apps/tracer && bun x playwright test — 14 passed (2.0m). The 10 new production-Chromium cases cover all seven reproducers and all three full fixtures at 205x56; each survived and painted the canary. The 500-key probe measured pinned p50/p99/resume 14.90/15.70/77.60 ms versus a same-host 0.4.0 run at 14.90/16.40/78.70 ms. Both pass the unchanged 20 ms local p99 gate; this is one sequential sample, not the perf-VM budget verdict.

Step 4 committed as 9551130. Gate output: cd apps/tracer && bun x playwright test — 14 passed (2.0m). The 10 new production-Chromium cases cover all seven reproducers and all three full fixtures at 205x56; each survived and painted the canary. The 500-key probe measured pinned p50/p99/resume 14.90/15.70/77.60 ms versus a same-host 0.4.0 run at 14.90/16.40/78.70 ms. Both pass the unchanged 20 ms local p99 gate; this is one sequential sample, not the perf-VM budget verdict.

kayg commented

2026-10-06 20:03:30 +00:00

Author

Owner

Step 5 committed as 845c293. Added apps/tracer/RENDERER-VERIFICATION.md and linked it from the tracer README. It records the 5f0c6b7/Ghostty 5714ed0 source pin, three checksums, 10 production Chromium payloads, 1/8-session RSS, bundle deltas (+13,263 B raw / +3,240 B gzip across JS), and matched probe p99/resume 16.40/78.70 ms on 0.4.0 vs 15.70/77.60 ms pinned. Final fast gate is running.

Step 5 committed as 845c293. Added apps/tracer/RENDERER-VERIFICATION.md and linked it from the tracer README. It records the 5f0c6b7/Ghostty 5714ed0 source pin, three checksums, 10 production Chromium payloads, 1/8-session RSS, bundle deltas (+13,263 B raw / +3,240 B gzip across JS), and matched probe p99/resume 16.40/78.70 ms on 0.4.0 vs 15.70/77.60 ms pinned. Final fast gate is running.

kayg commented

2026-10-06 20:04:08 +00:00

Author

Owner

Finished at head 845c293 (docs-only slice after code head 9551130).

Gate command: nix shell nixpkgs#binutils nixpkgs#file nixpkgs#gnumake nixpkgs#perl -c scripts/verify fast
Verbatim summary output:

| Rust gate                 | Result       |
| fmt                       | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | PASS         |
dist/ghostty-web.js: OK
dist/ghostty-vt.wasm: OK
dist/index.d.ts: OK
 7 pass
 0 fail
 34390 expect() calls
Ran 7 tests across 2 files. [2.65s]

Gate command: cd apps/tracer && bun x playwright test
Verbatim output: 14 passed (2.0m)
The browser run covered all seven reproducers and all three full fixtures at 205x56, with canary paint after each. The docs commit changed no executable code. No budgets changed; no deploy or issue close.

Finished at head 845c293 (docs-only slice after code head 9551130). Gate command: `nix shell nixpkgs#binutils nixpkgs#file nixpkgs#gnumake nixpkgs#perl -c scripts/verify fast` Verbatim summary output: ``` | Rust gate | Result | | fmt | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | PASS | dist/ghostty-web.js: OK dist/ghostty-vt.wasm: OK dist/index.d.ts: OK 7 pass 0 fail 34390 expect() calls Ran 7 tests across 2 files. [2.65s] ``` Gate command: `cd apps/tracer && bun x playwright test` Verbatim output: `14 passed (2.0m)` The browser run covered all seven reproducers and all three full fixtures at 205x56, with canary paint after each. The docs commit changed no executable code. No budgets changed; no deploy or issue close.

kayg referenced this issue

2026-10-06 20:05:06 +00:00

Perf: browser renderer keeps ~812 MiB per session after a flood (WASM memory never shrinks) #38

kayg commented

2026-10-06 20:05:06 +00:00

Author

Owner

Orchestrator check of ghostty-pin (t-web 845c293): accepted.

5f0c6b7 (contains bec9e162) is vendored with a reproducible build; the hashes match on rebuild.
14/14 Playwright tests pass, including the 7 reproducers and 3 fixtures; the gate is green.
Bundle +3 KB gzip. Keystroke p50/p99 is unchanged (14.9/15.7 ms).
The memory finding (812 MiB per session after a flood) is filed separately.

Orchestrator check of ghostty-pin (t-web 845c293): accepted. - 5f0c6b7 (contains bec9e162) is vendored with a reproducible build; the hashes match on rebuild. - 14/14 Playwright tests pass, including the 7 reproducers and 3 fixtures; the gate is green. - Bundle +3 KB gzip. Keystroke p50/p99 is unchanged (14.9/15.7 ms). - The memory finding (812 MiB per session after a flood) is filed separately.

kayg referenced this issue

2026-10-06 20:05:28 +00:00

Status board: work in flight (kept current) #24

kayg referenced this issue

2026-10-06 20:35:41 +00:00

Perf: browser renderer keeps ~812 MiB per session after a flood (WASM memory never shrinks) #38

kayg referenced this issue

2026-10-06 23:26:27 +00:00

S1 integration: merge tracer lanes into dev, deploy to monitor01 #34