Renderer: a new terminal can start with cells from a closed terminal (WASM page reuse)

kayg commented

2026-10-17 14:52:02 +00:00

Owner

A new terminal window can start with the cells of a terminal that was closed earlier in the same browser tab.

Seen in the real-logins demo (#172, 2026-10-17): a new window on ash1 showed rows from the build-host console session (its prompt shim-test) and from an earlier root window on ash1, mixed with the new session's own output (root-testwhoami, then kayg-testwhoami after whoami overwrote the first four cells). Seen twice: after a page reload, and after the add-host flow.

Cause (client side)

The browser renderer runs every terminal in one shared WASM instance. The pinned Ghostty (5714ed07, apps/tracer/vendor/ghostty-web/build.sh:7) zeroes a new page only under runtime safety:

// src/terminal/PageList.zig (pin 5714ed07), lines 344 and 2678
if (comptime std.debug.runtime_safety) @memset(page_buf, 0);

The comment assumes the OS gives zeroed pages. In the ReleaseSmall WASM build the allocator reuses freed slots without zeroing, so a new terminal's first page holds the cells of the last terminal that was freed. Newer Ghostty zeroes on freestanding too (or builtin.os.tag == .freestanding).

Paths in the tracer that free a WASM terminal and create another:

closing a window, then opening a window (TerminalClient.open → new Terminal, apps/tracer/src/terminal.ts ~line 433),
Terminal.reset(), which frees and recreates the WASM terminal (vendor dist ghostty-web.js:2949), called by restartCursors() (terminal.ts ~line 1157): session switch in a window (retarget), "open a new shell" and a reconnect the server did not resume.

Not server side

The server and shim send the right stream. The leaked text is the content of a terminal this same tab had shown before. It does not cross browser tabs, people or accounts on the server. Within one tab it does cross sessions, hosts and host accounts (root output in a kayg window). It is also in readScreenText and selection/copy until the rows are overwritten.

Affected builds

The same vendored WASM (blob 43a7694bee) is in release-prod-live, dev and the demo build (290ea53f9).

Evidence (video frames, a minimal WASM reproduction) and the failing tests follow in comments.

A new terminal window can start with the cells of a terminal that was closed earlier in the same browser tab. Seen in the real-logins demo (#172, 2026-10-17): a new window on ash1 showed rows from the build-host console session (its prompt `shim-test`) and from an earlier root window on ash1, mixed with the new session's own output (`root-testwhoami`, then `kayg-testwhoami` after `whoami` overwrote the first four cells). Seen twice: after a page reload, and after the add-host flow. ## Cause (client side) The browser renderer runs every terminal in one shared WASM instance. The pinned Ghostty (`5714ed07`, apps/tracer/vendor/ghostty-web/build.sh:7) zeroes a new page only under runtime safety: ```zig // src/terminal/PageList.zig (pin 5714ed07), lines 344 and 2678 if (comptime std.debug.runtime_safety) @memset(page_buf, 0); ``` The comment assumes the OS gives zeroed pages. In the ReleaseSmall WASM build the allocator reuses freed slots without zeroing, so a new terminal's first page holds the cells of the last terminal that was freed. Newer Ghostty zeroes on freestanding too (`or builtin.os.tag == .freestanding`). Paths in the tracer that free a WASM terminal and create another: - closing a window, then opening a window (TerminalClient.open → `new Terminal`, apps/tracer/src/terminal.ts ~line 433), - `Terminal.reset()`, which frees and recreates the WASM terminal (vendor dist ghostty-web.js:2949), called by `restartCursors()` (terminal.ts ~line 1157): session switch in a window (`retarget`), "open a new shell" and a reconnect the server did not resume. ## Not server side The server and shim send the right stream. The leaked text is the content of a terminal this same tab had shown before. It does not cross browser tabs, people or accounts on the server. Within one tab it does cross sessions, hosts and host accounts (root output in a kayg window). It is also in `readScreenText` and selection/copy until the rows are overwritten. ## Affected builds The same vendored WASM (blob 43a7694bee) is in release-prod-live, dev and the demo build (290ea53f9). Evidence (video frames, a minimal WASM reproduction) and the failing tests follow in comments.

frames-245-340s.png

2.2 MiB

frames-476-516s-new-window.png

950 KiB

probe-mixed-output.png

44 KiB

kayg commented

2026-10-17 14:54:36 +00:00

Author

Owner

Start: branch investigate-cross-session, base 3048580c1 (origin/release-prod-live). Failing tests: e0308c321.

Evidence from the demo video (rec/raw.webm)

~285–340 s (after the reload): a window shows the build-host console (%, shim-testwhoami, kayg, shim-test). It is replaced by an ash1 root window whose first screen is those same rows with the root session's output written over them: root@ash1:~# whoami / root-testwhoami / root@ash1:~# / shim-test.
~503 s (after re-adding ash1): the new kayg window opens with exactly the screen of the closed root window (root@ash1:~# whoami, root-testwhoami, root@ash1:~#, shim-test). Typing whoami then turns row 2 into kayg-testwhoami, because output that does not erase the line keeps the old cells after it. The stale rows stay in the terminal's state until something overwrites them. They are not just left-over pixels.

Minimal reproduction (vendored WASM, no browser)

A: [ "shim-test% whoami", "secret-from-session-A" ]   // write, then free
B fresh: [ "shim-test% whoami", "secret-from-session-A" ]  // createTerminal right after
B after "root\r\n": [ "root-test% whoami", "secret-from-session-A" ]

Scope checks (same script):

Only the new terminal's first page leaks: 24 of 24 rows after writing 5000 lines into B, and 0 lines from later pages. The scrolled-row clear from #180 covers the rest.
Writing RIS (ESC c) right after creation clears it: 0 rows leak, also across resizes up to 300×100 and on entering the alternate screen.

Code path

apps/tracer/vendor/ghostty-web/build.sh:7 pins Ghostty 5714ed07. src/terminal/PageList.zig:344 and :2678 at that pin: if (comptime std.debug.runtime_safety) @memset(page_buf, 0);. The release WASM build has no runtime safety.
apps/tracer/src/terminal.ts:433 TerminalClient.open → createRenderer → new Terminal (one per window, sharing the one WASM instance from init()).
apps/tracer/src/canvas/sessions/live.ts:300 Binding.dispose → client.dispose() frees the WASM terminal. The next window's terminal gets that memory.
vendor/ghostty-web/dist/ghostty-web.js:2949 Terminal.reset() = free + createTerminal, reached from terminal.ts:1157 restartCursors(). That is used by retarget (session switch), openNewSession and an unresumed reconnect. A session switch therefore starts with the previous session's screen in the same window.

Server side ruled out

The shim's VT state uses native libghostty-vt: mmap gives zeroed pages, and its newer pin also zeroes on freestanding. The leaked text in both demo cases is content this tab had already shown in another window.

Tests (fail on release-prod-live)

apps/tracer/src/renderer-fresh-terminal.test.ts (bun): fails, the fresh screen holds the old rows.
apps/tracer/test/renderer-fresh-terminal.spec.ts (Chromium, port 4477): 2 failed. A window opened after another closed shows shim-test% whoami / from-the-closed-session; after reset it shows root@host:~# whoami / root.

Start: branch `investigate-cross-session`, base `3048580c1` (origin/release-prod-live). Failing tests: `e0308c321`. ## Evidence from the demo video (rec/raw.webm) - ~285–340 s (after the reload): a window shows the build-host console (`%`, `shim-testwhoami`, `kayg`, `shim-test`). It is replaced by an ash1 root window whose first screen is those same rows with the root session's output written over them: `root@ash1:~# whoami` / `root-testwhoami` / `root@ash1:~#` / `shim-test`. - ~503 s (after re-adding ash1): the new kayg window opens with exactly the screen of the closed root window (`root@ash1:~# whoami`, `root-testwhoami`, `root@ash1:~#`, `shim-test`). Typing `whoami` then turns row 2 into `kayg-testwhoami`, because output that does not erase the line keeps the old cells after it. The stale rows stay in the terminal's state until something overwrites them. They are not just left-over pixels. ## Minimal reproduction (vendored WASM, no browser) ``` A: [ "shim-test% whoami", "secret-from-session-A" ] // write, then free B fresh: [ "shim-test% whoami", "secret-from-session-A" ] // createTerminal right after B after "root\r\n": [ "root-test% whoami", "secret-from-session-A" ] ``` Scope checks (same script): - Only the new terminal's first page leaks: 24 of 24 rows after writing 5000 lines into B, and 0 lines from later pages. The scrolled-row clear from #180 covers the rest. - Writing RIS (`ESC c`) right after creation clears it: 0 rows leak, also across resizes up to 300×100 and on entering the alternate screen. ## Code path - `apps/tracer/vendor/ghostty-web/build.sh:7` pins Ghostty `5714ed07`. `src/terminal/PageList.zig:344` and `:2678` at that pin: `if (comptime std.debug.runtime_safety) @memset(page_buf, 0);`. The release WASM build has no runtime safety. - `apps/tracer/src/terminal.ts:433` `TerminalClient.open` → `createRenderer` → `new Terminal` (one per window, sharing the one WASM instance from `init()`). - `apps/tracer/src/canvas/sessions/live.ts:300` `Binding.dispose` → `client.dispose()` frees the WASM terminal. The next window's terminal gets that memory. - `vendor/ghostty-web/dist/ghostty-web.js:2949` `Terminal.reset()` = free + createTerminal, reached from `terminal.ts:1157` `restartCursors()`. That is used by `retarget` (session switch), `openNewSession` and an unresumed reconnect. A session switch therefore starts with the previous session's screen in the same window. ## Server side ruled out The shim's VT state uses native libghostty-vt: mmap gives zeroed pages, and its newer pin also zeroes on freestanding. The leaked text in both demo cases is content this tab had already shown in another window. ## Tests (fail on release-prod-live) - `apps/tracer/src/renderer-fresh-terminal.test.ts` (bun): fails, the fresh screen holds the old rows. - `apps/tracer/test/renderer-fresh-terminal.spec.ts` (Chromium, port 4477): 2 failed. A window opened after another closed shows `shim-test% whoami` / `from-the-closed-session`; after reset it shows `root@host:~# whoami` / `root`.

kayg commented

2026-10-17 14:54:59 +00:00

Author

Owner

Demo frames (rec/raw.webm): 245–340 s, the window that turns from build-host into ash1 root; 476–516 s, the new kayg window after re-adding ash1; and the driver's own probe screenshot.

Demo frames (rec/raw.webm): 245–340 s, the window that turns from build-host into ash1 root; 476–516 s, the new kayg window after re-adding ash1; and the driver's own probe screenshot. ![frames-245-340s.png](https://git.kayg.org/attachments/f0643a21-c6bd-423d-908e-d916955b47fd) ![frames-476-516s-new-window.png](https://git.kayg.org/attachments/12602bcd-c317-4623-82cd-4e242d3568ab) ![probe-mixed-output.png](https://git.kayg.org/attachments/c4051a43-ff74-4ea2-8229-565e45eb166f)

kayg commented

2026-10-17 15:18:47 +00:00

Author

Owner

Finish: head e0308c321 on investigate-cross-session. It contains the failing tests only. There is no fix.

Gate scripts/verify fast. The TS part passes, including bun run check with the new files. The new tests are not in the gate list. The Rust table:

| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (101)   |

static builds failed because the host disk is full ("No space left on device", / at 100%, 441 MB free). The change is not the cause. I deleted my 22 GB target dir afterwards. The first run failed earlier, on the latency-proxy self-test (p50 344 ms against a 300 ms target), because the host was under load.

Smallest correct fix: zero the page buffer on freestanding in the vendored WASM build. That means a ghostty-web build patch applied to the Ghostty submodule after submodule update in vendor/ghostty-web/build.sh, adding or builtin.os.tag == .freestanding at both @memset(page_buf, 0) sites in src/terminal/PageList.zig, as newer Ghostty does. Then rebuild dist/, update SHA256SUMS and add both tests to the gate. A JS-only stopgap also works: write ESC c to the WASM terminal right after new Terminal/open and after every reset(), before the first paint. In my probes it cleared every case I tried, but it does not cover allocation paths I did not exercise.

Finish: head `e0308c321` on `investigate-cross-session`. It contains the failing tests only. There is no fix. Gate `scripts/verify fast`. The TS part passes, including `bun run check` with the new files. The new tests are not in the gate list. The Rust table: ``` | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (101) | ``` `static builds` failed because the host disk is full ("No space left on device", / at 100%, 441 MB free). The change is not the cause. I deleted my 22 GB target dir afterwards. The first run failed earlier, on the latency-proxy self-test (p50 344 ms against a 300 ms target), because the host was under load. Smallest correct fix: zero the page buffer on freestanding in the vendored WASM build. That means a ghostty-web build patch applied to the Ghostty submodule after `submodule update` in `vendor/ghostty-web/build.sh`, adding `or builtin.os.tag == .freestanding` at both `@memset(page_buf, 0)` sites in `src/terminal/PageList.zig`, as newer Ghostty does. Then rebuild `dist/`, update SHA256SUMS and add both tests to the gate. A JS-only stopgap also works: write `ESC c` to the WASM terminal right after `new Terminal`/`open` and after every `reset()`, before the first paint. In my probes it cleared every case I tried, but it does not cover allocation paths I did not exercise.

kayg commented

2026-10-17 17:17:52 +00:00

Author

Owner

Fixed and deployed web-only 2026-10-17 ~23:40 IST as cef7224ee (merge of renderer-zero-pages 47080f0aa: the pinned engine zeroes page memory in the freestanding build). release-prod-live = cef7224ee. Both investigation tests pass. Reload the tab to get it.

Fixed and deployed web-only 2026-10-17 ~23:40 IST as cef7224ee (merge of renderer-zero-pages 47080f0aa: the pinned engine zeroes page memory in the freestanding build). release-prod-live = cef7224ee. Both investigation tests pass. Reload the tab to get it.

kayg referenced this issue

2026-10-17 21:04:57 +00:00

Add-host retry is blocked forever by a failed attempt's rollback receipt (production) #230

kayg referenced this issue

2026-10-17 21:06:11 +00:00

Add-host retry is blocked forever by a failed attempt's rollback receipt (production) #230

kayg referenced this issue

2026-10-18 02:49:20 +00:00

Scheduled ping closes a healthy host connection when the control queue is full #234

kayg referenced this issue

2026-10-18 02:49:20 +00:00

Add-host ridiculously fast: under 1.5 s local, under 4 s at 150 ms #189

kayg referenced this issue

2026-10-20 20:42:58 +00:00

Status board: work in flight (kept current) #24

kayg referenced this issue

2026-10-20 21:49:38 +00:00

Status board: work in flight (kept current) #24

kayg commented

2026-10-28 17:53:24 +00:00

Author

Owner

Closed by the orchestrator during the 2026-10-28 handover (#254), on the owner's request to close finished issues. Evidence: WASM page reuse fixed by renderer-zero-pages, merged as cef7224ee (in dev and prod) and deployed web-only 2026-10-17. Reopen if anything is still missing.

Closed by the orchestrator during the 2026-10-28 handover (#254), on the owner's request to close finished issues. Evidence: WASM page reuse fixed by renderer-zero-pages, merged as cef7224ee (in dev and prod) and deployed web-only 2026-10-17. Reopen if anything is still missing.

kayg closed this issue

2026-10-28 17:53:24 +00:00

~~kayg referenced this issue 2026-10-28 17:53:55 +00:00~~

Orchestrator handover 2026-10-28: moving to a new machine #254

kayg referenced this issue

2026-10-28 18:00:45 +00:00

Orchestrator handover 2026-10-28: moving to a new machine #254

kayg referenced this issue

2026-10-31 16:12:40 +00:00

Client-side mux and mesh (D288) #251

kayg referenced this issue

2026-10-31 16:15:48 +00:00