Research: terminal images and safe image decoding (HEIC/HEIF)

kayg commented

2026-10-09 09:13:17 +00:00

Owner

Owner 2026-10-09: research images in the terminal (D93: kitty graphics protocol, ours, intercepted before ghostty-web) end to end, and how we keep image handling safe against malformed-image decoder vulnerabilities, specifically the recent HEIC/HEIF decoder issue the owner calls 'heifheist'. Research only: measured candidates and a recommendation; the owner decides.

kayg commented

2026-10-09 09:14:44 +00:00

Author

Owner

Starting research on branch research-images, based at 63b1f10e21.

Starting research on branch research-images, based at 63b1f10e2112b4d06f7059d4faf196251a58d6a8.

kayg commented

2026-10-09 09:21:43 +00:00

Author

Owner

Finding: upstream libghostty-vt has a Kitty graphics storage API behind a build feature, including image and placement access plus an embedder-supplied PNG decoder. This checkout's shim exposes only terminal feed/resize/get and text snapshot calls (crates/infraternal-shim/src/terminal_native.rs); the build adapter exports only snapshot helpers (scripts/build-vt), v1 has byte Output/Snapshot frames capped at 1 MiB (crates/infraternal-shim/src/frame.rs), and the browser currently paints the text canvas (apps/tracer/src/terminal.ts). Reusing engine parsing may avoid a duplicate parser, but the pinned build's feature/callback and image-state transport/layer are not wired. Sources: https://docs.rs/libghostty-vt/latest/libghostty_vt/kitty/graphics/index.html and https://github.com/ghostty-org/ghostty/issues/12111.

Finding: upstream libghostty-vt has a Kitty graphics storage API behind a build feature, including image and placement access plus an embedder-supplied PNG decoder. This checkout's shim exposes only terminal feed/resize/get and text snapshot calls (`crates/infraternal-shim/src/terminal_native.rs`); the build adapter exports only snapshot helpers (`scripts/build-vt`), v1 has byte `Output`/`Snapshot` frames capped at 1 MiB (`crates/infraternal-shim/src/frame.rs`), and the browser currently paints the text canvas (`apps/tracer/src/terminal.ts`). Reusing engine parsing may avoid a duplicate parser, but the pinned build's feature/callback and image-state transport/layer are not wired. Sources: https://docs.rs/libghostty-vt/latest/libghostty_vt/kitty/graphics/index.html and https://github.com/ghostty-org/ghostty/issues/12111.

kayg commented

2026-10-09 09:22:20 +00:00

Author

Owner

Step 1 committed as aff23518e6. research/terminal-images/NOTES.md records the local shim/protocol/render path and primary-source findings. Numbers: v1 frames cap at 1 MiB; D170 is 200 concurrent terminal sessions per server (issue #97 history). No product code changed.

Step 1 committed as aff23518e69e73fbbfcb8e2df58defa6590e938a. `research/terminal-images/NOTES.md` records the local shim/protocol/render path and primary-source findings. Numbers: v1 frames cap at 1 MiB; D170 is 200 concurrent terminal sessions per server (issue #97 history). No product code changed.

kayg commented

2026-10-09 09:24:31 +00:00

Author

Owner

Finding: the September 1, 2026 libheif advisory matching “heifheist” is GHSA-x8r2-mggj-j6wr (no CVE assigned in the maintainer advisory), Critical; libheif 1.18.0–1.23.2 is affected and 1.23.3 fixes it. The unci mixed-interleave path writes Cb16 samples into a Cr8 plane. The advisory's file-read/PHP execution demo is explicitly one build/deployment-specific WordPress pipeline, not universal RCE. Safari 17 added HEIC support and current WebKit source routes it through ImageIO; that is not evidence Apple's decoder is affected by this libheif GHSA. Sources: https://github.com/strukturag/libheif/security/advisories/GHSA-x8r2-mggj-j6wr ; https://developer.apple.com/documentation/safari-release-notes/safari-17-release-notes ; https://github.com/WebKit/WebKit/blob/main/Source/WebCore/platform/graphics/cg/ImageDecoderCG.cpp

Finding: the September 1, 2026 libheif advisory matching “heifheist” is GHSA-x8r2-mggj-j6wr (no CVE assigned in the maintainer advisory), Critical; libheif 1.18.0–1.23.2 is affected and 1.23.3 fixes it. The `unci` mixed-interleave path writes Cb16 samples into a Cr8 plane. The advisory's file-read/PHP execution demo is explicitly one build/deployment-specific WordPress pipeline, not universal RCE. Safari 17 added HEIC support and current WebKit source routes it through ImageIO; that is not evidence Apple's decoder is affected by this libheif GHSA. Sources: https://github.com/strukturag/libheif/security/advisories/GHSA-x8r2-mggj-j6wr ; https://developer.apple.com/documentation/safari-release-notes/safari-17-release-notes ; https://github.com/WebKit/WebKit/blob/main/Source/WebCore/platform/graphics/cg/ImageDecoderCG.cpp

kayg commented

2026-10-09 09:29:05 +00:00

Author

Owner

Pinned-build inspection corrects the earlier feature uncertainty: scripts/build-vt pins Ghostty a887df42c56f6de86c0fe6da9c4eeca37931e083, and its native build enables Kitty graphics. At that pin, the library default image-storage limit for .lib is 10 MiB per terminal; src/terminal/c/kitty_graphics.zig exposes stored image and placement queries. The shim does not export those queries or install GhosttySysOption.decode_png; src/terminal/c/sys.zig returns InvalidData for PNG when that callback is absent. Direct RGB/RGBA payloads are therefore the immediately viable image path; PNG and path/shared-memory transmission still need bridge/configuration work. Evidence is from the checksum-verified source archive and crates/infraternal-shim/src/terminal_native.rs.

Pinned-build inspection corrects the earlier feature uncertainty: scripts/build-vt pins Ghostty a887df42c56f6de86c0fe6da9c4eeca37931e083, and its native build enables Kitty graphics. At that pin, the library default image-storage limit for `.lib` is 10 MiB per terminal; `src/terminal/c/kitty_graphics.zig` exposes stored image and placement queries. The shim does not export those queries or install `GhosttySysOption.decode_png`; `src/terminal/c/sys.zig` returns InvalidData for PNG when that callback is absent. Direct RGB/RGBA payloads are therefore the immediately viable image path; PNG and path/shared-memory transmission still need bridge/configuration work. Evidence is from the checksum-verified source archive and `crates/infraternal-shim/src/terminal_native.rs`.

kayg commented

2026-10-09 09:50:10 +00:00

Author

Owner

Step: added the standalone Rust/browser benchmark harness. Commit 554f50db8eacbb3420a939e7ef893762ed93d983. It generates four fixtures (1 MP and 4 MP PNG, 208,422-byte 1 MP JPEG, and 24-frame GIF), measures decoder paths, transfer sizes, and a canvas layer; cargo check --locked passed under the required CPUWeight scope. No product dependencies changed.

Step: added the standalone Rust/browser benchmark harness. Commit `554f50db8eacbb3420a939e7ef893762ed93d983`. It generates four fixtures (1 MP and 4 MP PNG, 208,422-byte 1 MP JPEG, and 24-frame GIF), measures decoder paths, transfer sizes, and a canvas layer; `cargo check --locked` passed under the required CPUWeight scope. No product dependencies changed.

kayg commented

2026-10-09 09:59:08 +00:00

Author

Owner

Correction to my earlier storage-limit comment: the pinned Ghostty .lib 10,000,000-byte budget is per screen and counts fully decoded Image.data pixels, not encoded input. graphics_image.zig documents stored data as decoded RGB/RGBA; graphics_storage.zig accounts img.data.len; Terminal.zig sets the .lib limit to 10 * 1000 * 1000. A single 4 MP image is 12,582,912 B RGB or 16,777,216 B RGBA, so either exceeds the default screen limit. At D170's 200 sessions, storing one 4 MP RGBA image per screen requires at least 3.36 GB; two screens require 6.71 GB. Report and NOTES now use these corrected figures.

Correction to my earlier storage-limit comment: the pinned Ghostty `.lib` 10,000,000-byte budget is per screen and counts fully decoded `Image.data` pixels, not encoded input. `graphics_image.zig` documents stored data as decoded RGB/RGBA; `graphics_storage.zig` accounts `img.data.len`; `Terminal.zig` sets the `.lib` limit to `10 * 1000 * 1000`. A single 4 MP image is 12,582,912 B RGB or 16,777,216 B RGBA, so either exceeds the default screen limit. At D170's 200 sessions, storing one 4 MP RGBA image per screen requires at least 3.36 GB; two screens require 6.71 GB. Report and NOTES now use these corrected figures.

kayg commented

2026-10-09 10:03:16 +00:00

Author

Owner

Step: updated research/terminal-images/NOTES.md with pinned C API findings and measured costs. Commit 7cb912afe0e800f7d8632bbb5460ae775c8bb614. Evidence: 4 MP Rust PNG decode p50 34.942 ms at 1× / 137.975 ms at 4×; Chromium createImageBitmap 58.7/178.5 ms; binary PNG 5,695,718 B vs Kitty-base64 7,594,292 B. The per-screen Ghostty limit is decoded pixels: 10,000,000 B; one 4 MP RGBA image is 16,777,216 B and exceeds it.

Step: updated `research/terminal-images/NOTES.md` with pinned C API findings and measured costs. Commit `7cb912afe0e800f7d8632bbb5460ae775c8bb614`. Evidence: 4 MP Rust PNG decode p50 34.942 ms at 1× / 137.975 ms at 4×; Chromium `createImageBitmap` 58.7/178.5 ms; binary PNG 5,695,718 B vs Kitty-base64 7,594,292 B. The per-screen Ghostty limit is decoded pixels: 10,000,000 B; one 4 MP RGBA image is 16,777,216 B and exceeds it.

kayg commented

2026-10-09 10:04:13 +00:00

Author

Owner

Correction: fixed the RGBA Base64 transfer projections in research/terminal-images/NOTES.md. Commit adb9a57a5b7d0037329262f59145900d85e7771b. Base64 uses four-byte output groups, so raw RGBA totals are 5,592,408 B (1 MP) and 22,369,624 B (4 MP), including padding. Binary pixel sizes and all decode timings are unchanged.

Correction: fixed the RGBA Base64 transfer projections in `research/terminal-images/NOTES.md`. Commit `adb9a57a5b7d0037329262f59145900d85e7771b`. Base64 uses four-byte output groups, so raw RGBA totals are 5,592,408 B (1 MP) and 22,369,624 B (4 MP), including padding. Binary pixel sizes and all decode timings are unchanged.

kayg commented

2026-10-09 10:04:29 +00:00

Author

Owner

Step: published the owner report in research/terminal-images/REPORT.md. Commit c8624b58477dc1bc6f42b46f04ec44be78775691. It compares decoded-pixel and preserved-PNG routes, summarizes protocol and browser reachability, and includes the D170 memory/transfer projections and local decode/layer measurements. Decoder choice remains an owner grill question.

Step: published the owner report in `research/terminal-images/REPORT.md`. Commit `c8624b58477dc1bc6f42b46f04ec44be78775691`. It compares decoded-pixel and preserved-PNG routes, summarizes protocol and browser reachability, and includes the D170 memory/transfer projections and local decode/layer measurements. Decoder choice remains an owner grill question.

kayg commented

2026-10-09 10:15:23 +00:00

Author

Owner

Verification environment finding: CARGO_TARGET_DIR in scratch left the worktree-relative target/ path absent. agent_review::initial_attach_requires_explicit_credit_before_replay failed creating target/ac<PID> (scripts/verify-agent-attach-credit.py:17); static builds failed because .cargo/musl-cc.sh:22 resolves Zig through target/auth-tools/zig/zig. No tracked files changed. The rerun will link the ignored worktree target/ path into scratch and expose the installed Zig binary there.

Verification environment finding: `CARGO_TARGET_DIR` in scratch left the worktree-relative `target/` path absent. `agent_review::initial_attach_requires_explicit_credit_before_replay` failed creating `target/ac<PID>` (`scripts/verify-agent-attach-credit.py:17`); static builds failed because `.cargo/musl-cc.sh:22` resolves Zig through `target/auth-tools/zig/zig`. No tracked files changed. The rerun will link the ignored worktree `target/` path into scratch and expose the installed Zig binary there.

kayg commented

2026-10-09 10:45:51 +00:00

Author

Owner

Finished research-only #138. HEAD c8624b58477dc1bc6f42b46f04ec44be78775691; worktree clean. Deliverables are research/terminal-images/REPORT.md, NOTES.md, and the standalone bench/ harness. No product code changed.

scripts/verify fast exited 0. Verbatim gate output excerpts:

METRIC name=network_rtt_selftest configured_rtt_ms=300 jitter_ms=30 loss=unsupported transport=tcp-stream n=3 p50_ms=294.41 p99_ms=310.46 result=PASS

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+

PT_INTERP: absent; size: 15965392 bytes (baseline 16142672 + allowance 262144 = max 16404816)
PT_INTERP: absent; size: 13426064 bytes (baseline 13555672 + allowance 262144 = max 13817816)
advisories ok, bans ok, licenses ok, sources ok
dist/ghostty-web.js: OK
dist/ghostty-vt.wasm: OK
dist/index.d.ts: OK

 54 pass
 0 fail
 9930 expect() calls
Ran 54 tests across 12 files. [446.00ms]
JavaScript transfer, critical: identity=920146 gzip=282933/307200 brotli=234805/256000 bytes
JavaScript transfer, deferred: identity=138800 gzip=49880/51200 brotli=43657/45056 bytes
JavaScript transfer, onDemand: identity=53749 gzip=22161/20480 brotli=19812/18432 bytes

stream differential: 134 streams, 11439 schedules, zero divergences
 7 pass
 0 fail
 34390 expect() calls
Ran 7 tests across 2 files. [3.20s]

Finished research-only #138. HEAD `c8624b58477dc1bc6f42b46f04ec44be78775691`; worktree clean. Deliverables are `research/terminal-images/REPORT.md`, `NOTES.md`, and the standalone `bench/` harness. No product code changed. `scripts/verify fast` exited 0. Verbatim gate output excerpts: ```text METRIC name=network_rtt_selftest configured_rtt_ms=300 jitter_ms=30 loss=unsupported transport=tcp-stream n=3 p50_ms=294.41 p99_ms=310.46 result=PASS +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | PASS | +---------------------------+--------------+ PT_INTERP: absent; size: 15965392 bytes (baseline 16142672 + allowance 262144 = max 16404816) PT_INTERP: absent; size: 13426064 bytes (baseline 13555672 + allowance 262144 = max 13817816) advisories ok, bans ok, licenses ok, sources ok dist/ghostty-web.js: OK dist/ghostty-vt.wasm: OK dist/index.d.ts: OK 54 pass 0 fail 9930 expect() calls Ran 54 tests across 12 files. [446.00ms] JavaScript transfer, critical: identity=920146 gzip=282933/307200 brotli=234805/256000 bytes JavaScript transfer, deferred: identity=138800 gzip=49880/51200 brotli=43657/45056 bytes JavaScript transfer, onDemand: identity=53749 gzip=22161/20480 brotli=19812/18432 bytes stream differential: 134 streams, 11439 schedules, zero divergences 7 pass 0 fail 34390 expect() calls Ran 7 tests across 2 files. [3.20s] ```