VNC window kind: libraries vs our own RFB client, measured

kayg commented

2026-10-15 09:08:14 +00:00

Owner

Owner 2026-10-15: VNC becomes a first-class window kind, not only a fallback (extends D104, D200, D221). Research: Rust RFB libraries vs our own implementation, and the pipeline choice. Measured candidates go to the owner (CLAUDE.md foundational choices).

pipeline-b-frame.png

13 KiB

pipeline-b-run.log

4.1 KiB

verify-fast.log

229 KiB

verify-fast-absolute.log

278 KiB

verify-fast-absolute.log

278 KiB

baseline-check.txt

20 KiB

typescript-check.txt

29 KiB

kayg referenced this issue from a commit

2026-10-15 09:08:48 +00:00

decisions: D282 VNC is a general window kind, not only a fallback

kayg commented

2026-10-15 09:09:22 +00:00

Author

Owner

Started research-vnc on branch research-vnc at base/head e972f91dad. Read #223 (no comments) and the related VNC research issue #23 (all eight comments), plus CLAUDE.md, DECISIONS.md, docs/ARCHITECTURE.md and docs/PROCESS.md. Continuing #23's earlier browser-only decode findings, which are not pipeline or target evidence. No library or architecture selected; owner grill remains the outcome.

Started research-vnc on branch research-vnc at base/head e972f91dad67ac1f1274a4e4cd70df24ee052bc1. Read #223 (no comments) and the related VNC research issue #23 (all eight comments), plus CLAUDE.md, DECISIONS.md, docs/ARCHITECTURE.md and docs/PROCESS.md. Continuing #23's earlier browser-only decode findings, which are not pipeline or target evidence. No library or architecture selected; owner grill remains the outcome.

kayg commented

2026-10-15 09:13:41 +00:00

Author

Owner

Pipeline reuse finding: crates/plugins/desktop/src/media.rs, session.rs and apps/tracer/src/canvas/appwin/decode.worker.on-demand.ts show the app-window session carries bounded H.264/HEVC access units, caches keyframes for join/resume, and decodes with WebCodecs in a worker. Its measured encoder implementation is crates/plugins/desktop/native/Encoder.swift (VideoToolbox). QEMU's ui/vnc.h lists rectangle encoders and its RFB path has no H.264 encoding, so QEMU would require an added RFB decode + video encode step to reuse that browser path. D88 rejects a WASM bridge for client logic; an RFB protocol engine is client logic, so the browser-decoder candidate needs a TypeScript path unless the owner changes D88. No candidate or architecture selected.

Pipeline reuse finding: `crates/plugins/desktop/src/media.rs`, `session.rs` and `apps/tracer/src/canvas/appwin/decode.worker.on-demand.ts` show the app-window session carries bounded H.264/HEVC access units, caches keyframes for join/resume, and decodes with WebCodecs in a worker. Its measured encoder implementation is `crates/plugins/desktop/native/Encoder.swift` (VideoToolbox). QEMU's `ui/vnc.h` lists rectangle encoders and its RFB path has no H.264 encoding, so QEMU would require an added RFB decode + video encode step to reuse that browser path. D88 rejects a WASM bridge for client logic; an RFB protocol engine is client logic, so the browser-decoder candidate needs a TypeScript path unless the owner changes D88. No candidate or architecture selected.

kayg referenced this issue from a commit

2026-10-15 09:14:33 +00:00

decisions: D282 Proxmox VMs open as VNC windows with zero configuration via the agent on the node

kayg commented

2026-10-15 09:15:25 +00:00

Author

Owner

Continuing #223 on branch research-vnc; base SHA e972f91dad67ac1f1274a4e4cd70df24ee052bc1 (clean worktree). Read all #223 and parent #23 comments. This pass covers the owner’s Proxmox-node correction, current Rust client libraries and macOS security types, reuse against the current app-window implementation, and real-target pipeline measurements. No library or pipeline selected.

Continuing #223 on branch `research-vnc`; base SHA `e972f91dad67ac1f1274a4e4cd70df24ee052bc1` (clean worktree). Read all #223 and parent #23 comments. This pass covers the owner’s Proxmox-node correction, current Rust client libraries and macOS security types, reuse against the current app-window implementation, and real-target pipeline measurements. No library or pipeline selected.

kayg commented

2026-10-15 09:23:06 +00:00

Author

Owner

Fast-forwarded research-vnc from e972f91 to origin/dev 67010de (D282 only). Rechecked origin/app-windows-ship: crates/plugins/desktop/src/media.rs and session.rs hold immutable H.264/HEVC access units, retain a keyframe for joins, and bound viewers at 5; apps/tracer/src/canvas/appwin/decode.worker.on-demand.ts decodes with WebCodecs in a worker. The source encoder is Swift VideoToolbox, so that implementation is Mac-only; Linux VNC would need a separate encoder. Perf VM preflight: QEMU and KVM available; initial load average 1.08/1.43/0.82. No benchmark VM started yet.

Fast-forwarded `research-vnc` from `e972f91` to `origin/dev` `67010de` (D282 only). Rechecked `origin/app-windows-ship`: `crates/plugins/desktop/src/media.rs` and `session.rs` hold immutable H.264/HEVC access units, retain a keyframe for joins, and bound viewers at 5; `apps/tracer/src/canvas/appwin/decode.worker.on-demand.ts` decodes with WebCodecs in a worker. The source encoder is Swift VideoToolbox, so that implementation is Mac-only; Linux VNC would need a separate encoder. Perf VM preflight: QEMU and KVM available; initial load average 1.08/1.43/0.82. No benchmark VM started yet.

kayg commented

2026-10-15 10:22:15 +00:00

Author

Owner

Source review (2026-10-15): Proxmox documents pvesh as a root-only local API shell; pvecm documents cluster-wide pmxcfs configuration and multi-master nodes. qm vncproxy source requires LC_PVE_TICKET, sets it as QEMU's temporary VNC password and expires it after 30 seconds; Helpers::vnc_socket resolves /var/run/qemu-server/<vmid>.vnc. Recommendation for the agent path: discover with pvesh, use Proxmox's qm vncproxy process pipe with a short-lived in-memory ticket, and keep one upstream RFB session per VM for viewer fan-out. Remote-node proxy startup and web-console concurrency still need a real PVE cluster check.

Sources: https://github.com/proxmox/pve-docs/blob/master/pvesh.adoc ; https://github.com/proxmox/pve-docs/blob/master/pvecm.adoc ; https://github.com/proxmox/qemu-server/blob/master/src/PVE/CLI/qm.pm ; https://github.com/proxmox/qemu-server/blob/master/src/PVE/QemuServer/Helpers.pm ; https://github.com/proxmox/qemu-server/blob/master/debian/changelog

Source review (2026-10-15): Proxmox documents `pvesh` as a root-only local API shell; `pvecm` documents cluster-wide pmxcfs configuration and multi-master nodes. `qm vncproxy` source requires `LC_PVE_TICKET`, sets it as QEMU's temporary VNC password and expires it after 30 seconds; `Helpers::vnc_socket` resolves `/var/run/qemu-server/<vmid>.vnc`. Recommendation for the agent path: discover with `pvesh`, use Proxmox's `qm vncproxy` process pipe with a short-lived in-memory ticket, and keep one upstream RFB session per VM for viewer fan-out. Remote-node proxy startup and web-console concurrency still need a real PVE cluster check. Sources: https://github.com/proxmox/pve-docs/blob/master/pvesh.adoc ; https://github.com/proxmox/pve-docs/blob/master/pvecm.adoc ; https://github.com/proxmox/qemu-server/blob/master/src/PVE/CLI/qm.pm ; https://github.com/proxmox/qemu-server/blob/master/src/PVE/QemuServer/Helpers.pm ; https://github.com/proxmox/qemu-server/blob/master/debian/changelog

kayg commented

2026-10-15 10:29:09 +00:00

Author

Owner

Additional Rust candidate found in the current source search: DeskVNC workspace version 0.27.2 is MIT OR Apache-2.0, pre-1.0, maintained by one engineer. Its internal vnc-core advertises Tokio, RFB 3.3–3.8, QEMU key/LED/pointer extensions, ContinuousUpdates, VeNCrypt/X.509, RA2 and Apple Diffie-Hellman type 30; the project architecture declares forbid(unsafe_code) on network-facing crates. It claims interoperability with QEMU and macOS Screen Sharing but gives no macOS version matrix. The crate is not independently published, and no isolated binary-size result was found. This is the strongest Rust source candidate to evaluate, not a selection.

Sources: https://github.com/psmux/DeskVNC ; https://github.com/psmux/DeskVNC/blob/main/docs/ARCHITECTURE.md ; https://github.com/psmux/DeskVNC/blob/main/Cargo.toml

Additional Rust candidate found in the current source search: DeskVNC workspace version 0.27.2 is MIT OR Apache-2.0, pre-1.0, maintained by one engineer. Its internal `vnc-core` advertises Tokio, RFB 3.3–3.8, QEMU key/LED/pointer extensions, ContinuousUpdates, VeNCrypt/X.509, RA2 and Apple Diffie-Hellman type 30; the project architecture declares `forbid(unsafe_code)` on network-facing crates. It claims interoperability with QEMU and macOS Screen Sharing but gives no macOS version matrix. The crate is not independently published, and no isolated binary-size result was found. This is the strongest Rust source candidate to evaluate, not a selection. Sources: https://github.com/psmux/DeskVNC ; https://github.com/psmux/DeskVNC/blob/main/docs/ARCHITECTURE.md ; https://github.com/psmux/DeskVNC/blob/main/Cargo.toml

kayg commented

2026-10-15 10:39:40 +00:00

Author

Owner

Proxmox source detail: qm vncproxy asserts the VM config exists on the current node before opening that node's /var/run/qemu-server/<vmid>.vnc socket. A root agent must run the process pipe on the VM's owner node; for other cluster nodes it needs the node API/WebSocket path or Proxmox's root SSH route. The report now records both paths and marks remote-node behavior for a real-cluster functional check.

Proxmox source detail: `qm vncproxy` asserts the VM config exists on the current node before opening that node's `/var/run/qemu-server/<vmid>.vnc` socket. A root agent must run the process pipe on the VM's owner node; for other cluster nodes it needs the node API/WebSocket path or Proxmox's root SSH route. The report now records both paths and marks remote-node behavior for a real-cluster functional check.

kayg commented

2026-10-15 11:01:10 +00:00

Author

Owner

Measurement harness and QEMU guest are ready; this job has been queued behind the host-wide build slot for about 48 minutes. No perf VM lock is held by this job. Waiting for the orchestrator slot to run the requested target measurements, then for an available perf lock.

kayg commented

2026-10-15 11:11:26 +00:00

Author

Owner

Perf VM run complete: QEMU 10.0.13, Alpine 3.24.2, 1280x800, 4 vCPU, Chromium Headless Shell 153.0.8010.12. One-viewer idle (8 s): 0 RFB bytes, QEMU 6.5% CPU, Chromium aggregate 13.25%, RSS 750.45/551.27 MiB. Five-viewer idle (8 s): 0 RFB bytes, QEMU 6.625%, Chromium aggregate 38.5%, RSS 759.29/1,304.13 MiB; page heaps 10.0–17.1 MB. First-pixel join: 817.08 ms for the initial viewer, 748.55–896.93 ms for four additional viewers. Full-frame join and input-to-pixel p50/p99 remain unmeasured. The 755 B nominal scroll interval coincided with delayed terminal typing; typing/scroll/video rates are excluded. Pipelines A and C remain unmeasured. Raw instrumentation:

Perf VM run complete: QEMU 10.0.13, Alpine 3.24.2, 1280x800, 4 vCPU, Chromium Headless Shell 153.0.8010.12. One-viewer idle (8 s): 0 RFB bytes, QEMU 6.5% CPU, Chromium aggregate 13.25%, RSS 750.45/551.27 MiB. Five-viewer idle (8 s): 0 RFB bytes, QEMU 6.625%, Chromium aggregate 38.5%, RSS 759.29/1,304.13 MiB; page heaps 10.0–17.1 MB. First-pixel join: 817.08 ms for the initial viewer, 748.55–896.93 ms for four additional viewers. Full-frame join and input-to-pixel p50/p99 remain unmeasured. The 755 B nominal scroll interval coincided with delayed terminal typing; typing/scroll/video rates are excluded. Pipelines A and C remain unmeasured. Raw instrumentation: ![pipeline-b-run.log](https://git.kayg.org/attachments/78fea5d5-75ba-43d2-9206-f31d257d3047)

kayg commented

2026-10-15 11:12:29 +00:00

Author

Owner

Research report committed at 2ce4930d. The committed report includes the Rust candidate survey, client-size scope, Proxmox/macOS findings, and valid perf-VM idle observations: 0 RFB bytes/8 s for one and five viewers; QEMU CPU 6.5%/6.625%; Chromium aggregate CPU 13.25%/38.5%; first-pixel joins 817.08 ms and 748.55–896.93 ms. Input p50/p99, full-frame join, active workloads, and pipelines A/C remain unmeasured. Final scripts/verify fast is queued through the host build slot.

Research report committed at 2ce4930d. The committed report includes the Rust candidate survey, client-size scope, Proxmox/macOS findings, and valid perf-VM idle observations: 0 RFB bytes/8 s for one and five viewers; QEMU CPU 6.5%/6.625%; Chromium aggregate CPU 13.25%/38.5%; first-pixel joins 817.08 ms and 748.55–896.93 ms. Input p50/p99, full-frame join, active workloads, and pipelines A/C remain unmeasured. Final `scripts/verify fast` is queued through the host build slot.

kayg commented

2026-10-15 12:12:55 +00:00

Author

Owner

At 2ce4930d, the first scripts/verify fast failed four mesh node_state tests at Directory::new with ENOENT; its static build also resolved Zig under another job's missing target path. The test helper uses CARGO_TARGET_DIR as its temporary-directory parent. One focused agent-feature test passed with CARGO_TARGET_DIR pinned to this worktree's absolute target path. I have queued one final-gate rerun with that same absolute path so tests and the static compiler use this worktree's toolchain. First transcript:

At 2ce4930d, the first `scripts/verify fast` failed four mesh `node_state` tests at `Directory::new` with `ENOENT`; its static build also resolved Zig under another job's missing target path. The test helper uses `CARGO_TARGET_DIR` as its temporary-directory parent. One focused agent-feature test passed with `CARGO_TARGET_DIR` pinned to this worktree's absolute `target` path. I have queued one final-gate rerun with that same absolute path so tests and the static compiler use this worktree's toolchain. First transcript: ![verify-fast.log](https://git.kayg.org/attachments/835502a5-645e-4046-bfeb-6c6807b4ac55)

kayg commented

2026-10-15 13:10:40 +00:00

Author

Owner

The corrected gate measured the stripped static x86_64 binary at 22,277,392 bytes against the repository cap of 22,204,304 bytes (+73,088 bytes over cap). git diff 67010de9199eb5e7b2ac2d3540b5fcc902e51430..2ce4930ddf4494631ae33b689140ff7a7795e51b contains only research/vnc/REPORT.md, so executable-source/config/dependency delta against the merge base is 0 bytes. The size overage is therefore present in the merge-base source set; the gate is continuing through the aarch64 static build.

The corrected gate measured the stripped static x86_64 binary at 22,277,392 bytes against the repository cap of 22,204,304 bytes (+73,088 bytes over cap). `git diff 67010de9199eb5e7b2ac2d3540b5fcc902e51430..2ce4930ddf4494631ae33b689140ff7a7795e51b` contains only `research/vnc/REPORT.md`, so executable-source/config/dependency delta against the merge base is 0 bytes. The size overage is therefore present in the merge-base source set; the gate is continuing through the aarch64 static build.

kayg commented

2026-10-15 13:24:40 +00:00

Author

Owner

Finished at 2ce4930ddf4494631ae33b689140ff7a7795e51b on research-vnc. The report is research/vnc/REPORT.md. The pipeline B QEMU/RFB browser-decoder run measured initial first-pixel at 817.08 ms and four later joins at 748.55, 766.31, 896.93, and 772.20 ms; it did not establish full-frame join or valid input-to-pixel latency. The report recommends starting with browser RFB decode for the Proxmox local-agent path, then measuring any server re-encode or passthrough path before choosing it. No Mac was used. Raw final gate log: verify-fast-absolute.log.

The report-only commit adds 0 executable bytes against merge base 67010de9199e. The final gate's static binaries exceed the existing caps by 73,088 bytes (x86_64) and 66,368 bytes (aarch64); both full byte counts are below. Final scripts/verify fast summary, verbatim:

PT_INTERP: absent; size: 19705112 bytes; cap: 19638744 bytes; delta: +66368 bytes
binary exceeds the 19638744-byte size budget: 19705112 bytes
key memory: aarch64-unknown-linux-musl runtime not checked (host x86_64; native execution required)

(static builds: 1618s)

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Finished at `2ce4930ddf4494631ae33b689140ff7a7795e51b` on `research-vnc`. The report is `research/vnc/REPORT.md`. The pipeline B QEMU/RFB browser-decoder run measured initial first-pixel at 817.08 ms and four later joins at 748.55, 766.31, 896.93, and 772.20 ms; it did not establish full-frame join or valid input-to-pixel latency. The report recommends starting with browser RFB decode for the Proxmox local-agent path, then measuring any server re-encode or passthrough path before choosing it. No Mac was used. Raw final gate log: [verify-fast-absolute.log](https://git.kayg.org/attachments/8f66aac5-fe3d-4395-aa09-4406502f4071). The report-only commit adds 0 executable bytes against merge base `67010de9199e`. The final gate's static binaries exceed the existing caps by 73,088 bytes (x86_64) and 66,368 bytes (aarch64); both full byte counts are below. Final `scripts/verify fast` summary, verbatim: ``` PT_INTERP: absent; size: 19705112 bytes; cap: 19638744 bytes; delta: +66368 bytes binary exceeds the 19638744-byte size budget: 19705112 bytes key memory: aarch64-unknown-linux-musl runtime not checked (host x86_64; native execution required) (static builds: 1618s) +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ```

kayg referenced this issue from a commit

2026-10-15 13:25:54 +00:00

research: document VNC client options and Proxmox path

kayg referenced this issue from a commit

2026-10-15 16:27:04 +00:00

decisions: D282 write our own VNC client from scratch, with Screens 5's seamlessness as the bar

kayg commented

2026-10-15 16:28:14 +00:00

Author

Owner

Started design lane on branch vnc-design at base SHA 88332a4850d9f78b922f878fdd4770c9dff4f2f5. Read all 13 comments on #223 and all 8 comments on parent research issue #23. Carrying forward the QEMU measurements and their limitations; D282 requires our own RFB client and leaves pipeline decisions to measured candidates.

Started design lane on branch `vnc-design` at base SHA `88332a4850d9f78b922f878fdd4770c9dff4f2f5`. Read all 13 comments on #223 and all 8 comments on parent research issue #23. Carrying forward the QEMU measurements and their limitations; D282 requires our own RFB client and leaves pipeline decisions to measured candidates.

kayg commented

2026-10-15 16:32:36 +00:00

Author

Owner

Research findings for #223:

Screens 5 documents Touch and Trackpad cursor modes, adaptive quality, display selection, shared clipboard, Mac file transfer, per-connection behavior, and interactive toolbars. Curtain Mode requires Remote Management and explicitly does not work with macOS Screen Sharing: https://help.edovia.com/en-GB/screens-5/features/cursor-control-modes-and-other-gestures ; https://help.edovia.com/en-GB/screens-5/features/images-quality ; https://help.edovia.com/en-GB/screens-5/features/display-selection ; https://help.edovia.com/en-GB/screens-5/features/clipboard ; https://help.edovia.com/en-GB/screens-5/features/file-transfers-mac ; https://help.edovia.com/en-GB/screens-5/features/curtain-mode
Keyboard reports: Proxmox users report German/French layout and dead-key failures, including swapped Z/Y, missing umlauts and dead keys producing the following base character; a 2025 iPadOS report describes virtual-keyboard special characters failing. https://forum.proxmox.com/threads/novnc-keymap-issues.46804/ ; https://forum.proxmox.com/threads/once-more-keyboard-layout-in-novnc.24315/ ; https://forum.proxmox.com/threads/novnc-dt-umlaute-und-weitere-zeichen-verschwinden-auf-ipados.176917/ ; current noVNC issue #2071 reports Android Gboard mapping @ to 2: https://github.com/novnc/noVNC/issues/2071
Input/UI reports: noVNC #1757 reports the mobile keyboard control disappearing; #856 reports Android virtual-keyboard letters/symbols not being sent. Proxmox #120040 describes lack of clipboard copy/paste in the console. https://github.com/novnc/noVNC/issues/1757 ; https://github.com/novnc/noVNC/issues/856 ; https://forum.proxmox.com/threads/use-copy-paste-in-novnc-or-other.120040/
Rendering and session reports: noVNC #1469 records blurry local scaling that was crisp at 1:1; Proxmox #125321 reports console refresh/reconnect every 30 seconds when an intermediary closes idle WebSockets; Proxmox #53320 describes a console freeze that required reopening. https://github.com/novnc/noVNC/issues/1469 ; https://forum.proxmox.com/threads/gui-shell-and-console-concerns.125321/ ; https://forum.proxmox.com/threads/console-hangs-within-a-few-minutes.53320/
Recent noVNC reports include Chromium Image Cache/RAM growth (#2075) and an H.264 error path that could stall the framebuffer request loop (#2074). These are concrete reasons to cap hidden surfaces/queues and test recovery paths. https://github.com/novnc/noVNC/issues/2075 ; https://github.com/novnc/noVNC/pull/2074
Apple security types: IANA assigns 30–36 to Apple; a LibVNCServer engineering report says its author tested 33, 35 and 36, identifies 35 as Kerberos-based, and defers 34. This is compatibility evidence rather than an Apple wire specification. https://www.iana.org/assignments/rfb ; https://github.com/LibVNC/libvncserver/issues/696

No library or RFB pipeline was selected by the research; D282 already selects an in-house client. Existing issue measurements remain the only target numbers: QEMU with one and five viewers produced 0 RFB bytes in 8 seconds idle; initial first-pixel join was 817.08 ms and later joins were 748.55–896.93 ms. Full-frame join and input-to-pixel latency remain unmeasured.

Research findings for #223: - Screens 5 documents Touch and Trackpad cursor modes, adaptive quality, display selection, shared clipboard, Mac file transfer, per-connection behavior, and interactive toolbars. Curtain Mode requires Remote Management and explicitly does not work with macOS Screen Sharing: https://help.edovia.com/en-GB/screens-5/features/cursor-control-modes-and-other-gestures ; https://help.edovia.com/en-GB/screens-5/features/images-quality ; https://help.edovia.com/en-GB/screens-5/features/display-selection ; https://help.edovia.com/en-GB/screens-5/features/clipboard ; https://help.edovia.com/en-GB/screens-5/features/file-transfers-mac ; https://help.edovia.com/en-GB/screens-5/features/curtain-mode - Keyboard reports: Proxmox users report German/French layout and dead-key failures, including swapped Z/Y, missing umlauts and dead keys producing the following base character; a 2025 iPadOS report describes virtual-keyboard special characters failing. https://forum.proxmox.com/threads/novnc-keymap-issues.46804/ ; https://forum.proxmox.com/threads/once-more-keyboard-layout-in-novnc.24315/ ; https://forum.proxmox.com/threads/novnc-dt-umlaute-und-weitere-zeichen-verschwinden-auf-ipados.176917/ ; current noVNC issue #2071 reports Android Gboard mapping @ to 2: https://github.com/novnc/noVNC/issues/2071 - Input/UI reports: noVNC #1757 reports the mobile keyboard control disappearing; #856 reports Android virtual-keyboard letters/symbols not being sent. Proxmox #120040 describes lack of clipboard copy/paste in the console. https://github.com/novnc/noVNC/issues/1757 ; https://github.com/novnc/noVNC/issues/856 ; https://forum.proxmox.com/threads/use-copy-paste-in-novnc-or-other.120040/ - Rendering and session reports: noVNC #1469 records blurry local scaling that was crisp at 1:1; Proxmox #125321 reports console refresh/reconnect every 30 seconds when an intermediary closes idle WebSockets; Proxmox #53320 describes a console freeze that required reopening. https://github.com/novnc/noVNC/issues/1469 ; https://forum.proxmox.com/threads/gui-shell-and-console-concerns.125321/ ; https://forum.proxmox.com/threads/console-hangs-within-a-few-minutes.53320/ - Recent noVNC reports include Chromium Image Cache/RAM growth (#2075) and an H.264 error path that could stall the framebuffer request loop (#2074). These are concrete reasons to cap hidden surfaces/queues and test recovery paths. https://github.com/novnc/noVNC/issues/2075 ; https://github.com/novnc/noVNC/pull/2074 - Apple security types: IANA assigns 30–36 to Apple; a LibVNCServer engineering report says its author tested 33, 35 and 36, identifies 35 as Kerberos-based, and defers 34. This is compatibility evidence rather than an Apple wire specification. https://www.iana.org/assignments/rfb ; https://github.com/LibVNC/libvncserver/issues/696 No library or RFB pipeline was selected by the research; D282 already selects an in-house client. Existing issue measurements remain the only target numbers: QEMU with one and five viewers produced 0 RFB bytes in 8 seconds idle; initial first-pixel join was 817.08 ms and later joins were 748.55–896.93 ms. Full-frame join and input-to-pixel latency remain unmeasured.

kayg commented

2026-10-15 16:35:31 +00:00

Author

Owner

Architecture and corpus findings for #223:

Reviewed origin/app-windows-ship at bdb0ffc509c2c0f406b5c5467a522d56be04892f. crates/plugins/desktop/src/session.rs keeps a shared latest frame, caps the session at five viewers, uses one controller lease, and releases held input on handover/drop. apps/tracer/src/canvas/appwin/stream.ts detaches hidden viewers and the on-demand worker decodes off the UI thread. This is the lifecycle/fan-out pattern to reuse; it is not an RFB implementation.
D88 requires native TypeScript and Rust codecs, shared golden vectors, differential tests, and no WASM bridge. RFB image encodings therefore need mirrored bounded decoders: Rust applies each rectangle to the retained framebuffer; TypeScript applies the same ordered update in a worker and paints the canvas. A viewer sequence gap triggers a fresh full-frame snapshot, never a guessed partial restore.
The community RFB reference describes a single framebuffer with optional screen viewports and optional extensions. QEMU's current source exposes extended clipboard format flags including files; that capability is server-specific, not a general promise. References: https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst ; https://github.com/qemu/qemu/blob/master/ui/vnc.h
Public search located an RFB recording utility (rfbproxy) but no sanitized recorded QEMU or macOS session file suitable as a public fixture. Plan: record a deterministic QEMU test pattern in the perf-VM harness; use synthetic macOS handshake/update cases until a public sample exists. https://github.com/pmdumuid/rfbproxy

Design consequence: maintain one upstream connection and canonical framebuffer per target; reuse the app-window fan-out/control lifecycle, while bounding per-viewer queues and repairing missed RFB deltas with a snapshot.

Architecture and corpus findings for #223: - Reviewed `origin/app-windows-ship` at `bdb0ffc509c2c0f406b5c5467a522d56be04892f`. `crates/plugins/desktop/src/session.rs` keeps a shared latest frame, caps the session at five viewers, uses one controller lease, and releases held input on handover/drop. `apps/tracer/src/canvas/appwin/stream.ts` detaches hidden viewers and the on-demand worker decodes off the UI thread. This is the lifecycle/fan-out pattern to reuse; it is not an RFB implementation. - D88 requires native TypeScript and Rust codecs, shared golden vectors, differential tests, and no WASM bridge. RFB image encodings therefore need mirrored bounded decoders: Rust applies each rectangle to the retained framebuffer; TypeScript applies the same ordered update in a worker and paints the canvas. A viewer sequence gap triggers a fresh full-frame snapshot, never a guessed partial restore. - The community RFB reference describes a single framebuffer with optional screen viewports and optional extensions. QEMU's current source exposes extended clipboard format flags including files; that capability is server-specific, not a general promise. References: https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst ; https://github.com/qemu/qemu/blob/master/ui/vnc.h - Public search located an RFB recording utility (`rfbproxy`) but no sanitized recorded QEMU or macOS session file suitable as a public fixture. Plan: record a deterministic QEMU test pattern in the perf-VM harness; use synthetic macOS handshake/update cases until a public sample exists. https://github.com/pmdumuid/rfbproxy Design consequence: maintain one upstream connection and canonical framebuffer per target; reuse the app-window fan-out/control lifecycle, while bounding per-viewer queues and repairing missed RFB deltas with a snapshot.

kayg commented

2026-10-15 16:41:37 +00:00

Author

Owner

Design doc committed: 013037b7. research/vnc/DESIGN.md has 199 lines covering the Screens 5 feature map, noVNC/Proxmox reports and fixes, Rust session/TypeScript canvas architecture, encoding and security order, budgets, sample corpus and eight build slices. Existing #223 perf-VM baseline cited: QEMU 10.0.13 at 1280x800; one/five-viewer idle traffic was 0 RFB bytes over 8 s; first-pixel join was 817.08 ms for the first viewer and 748.55–896.93 ms for later viewers. Final gate and branch/base static-size comparison pending.

Design doc committed: 013037b7. `research/vnc/DESIGN.md` has 199 lines covering the Screens 5 feature map, noVNC/Proxmox reports and fixes, Rust session/TypeScript canvas architecture, encoding and security order, budgets, sample corpus and eight build slices. Existing #223 perf-VM baseline cited: QEMU 10.0.13 at 1280x800; one/five-viewer idle traffic was 0 RFB bytes over 8 s; first-pixel join was 817.08 ms for the first viewer and 748.55–896.93 ms for later viewers. Final gate and branch/base static-size comparison pending.

kayg commented

2026-10-15 18:59:22 +00:00

Author

Owner

013037b7 final scripts/verify fast finding: auth::tests::socket_tests::review_liveness::review_mesh_watch_reports_liveness_expiry_without_another_edit failed; all other reported Rust tests passed. The static builds passed trust checks but exceeded current D243 caps: x86_64 22,329,104 bytes vs 22,204,304 (+124,800); aarch64 19,752,920 bytes vs 19,638,744 (+114,176). This commit changes only research/vnc/DESIGN.md; merge-base comparison is pending to confirm source and binary deltas. Baseline liveness test is pending in the same load window.

`013037b7` final `scripts/verify fast` finding: `auth::tests::socket_tests::review_liveness::review_mesh_watch_reports_liveness_expiry_without_another_edit` failed; all other reported Rust tests passed. The static builds passed trust checks but exceeded current D243 caps: x86_64 22,329,104 bytes vs 22,204,304 (+124,800); aarch64 19,752,920 bytes vs 19,638,744 (+114,176). This commit changes only `research/vnc/DESIGN.md`; merge-base comparison is pending to confirm source and binary deltas. Baseline liveness test is pending in the same load window.

kayg referenced this issue from a commit

2026-10-15 19:08:49 +00:00

docs(vnc): design first-party remote desktop window (#223)

kayg commented

2026-10-15 19:16:08 +00:00

Author

Owner

Baseline check status: branch 013037b7, merge base 88332a4850d9f78b922f878fdd4770c9dff4f2f5. The isolated auth liveness reruns and same-recipe x86_64/aarch64 size builds remain queued behind the host-wide build-slot cap (about 16 minutes as of 2026-10-15 19:15 UTC). The check is still queued; no code change is needed. I will record both results after a slot opens.

Baseline check status: branch `013037b7`, merge base `88332a4850d9f78b922f878fdd4770c9dff4f2f5`. The isolated auth liveness reruns and same-recipe x86_64/aarch64 size builds remain queued behind the host-wide build-slot cap (about 16 minutes as of 2026-10-15 19:15 UTC). The check is still queued; no code change is needed. I will record both results after a slot opens.

kayg commented

2026-10-15 19:45:12 +00:00

Author

Owner

Baseline confirmation remains queued after 45 minutes in the host-wide build-slot wrapper (2026-10-15 19:45 UTC). The command is still active and has not started; it will run the merge-base liveness test and static size builds when capacity opens. No change is needed from the owner.

kayg commented

2026-10-15 21:15:41 +00:00

Author

Owner

Finished design-only lane. Branch vnc-design, base 88332a4850d9f78b922f878fdd4770c9dff4f2f5, head 013037b74895e2f4177668517cf38a6e6b8e8331 (pushed to origin/vnc-design). research/vnc/DESIGN.md is 199 lines and covers the feature map, reported console annoyances and responses, Rust/TypeScript architecture, codec and security ordering, budgets, conformance corpus, staged implementation plan, and owner questions. No product code changed.

Evidence:

Merge-base comparison: baseline-check.txt
Separate TypeScript half: typescript-check.txt
The full workspace run's review_mesh_watch_reports_liveness_expiry_without_another_edit test failed under parallel load. In the same busy window the isolated test passed at merge base (load 5.02/4.29/6.09; 10.90 s) and branch (7.98/7.81/7.04; 11.62 s). The complete diff is only research/vnc/DESIGN.md.
Same-recipe static sizes at merge base and branch were identical: x86_64 22,329,104 bytes (0-byte delta; +124,800 over the 22,204,304 cap); aarch64 19,752,920 bytes (0-byte delta; +114,176 over the 19,638,744 cap). Both existing cap overruns remain unresolved; this doc adds no executable bytes.
Separate TypeScript half exited 0: bun x tsc --project tsconfig.json, tracer/browser bundle and transfer checks, and 37 protocol tests passed; the stream comparison covered 134 streams and 11,439 schedules with zero divergences.

Verbatim final scripts/verify fast Rust summary and static-size output:

PT_INTERP: absent; size: 22329104 bytes; cap: 22204304 bytes; delta: +124800 bytes
binary exceeds the 22204304-byte size budget: 22329104 bytes
size: 19752920 bytes; cap: 19638744 bytes; delta: +114176 bytes

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| test                      | FAIL (101)   |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

The full gate stopped before its integrated TypeScript phase at the Rust test and static-size failures; the separate TypeScript-half run above passed. Worktree is clean at the pushed head.

Open owner questions in the design:

Should the mirrored Rust/TypeScript decoder be the first measured candidate, or must the prototype also compare server-decoded pixel updates first?
With shared upstream quality and resolution, should the controller own both, or should the holder aggregate visible viewports and route quality?
Which macOS versions and Apple security types (30, 33, 35, 36) are first-wave requirements?
Are QEMU/Proxmox and macOS Screen Sharing the first compatibility gate, or must a named Linux VNC server be included?
Is text-only clipboard acceptable for the first usable slice, with richer formats and file transfer capability-gated later?
Confirm the proposed ≤128 KiB gzip deferred browser chunk and local p95/p99 latency targets before making them hard gates?
Is synthetic protocol coverage enough for the first decoder review until a sanitized macOS fixture is available?

Finished design-only lane. Branch `vnc-design`, base `88332a4850d9f78b922f878fdd4770c9dff4f2f5`, head `013037b74895e2f4177668517cf38a6e6b8e8331` (pushed to `origin/vnc-design`). `research/vnc/DESIGN.md` is 199 lines and covers the feature map, reported console annoyances and responses, Rust/TypeScript architecture, codec and security ordering, budgets, conformance corpus, staged implementation plan, and owner questions. No product code changed. Evidence: - Merge-base comparison: [baseline-check.txt](https://git.kayg.org/attachments/3888b2fb-f14f-4b8d-af32-ec1e25c3068b) - Separate TypeScript half: [typescript-check.txt](https://git.kayg.org/attachments/1273c1ae-c2c5-4091-87d8-890f59edabb0) - The full workspace run's `review_mesh_watch_reports_liveness_expiry_without_another_edit` test failed under parallel load. In the same busy window the isolated test passed at merge base (load 5.02/4.29/6.09; 10.90 s) and branch (7.98/7.81/7.04; 11.62 s). The complete diff is only `research/vnc/DESIGN.md`. - Same-recipe static sizes at merge base and branch were identical: x86_64 22,329,104 bytes (0-byte delta; +124,800 over the 22,204,304 cap); aarch64 19,752,920 bytes (0-byte delta; +114,176 over the 19,638,744 cap). Both existing cap overruns remain unresolved; this doc adds no executable bytes. - Separate TypeScript half exited 0: `bun x tsc --project tsconfig.json`, tracer/browser bundle and transfer checks, and 37 protocol tests passed; the stream comparison covered 134 streams and 11,439 schedules with zero divergences. Verbatim final `scripts/verify fast` Rust summary and static-size output: ```text PT_INTERP: absent; size: 22329104 bytes; cap: 22204304 bytes; delta: +124800 bytes binary exceeds the 22204304-byte size budget: 22329104 bytes size: 19752920 bytes; cap: 19638744 bytes; delta: +114176 bytes +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | test | FAIL (101) | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` The full gate stopped before its integrated TypeScript phase at the Rust test and static-size failures; the separate TypeScript-half run above passed. Worktree is clean at the pushed head. Open owner questions in the design: 1. Should the mirrored Rust/TypeScript decoder be the first measured candidate, or must the prototype also compare server-decoded pixel updates first? 2. With shared upstream quality and resolution, should the controller own both, or should the holder aggregate visible viewports and route quality? 3. Which macOS versions and Apple security types (30, 33, 35, 36) are first-wave requirements? 4. Are QEMU/Proxmox and macOS Screen Sharing the first compatibility gate, or must a named Linux VNC server be included? 5. Is text-only clipboard acceptable for the first usable slice, with richer formats and file transfer capability-gated later? 6. Confirm the proposed ≤128 KiB gzip deferred browser chunk and local p95/p99 latency targets before making them hard gates? 7. Is synthetic protocol coverage enough for the first decoder review until a sanitized macOS fixture is available?

kayg referenced this issue from a commit

2026-10-15 21:19:18 +00:00

docs(vnc): design first-party remote desktop window (#223)

~~kayg referenced this issue 2026-10-28 17:53:56 +00:00~~

Orchestrator handover 2026-10-28: moving to a new machine #254

kayg referenced this issue

2026-10-28 18:00:45 +00:00

Orchestrator handover 2026-10-28: moving to a new machine #254

Rows
Columns

VNC window kind: libraries vs our own RFB client, measured #223