kayg/infraternal
Watch
1
0
Fork
You've already forked infraternal
 0
Code Issues 356 Pull requests Projects Releases Packages Wiki Activity Actions

S4 research: VNC client libraries and pipeline (performance first) #23

New issue
Closed
opened 2026-10-06 14:40:28 +00:00 by kayg · 9 comments
kayg commented 2026-10-06 14:40:28 +00:00
Owner
Copy link

Owner request (2026-10-06): research the best VNC client libraries for S4 (DECISIONS 92). The goal is performance, performance, and more performance. infraternal builds the UI itself: split panes (terminal on top and VNC below, 2×2, and so on), very fluid. This issue holds the research. The library choice is foundational, so it goes to the owner as a grill question with measured trade-offs (CLAUDE.md, "Foundational choices").

Owner request (2026-10-06): research the best VNC client libraries for S4 (DECISIONS 92). The goal is performance, performance, and more performance. infraternal builds the UI itself: split panes (terminal on top and VNC below, 2×2, and so on), very fluid. This issue holds the research. The library choice is foundational, so it goes to the owner as a grill question with measured trade-offs (CLAUDE.md, "Foundational choices").
kayg commented 2026-10-06 14:41:32 +00:00
Author
Owner
Copy link

Starting VNC client-library and pipeline research on branch research-vnc; base SHA 8c70f724dc.

Starting VNC client-library and pipeline research on branch research-vnc; base SHA 8c70f724dcbd9080dd85e2f9862cb16b12ba878a.
kayg commented 2026-10-06 14:46:52 +00:00
Author
Owner
Copy link

Finding (2026-10-06): noVNC is active and MPL-2.0, with 2,619 commits and 71–72 open issues on GitHub; release 1.7.0 is dated 2026-04-28 and added H.264 detection improvement (v1.6.0 added H.264). Its current source advertises Raw/CopyRect/RRE/Hextile/Zlib/Tight/ZRLE/TightPNG/JPEG/H.264 and QEMU ExtendedKeyEvent, ExtendedDesktopSize, Fence, ContinuousUpdates, ExtendedClipboard. Sources: https://github.com/novnc/noVNC/releases and https://github.com/novnc/noVNC/blob/master/core/encodings.js. Finding: vnc-rs is dual MIT/Apache-2.0, has a WASM path, and its current README says tested Tight/ZRLE/Raw, with Hextile/RRE not implemented; it is a client core, unlike the Rust rfb-encodings crate (Apache-2.0), which is an encoder library. Sources: https://github.com/HsuJv/vnc-rs and https://github.com/rustvnc/rfb-encodings. Finding: KasmVNC 1.5.0 (2026-07-29) adds hardware/software H.264, H.265 and AV1 video streaming; configuration exposes encoded video mode and WebP/JPEG rectangle modes. Sources: https://github.com/kasmtech/KasmVNC/releases and https://docs.kasmvnc.com/docs/configuration/.

Finding (2026-10-06): noVNC is active and MPL-2.0, with 2,619 commits and 71–72 open issues on GitHub; release 1.7.0 is dated 2026-04-28 and added H.264 detection improvement (v1.6.0 added H.264). Its current source advertises Raw/CopyRect/RRE/Hextile/Zlib/Tight/ZRLE/TightPNG/JPEG/H.264 and QEMU ExtendedKeyEvent, ExtendedDesktopSize, Fence, ContinuousUpdates, ExtendedClipboard. Sources: https://github.com/novnc/noVNC/releases and https://github.com/novnc/noVNC/blob/master/core/encodings.js. Finding: vnc-rs is dual MIT/Apache-2.0, has a WASM path, and its current README says tested Tight/ZRLE/Raw, with Hextile/RRE not implemented; it is a client core, unlike the Rust rfb-encodings crate (Apache-2.0), which is an encoder library. Sources: https://github.com/HsuJv/vnc-rs and https://github.com/rustvnc/rfb-encodings. Finding: KasmVNC 1.5.0 (2026-07-29) adds hardware/software H.264, H.265 and AV1 video streaming; configuration exposes encoded video mode and WebP/JPEG rectangle modes. Sources: https://github.com/kasmtech/KasmVNC/releases and https://docs.kasmvnc.com/docs/configuration/.
kayg commented 2026-10-06 15:11:20 +00:00
Author
Owner
Copy link

Finding (2026-10-06): on this host, a software-preferred Chromium WebCodecs H.264 + OffscreenCanvas 2D run decoded 180×1280×720 frames at 193.59 fps for one pane (p95 submit-to-draw 24.1 ms); four interleaved decoders processed 720 frames at 246.07 aggregate fps (p95 80.9 ms, about 61.5 fps/pane). This is a desktop reference point, not iPhone evidence; runner: research/vnc/benchmark/decode.mjs. QEMU's current RFB implementation exposes classic encodings and QEMU extended key events, but no H.264 RFB encoding, Fence or ContinuousUpdates; Proxmox vncproxy/vncwebsocket remains a ticketed RFB path, so an agent-to-video path would decode+re-encode ordinary QEMU frames. Sources checked 2026-10-06: https://github.com/qemu/qemu/blob/master/ui/vnc.h ; https://github.com/proxmox/qemu-server/blob/master/debian/changelog ; https://www.w3.org/TR/webcodecs/ . Full trade-offs and caveats will be in research/vnc/REPORT.md.

Finding (2026-10-06): on this host, a software-preferred Chromium WebCodecs H.264 + OffscreenCanvas 2D run decoded 180×1280×720 frames at 193.59 fps for one pane (p95 submit-to-draw 24.1 ms); four interleaved decoders processed 720 frames at 246.07 aggregate fps (p95 80.9 ms, about 61.5 fps/pane). This is a desktop reference point, not iPhone evidence; runner: research/vnc/benchmark/decode.mjs. QEMU's current RFB implementation exposes classic encodings and QEMU extended key events, but no H.264 RFB encoding, Fence or ContinuousUpdates; Proxmox vncproxy/vncwebsocket remains a ticketed RFB path, so an agent-to-video path would decode+re-encode ordinary QEMU frames. Sources checked 2026-10-06: https://github.com/qemu/qemu/blob/master/ui/vnc.h ; https://github.com/proxmox/qemu-server/blob/master/debian/changelog ; https://www.w3.org/TR/webcodecs/ . Full trade-offs and caveats will be in research/vnc/REPORT.md.
kayg commented 2026-10-06 15:27:03 +00:00
Author
Owner
Copy link

Benchmark correction: the earlier figures were single host runs and are superseded by a variable-load note. Three four-decoder runs measured 80.40, 153.70, and 240.95 aggregate fps. The instrumented 2026-10-06 15:26 UTC run reported host load average 15.39→16.40 and 146.6 ms p95 submit-to-draw (38.43 fps/pane); earlier runs did not capture load. This is decode/draw only, not phone or capacity evidence. The runner and report will record the conditions and spread: research/vnc/benchmark/decode.mjs and research/vnc/REPORT.md.

Benchmark correction: the earlier figures were single host runs and are superseded by a variable-load note. Three four-decoder runs measured 80.40, 153.70, and 240.95 aggregate fps. The instrumented 2026-10-06 15:26 UTC run reported host load average 15.39→16.40 and 146.6 ms p95 submit-to-draw (38.43 fps/pane); earlier runs did not capture load. This is decode/draw only, not phone or capacity evidence. The runner and report will record the conditions and spread: research/vnc/benchmark/decode.mjs and research/vnc/REPORT.md.
kayg commented 2026-10-06 15:30:37 +00:00
Author
Owner
Copy link

Gate finding: the worktree has no scripts/verify entry (scripts/ contains only perf-run). The required invocation exits 127 with: zsh:1: no such file or directory: scripts/verify. I am recording this limitation; report review and the benchmark syntax/diff checks pass, but this gate cannot run in this checkout.

Gate finding: the worktree has no scripts/verify entry (scripts/ contains only perf-run). The required invocation exits 127 with: zsh:1: no such file or directory: scripts/verify. I am recording this limitation; report review and the benchmark syntax/diff checks pass, but this gate cannot run in this checkout.
kayg commented 2026-10-06 15:30:49 +00:00
Author
Owner
Copy link

Clarification: the exact required command is scripts/verify fast. It exits 127 with the output: zsh:1: no such file or directory: scripts/verify. scripts/ contains only perf-run in this checkout.

Clarification: the exact required command is scripts/verify fast. It exits 127 with the output: zsh:1: no such file or directory: scripts/verify. scripts/ contains only perf-run in this checkout.
kayg commented 2026-10-06 15:31:36 +00:00
Author
Owner
Copy link

Finished the VNC research lane. research/vnc/REPORT.md compares browser RFB decode, agent-side video/WebCodecs, and a Guacamole proxy; includes dated sources, client and server compatibility tables, Safari support, multi-pane/input guidance, license notes, and owner grill questions. No library or architecture is selected. The host decode/draw run measured 153.70 aggregate fps across four panes (38.43 per pane), 146.6 ms p95 at load average 15.39 to 16.40; three runs ranged from 80.40 to 240.95 aggregate fps, so this is not phone or capacity evidence. Runner: research/vnc/benchmark/decode.mjs. Commits: 7c6dda9 and 137a569. Head SHA: 137a5694b1. scripts/verify is absent; scripts/ contains only perf-run. Gate output verbatim: zsh:1: no such file or directory: scripts/verify.

Finished the VNC research lane. research/vnc/REPORT.md compares browser RFB decode, agent-side video/WebCodecs, and a Guacamole proxy; includes dated sources, client and server compatibility tables, Safari support, multi-pane/input guidance, license notes, and owner grill questions. No library or architecture is selected. The host decode/draw run measured 153.70 aggregate fps across four panes (38.43 per pane), 146.6 ms p95 at load average 15.39 to 16.40; three runs ranged from 80.40 to 240.95 aggregate fps, so this is not phone or capacity evidence. Runner: research/vnc/benchmark/decode.mjs. Commits: 7c6dda9 and 137a569. Head SHA: 137a5694b16e6dd9279d0bd808d8667770e9310d. scripts/verify is absent; scripts/ contains only perf-run. Gate output verbatim: zsh:1: no such file or directory: scripts/verify.
kayg commented 2026-10-06 15:42:52 +00:00
Author
Owner
Copy link

Owner answers recorded as DECISIONS 104–107 (b66edda). The report is merged into main (cdc7c59). Follow-up spike: #35.

Owner answers recorded as DECISIONS 104–107 (b66edda). The report is merged into main (cdc7c59). Follow-up spike: #35.
kayg commented 2026-10-28 17:52:14 +00:00
Author
Owner
Copy link

Closed by the orchestrator during the 2026-10-28 handover (#254), on the owner's request to close finished issues. Evidence: VNC research delivered (137a569, cdc7c59 in dev); owner answers recorded as D104-107, follow-up spike is #35. Reopen if anything is still missing.

Closed by the orchestrator during the 2026-10-28 handover (#254), on the owner's request to close finished issues. Evidence: VNC research delivered (137a569, cdc7c59 in dev); owner answers recorded as D104-107, follow-up spike is #35. Reopen if anything is still missing.
kayg closed this issue 2026-10-28 17:52:15 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
j470b-server-mesh-rejoin
mesh-compat-382
j444b-d381-key-wins-flood
j461b-macos-utun-reader-thread
j475b-remove-host-motion
fix-reconnect-new-link
j481-d380
dev
j479-mesh-soak
fix/j476-host-facts
j475-motion-pill-split
j470-mac-node-join
root-upgrade-d376
j474-review-merge
latency-targets-120hz
nightly-suite-shards
gotatun-pin-j469
macos-utun-reader-thread
fix-flood-echo-stall
wip/j416
motion-fast-ease-out
mesh-addhost-j259
takeover-close-terminals-d368
kbd-instant-followups
fix-trust-key-double-click
nightly-suite-partial-report
appwin-icons-j207
rusqlite-slice1
swipe-content-first-d374
mesh-mtu-381
addhost-instant-open
glass-contrast-d365-mock
glass-contrast-d365
signout-followups-442
fix-startup-toast-red
research-gotatun-matrix-j317
themes-sync-j304
fix-load-flakes
fix-nix-zig-patch
fix-compute01-browser-baseline
ratchet-size-baseline-419
predict-auto-d130
revoke-coverage-d367
nightly-suite-queue-timeout
nightly-browser-suite
fix-rustsec-2026-0459
fonts-ioskeley-obel
log-retention-d375
fix-kbd-no-anim-427
fix-dev-red-specs-2
takeover-proof-305-j205
wip/j410-server-pump
ui-consistency-d372
fix-osc52-clipboard
libghostty-vt-1.0
forget-retired-host-d369
j404-merge-426
fix-setup-lock-race
fix-touch-select-426
revoke-access-report-ui
fix-predictive-echo-specs
fix-selinux-agent-exec
wip/j395-checkpoint2
wip/j395
j391-review-scratch
ui-consistency-j195-deferred-rebased
fix-rustc-paths-421
decisions-wording-341
fix-install-suite-red
fix-dev-rtt-specs
fix-pending-removal-align
fix-privileged-skip
fix-install-suite-404
review-j368
mesh-dns-j254
revoke-access-report-d356b
revoke-privilege-round5
fix-removal-confirm
review-j360
gotatun-rotation-367
deps-refresh-340
review-j354
review-J351
review-j350
addhost-takeover-d359
revoke-report-items-347
revoke-end-sessions-d362
review-j348
size-mesh-ssh-346
size-trim-337
revoke-access-diff-d356
j259-pre-rebase-b06c5a6
j305-ended-at-start-spec
addhost-sudo-consent
fix-layout-real-specs-411
fix-shell-open-411
fix-console-hosts-373
fix-shell-exit
fuzz-host-name-353
fix-host-details-410
macos-filter-j197
research-gotatun-tricks
revoke-p3-j216l
stale-state-pill
revoke-followups-j216k
revoke-progress-j216e
backup/macos01-20261106/lume/j197
backup/macos01-20261106/lume/appwin-icons-j207-stash0
backup/macos01-20261106/claude/stash-0
backup/macos01-20261106/claude/stash-1
backup/macos01-20261106/claude/J80-macos-s1
backup/macos01-20261106/claude/J80
backup/macos01-20261106/claude/J80-slice1-20261102
backup/macos01-20261106/claude/J16b-appwin-on-dev
backup/macos01-20261106/claude/appwin-launch-idle-146
backup/macos01-20261106/claude/J80-s1-run
wip/j259
trust-285-p3c
runtime-lock-j216f
mesh-ssh-p3-j175
tun-proof-p3b
trust-285-p3
terminal-polish-j271
tun-proof-p3
ssh-install-flakes-j216i
fuzz-fixes-j264
trust-285-a
trust-285-b
restart-oversized-fixture-j198
backup/macos01-20261106/lume/appwin-icons-j207
restart-bound-j216h
up-preflight-p3-j198
backup/macos01-20261106/lume/j251
up-preflight-j198
wip/up-preflight-j198-j245-p2
wip/up-preflight-j198-j245-p1
agent-fsync-offloop-j196
wip/up-preflight-j198-j245
settings-offscreen-j231
wip/up-preflight-j198-j226-inspection
wip/j103f
themes-j206
wip/up-preflight-j198-j226
flakes-j216c
backup/macos01-20261106/claude/j197
flakes-j216b
backup/macos01-20261106/claude/j197-red
wip/up-preflight-j198-review
flakes-j216
backup/macos01-20261106/claude/appwin-icons-j207
engineering-j208b
appwin-mock-j209
toast-stack-j165
engineering-j208
decisions-d351
decisions-d350
decisions-d349
appearance-presets-mock-j129
decisions-d348
docs-audit-spec
doc-coverage-baseline-raise
decisions-d347
decisions-d346b
decisions-d346
j88h-profile
ui-consistency-j195-deferred
ui-consistency-j195
engineering-practices-j194
preferred-login-ui-j166
signin-pools-263-j174
macos-node-265-s1
decisions-d345
decisions-d344
proxy-cgroup-j164
verify-quick-fuzz-329
proxy-start-io-j168
timing-flakes-j163
up-263-s1
mcp-events-j147
verify-quick-fuzz-j158
proxy-p3-j159
agent-resource-j157-detect
wip/j103d-final
remote-cancel-hang-j155
proxy-traefik-281-j62d
j149-verify-quick
proxy-traefik-281-j62b
j150-drop-unnamed
wip/j79d-red-r4
wip/j103d-d332
rootless-toast-j142
wip/j103d-head
wip/j103d
j125c-mcp
macos-build-j136
backup/macos01-20261106/claude/j136
wip/j79d-red-cli
wip/j79d-red-server
j121-link-liveness
j118c-tracer-failures
wip/j79c-head
release-desktop-j131
j124c-signed-macos
wip/j79c-red-p3
wip/j79c-red-p23
wip/j79c-red-p22
merge-j128-check
perf-deferred-j128
sidebar-toggle-260
j127-shortcut-changed-dot
j125-mcp
j118-tracer-failures
j126-account-summary
settings-sections-mock-j92
j120-pinch-lock
sidebar-292
backup/macos01-20261106/claude/j120
taste-2026-11-03
row-actions-242-215
backup/macos01-20261106/claude/stash-0-untracked
backup/macos01-20261106/claude/J80-265-s1-run
backup/macos01-20261106/claude/J80-macos-s1-20261102
j112-canvas-commands
ui-no-overshoot-scroll
fix-recovery-restart
backup/macos01-20261106/claude/stash-1-untracked
taste-2026-11-02
size-baseline-ratchet
machine-cards-structure-j108
j109-boot-copy-mockups
fix-dev-red
wip/up-263-s1
j95b-settings-rows
proxy-traefik-281
j95-option-rows
docs-human-j91
docs-keys-palette
fix-flood-1013-270-j38d
mesh-ssh-s1
mesh-ssh-s0
addhost-d291-tunnel
test-stability-load
mesh-relay-220
repro-296
shim-thp-off-24
mesh-pins-258-v2
mesh-dns
signin-d311-merged
docs-pages
ui-size-283
fix-ui-batch-specs
wip/j12b-edge-update
gate-on-arm
fix-rename-sync-293
trust-285-j60b
fix-focus-ring-clip
integ-ui-1101
edge-update-v2
signin-d311
taste-2026-11-01
layout-per-tab-250
ui-approved-splitopt-224
ui-approved-splitopt-224-on-dev
icon-text-centre-282
ui-approved-189-212-216
research-reverse-proxy
transfer-budget-320
review-mesh-pins-258-r2
sidebar-pill-259
review-mesh-relay-220
wip/j12-edge-update-v2
nix-package
status-crash-loop-237
fix-dev-test-failures
fix-fixture-socket-path
wip/mesh-relay-220-pre-dev-advance-rebase
fix-canvas-side-clip
research-head-to-head
frosted-real-256
review-mesh-pins-258
wip/mesh-relay-220-pre-final-rebase
review-layout-per-tab-250-r2
broad-suite-triage
nix-devshell
review-status-crash-loop-237-r3
wip/nix-devshell-pre-p3
taste-frosted-real
wip/code-serve-demo-2026-10-30
wip/code-serve-glass-2026-10-30
wip/code-serve-clusters-2026-10-30
wip/code-merge-release94-dev-2026-10-30
wip/rproxy-delta-scratch-2026-10-30
wip/stale-note-257-staged-2026-10-30
wip/d241-size-staged-2026-10-30
review-broad-suite-207
wip/mesh-pins-258-pre-split
wip/mesh-relay-220-pre-rebase
mesh-measure-220
plan-211-update
review-edge-autoupdate-r3
perf24-reconnect
review-stale-note-r2
release-prod-live
d241-shell-marker
review-mesh-pins-server-258
review-revocation-r3
review-setup-token-r2
review-sync-journal-r3
wip/review-mesh-pins-258-prune-review-1995fee46
frosted-scroll-256
sync-journal-r3
revocation-per-account-wip
review-tokio-asyncfd-0386
review-sync-p3s
wip/stash-8by64-stash0-2026-10-30
mesh-drag-r2
setup-token-data-wip
release-9-4
auth-busy-1013
mesh-view-overlap
sync-p3s
review-auth-busy-wake
review-readd-r8-wip
mesh-drag-bounds
audit-auth-93
app-windows-stream-start
review-authority-cap-r3
app-windows-r7-measure
readd-minimal-r6
readd-minimal-r5
app-windows-r7-soak
readd-minimal-r4
app-windows-r7
app-windows-r6
readd-minimal-r3
readd-minimal-r2
readd-minimal
merge-release8-dev
review-busy-page-liveness-r2
review-busy-page-liveness
mesh-view-trim
review-authority-read-retry-r2
review-remove-uninstall-r2
release-retry-reattach
release-retry
review-remove-uninstall
review-authority-read-retry
app-windows-ship
mesh-autoswitch-p2
mesh-ssh
mesh-relay
mesh-macos-node
screen-and-control
session-labels
review-real-logins-r5
review-real-logins-r4
review-real-logins-r3
shim-memory-r2
review-real-logins-r2
shim-memory
shadow-addhost-instant
shadow-real-logins
shadow-same-session-flood
shadow-prod-perf-fix
hosts-from-server
app-windows-real
app-windows-transport
release-prod-6w
mesh-policy-verify
quiet-close-toast
dev-merge-r4
review-mesh-agent-transport
review-app-windows-transport
settings-roundtrip
review-release-pipeline-r4
mesh-agent-transport
ended-strip-fix
review-audit-ca-update-d244
settings-rows
phone-width
signout-clear
fix-restart-tests
dev-merge-rp
signer-hardening
review-audit-ca-update
terminal-fidelity
review-audit-ca-update-wip
mesh-integrate
review-mesh-agent-transport-515d3712
review-mesh-dataplane-feec1988
mesh-dataplane
signout
research-canvas-node
wip/review-renderer-cost
wip/review-terminal-fidelity
padlock-glyph
review-release-pipeline-2e95b689
phone-polish
sidebar-polish
inbox-generic
review-mesh-dataplane-378d4055
prod-batch
ui-inbox
ui-ca-unlock
ui-machine-sections
review-mesh-dataplane-a5f744fb
tls-session-prod
session-bounds-prod
fix-dead-session
session-bounds
tls-session
palette-polish
shortcuts-d222
research-appmeta
phone-gutter
ui-centre-button
wip/ui-centre-button
wip/tls-session
review-addhost-breakit
ui-platform
renderer-cost
remote-agents
mesh-api
ssh-bootstrap
prod-hotfix2
prod-fixes
review-remote-agents
review-mesh-dataplane
session-life
research-browser
review-mesh-api
research-capture
review-mesh-api-round3
review-mesh-api-round1
gate-stability
review-gate-stability
review-relay-direct
review-mesh-api-findings
proto-sessions
font-picker
proto-addhost
proto-windows
mesh-core
latency-gate
research-bsd
research-discovery
keytray-mods
mock-core
ui-tweaks
fix-motion-sel
fix-motion
integrate2
fix-robust
t-web
t-server
vt-ghostty
fix-snap-echo
t-web-auth
review-vt2
t-agent
lane-protocol
lane-shim
lane-auth
s1-base
s1-contracts
s1-workspace
No results found.
Labels
Clear labels   
launch-qa
Findings from launch QA.

No labels
launch-qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/infraternal#23
No description provided.