S4: agent-held VNC sessions, macOS Screen Sharing spike and codec benchmark

kayg commented

2026-10-06 15:42:37 +00:00

Owner

S4 remote desktop, after DECISIONS 104–107 (owner answers on #23):

The session is held by the server or its infraternal agent (like SSH), and clients subscribe.
The holder keeps the framebuffer for an instant full image on join or rejoin.
macOS Screen Sharing first, then Proxmox/QEMU.
The client floor is iOS/Safari 17 (WebCodecs always available).
The holder → subscriber encoding is chosen by benchmark for the lowest input-to-photon latency.

Spike plan (measure, don't choose; the owner decides the engine):

An RFB client in the holder against macOS Screen Sharing (auth: Apple DH and VNC password; record what current macOS offers). Report the negotiated encodings.
A framebuffer at the holder, then candidate encodings to the browser: dirty-rect lossless (zstd or QOI), Tight/JPEG passthrough, and H.264 low-delay (software x264 and VideoToolbox on a Mac).
Measure input-to-photon latency, bandwidth, phone decode CPU and battery (real iPhone), and server CPU per session × the expected count.

Needs: a macOS target. The M5 Ultra Mac Studio when it arrives, or ask the owner about using the calternal macOS VM.

S4 remote desktop, after DECISIONS 104–107 (owner answers on #23): - The session is held by the server or its infraternal agent (like SSH), and clients subscribe. - The holder keeps the framebuffer for an instant full image on join or rejoin. - macOS Screen Sharing first, then Proxmox/QEMU. - The client floor is iOS/Safari 17 (WebCodecs always available). - The holder → subscriber encoding is chosen by benchmark for the lowest input-to-photon latency. Spike plan (measure, don't choose; the owner decides the engine): 1. An RFB client in the holder against macOS Screen Sharing (auth: Apple DH and VNC password; record what current macOS offers). Report the negotiated encodings. 2. A framebuffer at the holder, then candidate encodings to the browser: dirty-rect lossless (zstd or QOI), Tight/JPEG passthrough, and H.264 low-delay (software x264 and VideoToolbox on a Mac). 3. Measure input-to-photon latency, bandwidth, phone decode CPU and battery (real iPhone), and server CPU per session × the expected count. **Needs:** a macOS target. The M5 Ultra Mac Studio when it arrives, or ask the owner about using the calternal macOS VM.

kayg referenced this issue

2026-10-06 15:42:51 +00:00

Status board: work in flight (kept current) #24

kayg referenced this issue

2026-10-06 15:42:52 +00:00

S4 research: VNC client libraries and pipeline (performance first) #23

kayg commented

2026-10-06 15:46:09 +00:00

Author

Owner

Target access checked (orchestrator, 2026-10-06):

netbird ssh --no-browser calternal@10.69.69.21 works. The VM runs macOS 27.0 (26A428), arm64, 4 vCPU, 8 GB. It is shared with calternal, so the spike must not change system settings and must leave the VM as it found it.
Screen Sharing only listens privately. netbird ssh -L 15900:localhost:5900 calternal@10.69.69.21 'sleep N' exposes it on localhost:15900.
The server banner is RFB 003.889 (Apple's dialect). The offered security types are 30, 33, 36, 31, 32, 35. Type 2 (VNC password) is not offered, so a session needs the Apple authentication methods (macOS account credentials).

Open (owner): which macOS account and password the spike may use, and where they are stored (not in the repo, logs or argv). Until then the spike surveys authentication and benchmarks encodings on screen content captured over SSH.

Target access checked (orchestrator, 2026-10-06): - `netbird ssh --no-browser calternal@10.69.69.21` works. The VM runs macOS 27.0 (26A428), arm64, 4 vCPU, 8 GB. It is shared with calternal, so the spike must not change system settings and must leave the VM as it found it. - Screen Sharing only listens privately. `netbird ssh -L 15900:localhost:5900 calternal@10.69.69.21 'sleep N'` exposes it on localhost:15900. - The server banner is `RFB 003.889` (Apple's dialect). The offered security types are 30, 33, 36, 31, 32, 35. **Type 2 (VNC password) is not offered**, so a session needs the Apple authentication methods (macOS account credentials). **Open (owner):** which macOS account and password the spike may use, and where they are stored (not in the repo, logs or argv). Until then the spike surveys authentication and benchmarks encodings on screen content captured over SSH.

kayg referenced this issue

2026-10-06 15:46:33 +00:00

Status board: work in flight (kept current) #24

kayg commented

2026-10-06 15:47:34 +00:00

Author

Owner

Starting phase A on branch spike-vnc at base SHA b66edda; reading existing #35 and parent #23 findings. No macOS account credentials are available, so the protocol survey will stop before authentication.

Starting phase A on branch spike-vnc at base SHA b66edda; reading existing #35 and parent #23 findings. No macOS account credentials are available, so the protocol survey will stop before authentication.

kayg commented

2026-10-06 15:47:52 +00:00

Author

Owner

Owner provided a macOS account for the spike (2026-10-06). It is stored at ~/.config/infraternal-secrets/macos-vnc.env (mode 600, keys VNC_USER and VNC_PASS). Load it into the environment only; never print it, pass it in argv, log it or commit it. Phase B (a real authenticated session) uses it after phase A reports.

Owner provided a macOS account for the spike (2026-10-06). It is stored at `~/.config/infraternal-secrets/macos-vnc.env` (mode 600, keys VNC_USER and VNC_PASS). Load it into the environment only; never print it, pass it in argv, log it or commit it. Phase B (a real authenticated session) uses it after phase A reports.

kayg commented

2026-10-06 15:57:31 +00:00

Author

Owner

Handshake spike committed as bb236e1d5a. Live probe against the SSH-forwarded macOS VM read RFB 003.889 and the six offered types [30,33,36,31,32,35]; selecting 30 sent one byte and closed before reading or answering auth. cargo build and cargo clippy both passed.

Handshake spike committed as bb236e1d5af67a43163a830608d56abdd49a0d55. Live probe against the SSH-forwarded macOS VM read RFB 003.889 and the six offered types [30,33,36,31,32,35]; selecting 30 sent one byte and closed before reading or answering auth. cargo build and cargo clippy both passed.

kayg commented

2026-10-06 16:12:38 +00:00

Author

Owner

Corpus step complete; manifest committed as 9819d80d50. The external corpus has 27 files / 146,739,237 bytes: 22 PNG stills at 5004x2652 and three 4-second H.264 captures at 4096x2170 (27, 51 and 45 frames). Terminal scroll and eight scripted window-movement positions are represented. worked over SSH. Payloads are in ; only hashes, sizes and capture metadata are committed in research/vnc/SPIKE-A-CORPUS-MANIFEST.md. The VM was left with the generated Terminal window closed and Notes foreground.

Corpus step complete; manifest committed as 9819d80d500a609d19e506ce3dc310c987ceec38. The external corpus has 27 files / 146,739,237 bytes: 22 PNG stills at 5004x2652 and three 4-second H.264 captures at 4096x2170 (27, 51 and 45 frames). Terminal scroll and eight scripted window-movement positions are represented. worked over SSH. Payloads are in ; only hashes, sizes and capture metadata are committed in research/vnc/SPIKE-A-CORPUS-MANIFEST.md. The VM was left with the generated Terminal window closed and Notes foreground.

kayg commented

2026-10-06 16:13:27 +00:00

Author

Owner

Correction to the prior corpus comment: shell quoting removed the screencapture -v and scratch-path text. Corpus step complete; commit 9819d80d50 indexes 27 external files / 146,739,237 bytes: 22 5004x2652 PNG stills and three 4096x2170 H.264 movies (27, 51, 45 frames). The videos were captured over SSH. Manifest: research/vnc/SPIKE-A-CORPUS-MANIFEST.md; payloads: /home/kayg/.cache/codex-scratch/spike-vnc/corpus/. The generated Terminal window was closed and Notes restored. No authentication data was read or sent.

Correction to the prior corpus comment: shell quoting removed the screencapture -v and scratch-path text. Corpus step complete; commit 9819d80d500a609d19e506ce3dc310c987ceec38 indexes 27 external files / 146,739,237 bytes: 22 5004x2652 PNG stills and three 4096x2170 H.264 movies (27, 51, 45 frames). The videos were captured over SSH. Manifest: research/vnc/SPIKE-A-CORPUS-MANIFEST.md; payloads: /home/kayg/.cache/codex-scratch/spike-vnc/corpus/. The generated Terminal window was closed and Notes restored. No authentication data was read or sent.

kayg commented

2026-10-06 16:37:42 +00:00

Author

Owner

Rect codec finding (d1afae3 + f09773f): release Rust encoder measured 22 screenshot frames per codec/resolution, including dirty diff and crop. At 1280x800: JPEG q75 13.32/23.34 ms p50/p99, 48,948 B/frame, terminal PSNR 40.22 dB; WebP q75 30.73/59.19 ms, 20,950 B/frame, 42.35 dB; zstd L1 3.85/5.86 ms, 253,520 B/frame; LZ4 3.00/7.98 ms, 325,435 B/frame; QOI 2.97/5.10 ms, 234,352 B/frame. Chromium 153 decode p50/p99: JPEG q75 4.4/11.1 ms, WebP q75 4.5/10.9 ms, zstd WASM 2.9/11.2 ms, LZ4 WASM 2.0/5.1 ms, QOI WASM 4.9/15.5 ms. Results and 396-row per-frame dataset are in $HOME/.cache/codex-scratch/spike-vnc/rects/rects.json and browser-decode.json; runner is committed under research/vnc/benchmark/.

Rect codec finding (d1afae3 + f09773f): release Rust encoder measured 22 screenshot frames per codec/resolution, including dirty diff and crop. At 1280x800: JPEG q75 13.32/23.34 ms p50/p99, 48,948 B/frame, terminal PSNR 40.22 dB; WebP q75 30.73/59.19 ms, 20,950 B/frame, 42.35 dB; zstd L1 3.85/5.86 ms, 253,520 B/frame; LZ4 3.00/7.98 ms, 325,435 B/frame; QOI 2.97/5.10 ms, 234,352 B/frame. Chromium 153 decode p50/p99: JPEG q75 4.4/11.1 ms, WebP q75 4.5/10.9 ms, zstd WASM 2.9/11.2 ms, LZ4 WASM 2.0/5.1 ms, QOI WASM 4.9/15.5 ms. Results and 396-row per-frame dataset are in $HOME/.cache/codex-scratch/spike-vnc/rects/rects.json and browser-decode.json; runner is committed under research/vnc/benchmark/.

kayg commented

2026-10-06 16:37:50 +00:00

Author

Owner

Path clarification for the codec finding: benchmark outputs are at /home/kayg/.cache/codex-scratch/spike-vnc/rects/rects.json and /home/kayg/.cache/codex-scratch/spike-vnc/rects/browser-decode.json.

kayg commented

2026-10-06 17:11:03 +00:00

Author

Owner

Rectangle benchmark step complete at c550f5a: browser decoder now times canvas composite separately; 396 rows cover 22 stills, 9 codecs and 2 sizes. At 1280x800 JPEG q75 measured 13.57/24.22 ms p50/p99, 48,948 B/frame, 40.23 dB terminal PSNR; WebP q75 32.40/61.70 ms, 20,950 B/frame, 42.39 dB; QOI 3.65/5.26 ms, 234,352 B/frame and lossless. These wire rates use the 23 s still sequence; native changed-rect JPEG q75 took 206.3/353.5 ms. Browser timings are headless Chromium on the Linux host.

Rectangle benchmark step complete at c550f5a: browser decoder now times canvas composite separately; 396 rows cover 22 stills, 9 codecs and 2 sizes. At 1280x800 JPEG q75 measured 13.57/24.22 ms p50/p99, 48,948 B/frame, 40.23 dB terminal PSNR; WebP q75 32.40/61.70 ms, 20,950 B/frame, 42.39 dB; QOI 3.65/5.26 ms, 234,352 B/frame and lossless. These wire rates use the 23 s still sequence; native changed-rect JPEG q75 took 206.3/353.5 ms. Browser timings are headless Chromium on the Linux host.

kayg commented

2026-10-06 17:12:17 +00:00

Author

Owner

H.264 benchmark step complete at 4d0fc55: 12 x264 cases, 180 frames each; all 12 decoded 180/180 through Chromium WebCodecs. At 1280x800, 4:2:0 CRF30 measured 1.79/7.45 ms encode wall p50/p99 (10.91/58.15 ms user+sys), 4,910 B/frame, 73.6 kB/s, terminal SSIM 0.999072 mean / 0.995984 min; browser decode 21.4/28.6 ms plus canvas 4.2/6.5 ms. At 4096x2170 the same setting had 14.08/26.90 ms encode wall and 88.4/99.4 ms browser decode plus 18.4/24.2 ms draw. Mac VideoToolbox was unavailable because ffmpeg is absent on the VM; no install was attempted.

H.264 benchmark step complete at 4d0fc55: 12 x264 cases, 180 frames each; all 12 decoded 180/180 through Chromium WebCodecs. At 1280x800, 4:2:0 CRF30 measured 1.79/7.45 ms encode wall p50/p99 (10.91/58.15 ms user+sys), 4,910 B/frame, 73.6 kB/s, terminal SSIM 0.999072 mean / 0.995984 min; browser decode 21.4/28.6 ms plus canvas 4.2/6.5 ms. At 4096x2170 the same setting had 14.08/26.90 ms encode wall and 88.4/99.4 ms browser decode plus 18.4/24.2 ms draw. Mac VideoToolbox was unavailable because ffmpeg is absent on the VM; no install was attempted.

kayg commented

2026-10-06 17:13:49 +00:00

Author

Owner

Runner cleanup finding: Chromium wrote into its temporary profile after its parent exited; one completed rect decode run wrote browser-decode.json but exited 1 with ENOTEMPTY removing .chromium-profile. Bounded fs.rm retries are in de35df0; rerun completed with exit 0 and removed the profile.

Runner cleanup finding: Chromium wrote into its temporary profile after its parent exited; one completed rect decode run wrote browser-decode.json but exited 1 with ENOTEMPTY removing .chromium-profile. Bounded fs.rm retries are in de35df0; rerun completed with exit 0 and removed the profile.

kayg commented

2026-10-06 17:21:41 +00:00

Author

Owner

Authentication survey finding: live offer order is 30,33,36,31,32,35; type 2 is absent. I selected 30 only to confirm the selection byte, then closed before reading its challenge. The reverse-engineered Apple write-up describes AES-CBC/zero-IV for type-30 credentials, while current noVNC and TigerVNC code use AES-ECB; macOS 27 interoperability is unresolved. IANA does not explain 31/32. No credentials were read or used; the owner inputs and source links are recorded in SPIKE-A.md.

kayg commented

2026-10-06 17:22:10 +00:00

Author

Owner

Correction to the earlier rectangle p50 figures: I used the lower-middle sample when summarizing 22 frames. Using the median (mean of the two middle samples) for the final table gives JPEG q75 13.76/24.22 ms, WebP q75 34.61/61.70 ms and QOI 3.66/5.26 ms p50/p99. Byte counts and quality figures are unchanged; SPIKE-A.md has the corrected values.

kayg commented

2026-10-06 17:25:06 +00:00

Author

Owner

Report step complete at e2ee906: research/vnc/SPIKE-A.md records the auth/source matrix, owner credential inputs, corpus reference, all 30 rectangle/H.264 cases and modeled stage budgets. Rechecked all 27 corpus manifest hashes/sizes against the out-of-repo corpus. No encoding or auth method is selected; open questions include the type-30 cipher discrepancy, types 31/32, phone/network targets and sessions per holder.

Report step complete at e2ee906: research/vnc/SPIKE-A.md records the auth/source matrix, owner credential inputs, corpus reference, all 30 rectangle/H.264 cases and modeled stage budgets. Rechecked all 27 corpus manifest hashes/sizes against the out-of-repo corpus. No encoding or auth method is selected; open questions include the type-30 cipher discrepancy, types 31/32, phone/network targets and sessions per holder.

kayg commented

2026-10-06 17:27:16 +00:00

Author

Owner

Phase A complete on branch spike-vnc. HEAD e2ee906b6b. Corpus and compact metrics remain in the permitted scratch directory; generated targets, codec payloads, FFmpeg intermediates, browser bundles/profiles and node_modules were removed. No push, deploy or issue closure.

cargo build output:
Finished dev profile [unoptimized + debuginfo] target(s) in 0.11s

cargo clippy output:
Finished dev profile [unoptimized + debuginfo] target(s) in 0.09s

Phase A complete on branch spike-vnc. HEAD e2ee906b6bfe59a17da454a59b053eb446f216e5. Corpus and compact metrics remain in the permitted scratch directory; generated targets, codec payloads, FFmpeg intermediates, browser bundles/profiles and node_modules were removed. No push, deploy or issue closure. cargo build output: Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.11s cargo clippy output: Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.09s

kayg referenced this issue

2026-10-06 17:29:12 +00:00

Status board: work in flight (kept current) #24

kayg commented

2026-10-06 17:30:12 +00:00

Author

Owner

Starting phase B on branch spike-vnc at base SHA e2ee906b6b. Issue #35 and parent #23 comments are read; phase A findings and owner decisions are carried forward.

Starting phase B on branch spike-vnc at base SHA e2ee906b6bfe59a17da454a59b053eb446f216e5. Issue #35 and parent #23 comments are read; phase A findings and owner decisions are carried forward.

kayg commented

2026-10-06 17:35:59 +00:00

Author

Owner

Network baseline: NetBird SSH forward is active on 127.0.0.1:15900. PING 10.69.69.21 (10.69.69.21) 56(84) bytes of data.
64 bytes from 10.69.69.21: icmp_seq=1 ttl=64 time=154 ms
64 bytes from 10.69.69.21: icmp_seq=2 ttl=64 time=154 ms
64 bytes from 10.69.69.21: icmp_seq=3 ttl=64 time=152 ms
64 bytes from 10.69.69.21: icmp_seq=4 ttl=64 time=152 ms
64 bytes from 10.69.69.21: icmp_seq=5 ttl=64 time=152 ms
64 bytes from 10.69.69.21: icmp_seq=6 ttl=64 time=152 ms
64 bytes from 10.69.69.21: icmp_seq=7 ttl=64 time=152 ms
64 bytes from 10.69.69.21: icmp_seq=8 ttl=64 time=182 ms
64 bytes from 10.69.69.21: icmp_seq=9 ttl=64 time=190 ms
64 bytes from 10.69.69.21: icmp_seq=10 ttl=64 time=156 ms
64 bytes from 10.69.69.21: icmp_seq=11 ttl=64 time=153 ms
64 bytes from 10.69.69.21: icmp_seq=12 ttl=64 time=152 ms
64 bytes from 10.69.69.21: icmp_seq=13 ttl=64 time=152 ms
64 bytes from 10.69.69.21: icmp_seq=14 ttl=64 time=153 ms
64 bytes from 10.69.69.21: icmp_seq=15 ttl=64 time=152 ms
64 bytes from 10.69.69.21: icmp_seq=16 ttl=64 time=153 ms
64 bytes from 10.69.69.21: icmp_seq=17 ttl=64 time=162 ms
64 bytes from 10.69.69.21: icmp_seq=18 ttl=64 time=776 ms
64 bytes from 10.69.69.21: icmp_seq=19 ttl=64 time=157 ms
64 bytes from 10.69.69.21: icmp_seq=20 ttl=64 time=154 ms

--- 10.69.69.21 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 19033ms
rtt min/avg/max/mdev = 151.914/188.068/775.816/135.210 ms received 20/20 (0% loss), RTT min/avg/max 151.556/229.208/1414.520 ms, mdev 277.305 ms; end-to-end latency samples will include substantial link jitter.

Network baseline: NetBird SSH forward is active on 127.0.0.1:15900. PING 10.69.69.21 (10.69.69.21) 56(84) bytes of data. 64 bytes from 10.69.69.21: icmp_seq=1 ttl=64 time=154 ms 64 bytes from 10.69.69.21: icmp_seq=2 ttl=64 time=154 ms 64 bytes from 10.69.69.21: icmp_seq=3 ttl=64 time=152 ms 64 bytes from 10.69.69.21: icmp_seq=4 ttl=64 time=152 ms 64 bytes from 10.69.69.21: icmp_seq=5 ttl=64 time=152 ms 64 bytes from 10.69.69.21: icmp_seq=6 ttl=64 time=152 ms 64 bytes from 10.69.69.21: icmp_seq=7 ttl=64 time=152 ms 64 bytes from 10.69.69.21: icmp_seq=8 ttl=64 time=182 ms 64 bytes from 10.69.69.21: icmp_seq=9 ttl=64 time=190 ms 64 bytes from 10.69.69.21: icmp_seq=10 ttl=64 time=156 ms 64 bytes from 10.69.69.21: icmp_seq=11 ttl=64 time=153 ms 64 bytes from 10.69.69.21: icmp_seq=12 ttl=64 time=152 ms 64 bytes from 10.69.69.21: icmp_seq=13 ttl=64 time=152 ms 64 bytes from 10.69.69.21: icmp_seq=14 ttl=64 time=153 ms 64 bytes from 10.69.69.21: icmp_seq=15 ttl=64 time=152 ms 64 bytes from 10.69.69.21: icmp_seq=16 ttl=64 time=153 ms 64 bytes from 10.69.69.21: icmp_seq=17 ttl=64 time=162 ms 64 bytes from 10.69.69.21: icmp_seq=18 ttl=64 time=776 ms 64 bytes from 10.69.69.21: icmp_seq=19 ttl=64 time=157 ms 64 bytes from 10.69.69.21: icmp_seq=20 ttl=64 time=154 ms --- 10.69.69.21 ping statistics --- 20 packets transmitted, 20 received, 0% packet loss, time 19033ms rtt min/avg/max/mdev = 151.914/188.068/775.816/135.210 ms received 20/20 (0% loss), RTT min/avg/max 151.556/229.208/1414.520 ms, mdev 277.305 ms; end-to-end latency samples will include substantial link jitter.

kayg commented

2026-10-06 17:37:43 +00:00

Author

Owner

Dependency step complete at 3bb26c361e. Added auth and async WebSocket dependencies to the spike only; cargo check passed in 29.70 s using the required cache target. No product engine choice.

Dependency step complete at 3bb26c361eef57f39a2291dcd36f589c99a47f78. Added auth and async WebSocket dependencies to the spike only; cargo check passed in 29.70 s using the required cache target. No product engine choice.

kayg commented

2026-10-06 17:40:41 +00:00

Author

Owner

Zeroization dependency step complete at e9210eee39. is limited to byte buffers, which it can reliably wipe; Rust limbs do not expose a guaranteed wipe API, so that experiment-specific limitation is documented. cargo check passed in 0.50 s.

Zeroization dependency step complete at e9210eee3912e24168f2170ba41c5e1d2394add4. is limited to byte buffers, which it can reliably wipe; Rust limbs do not expose a guaranteed wipe API, so that experiment-specific limitation is documented. cargo check passed in 0.50 s.

kayg commented

2026-10-06 17:40:50 +00:00

Author

Owner

Correction: the byte-buffer wiping dependency is zeroize, and Rust BigUint limbs still have no guaranteed wipe API. The limitation is specific to this temporary spike implementation. The e9210ee dependency step passed cargo check in 0.50 s.

Correction: the byte-buffer wiping dependency is zeroize, and Rust BigUint limbs still have no guaranteed wipe API. The limitation is specific to this temporary spike implementation. The e9210ee dependency step passed cargo check in 0.50 s.

kayg commented

2026-10-06 17:44:58 +00:00

Author

Owner

Type-30 auth step complete at 3f80228e18. In spikes/vnc/src/apple_auth.rs, ECB type 30 authenticated against macOS 27.0 build 26A428 after replying RFB 003.008 to 003.889. Result: 5004x2652, server pixel format 32 bpp / 24 depth / little-endian / true-colour / max 255,255,255 / shifts 16,8,0; desktop name calternal. Offered types remained [30,33,36,31,32,35], type 2 absent. Unit tests 2/2 and clippy passed. Since ECB worked, CBC and fallback types were not attempted.

Type-30 auth step complete at 3f80228e18685969fd065d0112072e786255bb22. In spikes/vnc/src/apple_auth.rs, ECB type 30 authenticated against macOS 27.0 build 26A428 after replying RFB 003.008 to 003.889. Result: 5004x2652, server pixel format 32 bpp / 24 depth / little-endian / true-colour / max 255,255,255 / shifts 16,8,0; desktop name calternal. Offered types remained [30,33,36,31,32,35], type 2 absent. Unit tests 2/2 and clippy passed. Since ECB worked, CBC and fallback types were not attempted.

kayg commented

2026-10-06 17:59:12 +00:00

Author

Owner

Framebuffer finding, first probe: spikes/vnc/src/rfb.rs authenticated successfully and negotiated 32 bpp / 24 depth, but over 10 seconds received only four metadata rectangles: Apple DisplayGeometry 1105, Capabilities 1107, KeyboardLayout 1109 and MachineModel 1110. Total was 255 RFB bytes; no Raw raster rectangle arrived. Isolating the SetEncodings offer and initial-update sequence before treating this as a server limitation.

kayg commented

2026-10-06 18:07:21 +00:00

Author

Owner

Finding: type-30 auth reaches ServerInit, but a standard RFB probe receives only 4 Apple metadata rectangles (255 bytes, 120 bytes sent in 10 s) and no pixels. The reverse-engineered Apple flow requires ViewerInfo, encryption setup/rekey, then encrypted SetDisplayConfiguration/SetEncodings and AutoFrameBufferUpdate before raster delivery. Source: https://github.com/renegadelink/iShareScreen/blob/main/docs/apple_vnc_rfc.md. No VM settings were changed; implementing that bootstrap next.

kayg commented

2026-10-06 18:12:05 +00:00

Author

Owner

Step complete: added the Apple 0x44f rekey unwrap and authenticated directional CBC record codec in spikes/vnc/src/apple_record.rs. Two focused tests pass (valid two-record chain and corrupt-tag rejection). Commit 5f6b814. This is protocol research from the reverse-engineered reference, not an engine choice.

Step complete: added the Apple 0x44f rekey unwrap and authenticated directional CBC record codec in spikes/vnc/src/apple_record.rs. Two focused tests pass (valid two-record chain and corrupt-tag rejection). Commit 5f6b814. This is protocol research from the reverse-engineered reference, not an engine choice.

kayg commented

2026-10-06 18:12:31 +00:00

Author

Owner

Step complete: type-30 auth now returns its MD5-derived wrap key in zeroizing storage for the Apple 0x44f rekey path. The existing synthetic-peer ECB auth test still passes. Commit cbf0238.

Step complete: type-30 auth now returns its MD5-derived wrap key in zeroizing storage for the Apple 0x44f rekey path. The existing synthetic-peer ECB auth test still passes. Commit cbf0238.

kayg commented

2026-10-06 18:22:39 +00:00

Author

Owner

Finding: type-30 wrap-key derivation and Apple 0x44f rekey succeed (generation 1); record tags validate and encrypted client controls are accepted. After a static 0x1d descriptor, SetEncodings, non-incremental update request, and AutoFrameBufferUpdate, the VM sends 4 metadata-only rectangles (DesktopSize, 1107 VendorKeysym, 1109 KeyboardInputSource, 1110 DeviceInfo) and no pixel rectangle. Testing the physical-console path without a display descriptor next; no persistent VM settings changed.

kayg commented

2026-10-06 18:25:38 +00:00

Author

Owner

Finding: the initial FramebufferUpdateRequest builder was 8 bytes; RFB requires 10. After correcting it, the live VM responds with Zlib rectangles (encoding 6), proving the physical-console path is active. The parser stopped at the first Zlib rectangle; adding a persistent inflater before continuing. This also confirms Raw is not the server-selected encoding for the advertised preference list.

kayg commented

2026-10-06 18:29:41 +00:00

Author

Owner

Finding: the physical-console bootstrap reaches a complete 5004×2652 keyframe. ServerInit and client SetPixelFormat are both 32 bpp/24 depth, little-endian true-colour, maxima 255, shifts 16/8/0. First image was a Zlib (encoding 6) rectangle with 6,746,778 compressed bytes; macOS also sent DesktopSize (-223), VendorKeysym (1107), KeyboardInputSource (1109), and DeviceInfo (1110). It did not send Raw or CopyRect in this run. Startup metadata accounted for 7 preceding rectangles. Code now parses the persistent Zlib stream.

kayg commented

2026-10-06 18:37:44 +00:00

Author

Owner

Task 1 complete in 2e7c368; cargo test passes 9/9. Live join-to-first-image is 7.910 s with AutoFBU and 10.277 s with 20 Hz incremental polling. In the incremental run, macOS sent 11 updates and 7,346,490 RFB bytes during the 10 s idle window (734,649 B/s received, 37 B/s sent). It selected Zlib (6), RichCursor (-239), DesktopSize (-223), and emitted Apple 1107 VendorKeysym, 1109 KeyboardInputSource, 1110 DeviceInfo. 1105 DisplayLayout was offered but not sent. Pixel formats and the type-30 ECB success are recorded above.

Task 1 complete in 2e7c368; cargo test passes 9/9. Live join-to-first-image is 7.910 s with AutoFBU and 10.277 s with 20 Hz incremental polling. In the incremental run, macOS sent 11 updates and 7,346,490 RFB bytes during the 10 s idle window (734,649 B/s received, 37 B/s sent). It selected Zlib (6), RichCursor (-239), DesktopSize (-223), and emitted Apple 1107 VendorKeysym, 1109 KeyboardInputSource, 1110 DeviceInfo. 1105 DisplayLayout was offered but not sent. Pixel formats and the type-30 ECB success are recorded above.

kayg commented

2026-10-06 18:54:54 +00:00

Author

Owner

Finding: the browser subscriber decodes/draws the first JPEG rectangle and WebCodecs H.264 keyframe. For a first 4-event JPEG pointer probe, server-acknowledged pointer overlay samples were 353.2, 244.9, 269.8, and 1010.9 ms; RFB arrival dominated (343.9, 233.2, 254.6, 997.1 ms). The holder draws the pointer overlay only after a subsequent RFB update carries that input sequence, so this is not browser-side optimistic echo.

kayg referenced this issue

2026-10-06 19:06:15 +00:00

Status board: work in flight (kept current) #24

kayg commented

2026-10-06 19:10:54 +00:00

Author

Owner

Measurement finding: 200 unique input-to-canvas samples per browser encoding on infraternal-dev (8-core i9-9900K, 32 GiB) to VirtualMac2,1/macOS 27.0 (4 vCPU, 8 GiB), NetBird RTT 20/20 avg 152.111 ms. JPEG p50/p99 382.1/1031.9 ms; H.264 703.9/952.9 ms. Median stage times JPEG RFB/scale/encode/WS/decode/draw 360.2/10.5/1.0/0.6/1.6/0.1 ms; H.264 374.3/17.9/286.3/2.4/10.3/5.6 ms. H.264 currently starts one ffmpeg per frame. Capture JSONs are under the allowed scratch directory. The canvas change is a holder diagnostic pointer marker applied after the next RFB update carries the input sequence; macOS sent no PointerPos updates, so this is a server-round-trip proxy, not native cursor or camera-measured photon latency.

kayg commented

2026-10-06 19:31:19 +00:00

Author

Owner

Task 2 complete at 8426daa: holder retains one 5004x2652 macOS RFB session and serves loopback HTTP/WebSocket subscribers. On the rebuilt release holder, first RFB image arrived in 3.419 s; the TCP connection remained established after JPEG and H.264 subscribers disconnected. Idle sample: 4 updates/15 s and 37,024 RFB bytes received (2,468 B/s); a later 15 s post-disconnect sample had no framebuffer updates and the holder remained connected.

Task 2 complete at 8426daa: holder retains one 5004x2652 macOS RFB session and serves loopback HTTP/WebSocket subscribers. On the rebuilt release holder, first RFB image arrived in 3.419 s; the TCP connection remained established after JPEG and H.264 subscribers disconnected. Idle sample: 4 updates/15 s and 37,024 RFB bytes received (2,468 B/s); a later 15 s post-disconnect sample had no framebuffer updates and the holder remained connected.

kayg commented

2026-10-06 19:31:36 +00:00

Author

Owner

Task 3 complete at 2c3595e: Chromium decoded both WebCodecs H.264 and JPEG changed-rectangle subscribers to the 1280x800 canvas; pointer and keyboard messages route through the holder. The two 200-event captures are saved in the task scratch until report data is recorded: JPEG 382.1/1031.9 ms p50/p99; H.264 703.9/952.9 ms. No iPhone/Safari run was available.

Task 3 complete at 2c3595e: Chromium decoded both WebCodecs H.264 and JPEG changed-rectangle subscribers to the 1280x800 canvas; pointer and keyboard messages route through the holder. The two 200-event captures are saved in the task scratch until report data is recorded: JPEG 382.1/1031.9 ms p50/p99; H.264 703.9/952.9 ms. No iPhone/Safari run was available.

kayg commented

2026-10-06 19:32:56 +00:00

Author

Owner

Parser finding and fix at 965235c: a 1 MiB Zlib payload length for a 1x1 rectangle previously reached allocation before EOF; the regression assertion failed before the bound and passes after it. Compressed bodies are now capped at decoded rectangle size plus 64 KiB (and 256 MiB absolute cap). Added fuzz targets for type-30 auth, Apple records, and RFB messages; runs completed 443, 891,239, and 60,769 cases respectively with no crash. Cargo tests pass 10/10.

Parser finding and fix at 965235c: a 1 MiB Zlib payload length for a 1x1 rectangle previously reached allocation before EOF; the regression assertion failed before the bound and passes after it. Compressed bodies are now capped at decoded rectangle size plus 64 KiB (and 256 MiB absolute cap). Added fuzz targets for type-30 auth, Apple records, and RFB messages; runs completed 443, 891,239, and 60,769 cases respectively with no crash. Cargo tests pass 10/10.

kayg commented

2026-10-06 19:39:25 +00:00

Author

Owner

Task 5 and work complete at head a557cc1633. Report: research/vnc/SPIKE-B.md. Type 30 ECB succeeded; JPEG/H.264 each have 200 samples; parser regression and three fuzz targets are recorded. cargo test passed 10/10, release build and Chromium smoke passed. Required gate output verbatim: zsh:1: no such file or directory: scripts/verify. Holder and NetBird forward are stopped, ports are closed, and the job scratch directory is removed. No encoding or engine was selected; no iPhone run was available.

Task 5 and work complete at head a557cc16334852c83ad78995e18e11850d9ce8d7. Report: research/vnc/SPIKE-B.md. Type 30 ECB succeeded; JPEG/H.264 each have 200 samples; parser regression and three fuzz targets are recorded. cargo test passed 10/10, release build and Chromium smoke passed. Required gate output verbatim: zsh:1: no such file or directory: scripts/verify. Holder and NetBird forward are stopped, ports are closed, and the job scratch directory is removed. No encoding or engine was selected; no iPhone run was available.

kayg referenced this issue

2026-10-06 19:40:57 +00:00

Status board: work in flight (kept current) #24

kayg commented

2026-10-06 19:41:22 +00:00

Author

Owner

Starting phase C on branch spike-vnc at base SHA a557cc1633. Read all comments on #35 and parent #23; carrying forward phase A/B measurements and owner decisions 104–107. Working in the prescribed scratch directory; the shared macOS display configuration will remain untouched.

Starting phase C on branch spike-vnc at base SHA a557cc16334852c83ad78995e18e11850d9ce8d7. Read all comments on #35 and parent #23; carrying forward phase A/B measurements and owner decisions 104–107. Working in the prescribed scratch directory; the shared macOS display configuration will remain untouched.

kayg commented

2026-10-06 20:02:51 +00:00

Author

Owner

Task 1 finding: with a 1x1 nonincremental FBU request, 320 update replies in 25 s measured request-to-update p50/p99 46.267/129.407 ms; same-window ICMP ping averaged 153.148 ms (0% loss), so this RFB value is not a trustworthy symmetric-path RTT estimate. Four outstanding requests produced reply intervals 153.709/782.951 ms p50/p99 but queued request ages 614.866/938.921 ms; pipelining depth 4 worsened freshness. Apple 0x09 AutoFBU yielded only 3 full updates/10 s idle; standard ContinuousUpdates (150) and Fence (248) produced no reply/stream evidence. Continuing with encoding and scale measurements.

kayg commented

2026-10-06 20:04:18 +00:00

Author

Owner

Task 1 refined with a direct-path probe: 20 TCP connects through local SSH forward 127.0.0.1:15900 to the RFB banner measured p50 158.706 ms, mean 162.894 ms (min/max 153.372/237.899 ms); contemporaneous 20-packet NetBird ICMP ping averaged 154.747 ms (0% loss). This matches the network floor. The prior 46.267 ms FBU age is not a valid RTT because replies were not causally correlated one-for-one with the request timestamps. Depth-4 queue-age result remains evidence against deep pipelining.

kayg commented

2026-10-06 20:09:13 +00:00

Author

Owner

Task 2 completed; measured logs are in /home/kayg/.cache/codex-scratch/spike-vnc-c/runs/enc-*.log. At 5004x2652, macOS emitted ZRLE (~3,992,385 B/update; 31.6 ms full-rectangle inflate-only CPU), Apple 1000 (~128,241 B; 1.43–1.56 ms), Apple 1001 (~352,452 B; 4.05–5.90 ms), Apple 1002 (~1,716,205 B; 19.1–20.3 ms), and Apple 1011 (213,069–306,229 B full body; DCT decoder unavailable). The Apple CPU values exclude pixel reconstruction; 1011 handling-wall excludes actual DCT decode. Tight/Hextile/TRLE offers were previously answered with Raw (~53,082,444 B); CopyRect was not emitted on the static view. Candidate parser adds 2 bound tests; cargo test passes 12.

kayg commented

2026-10-06 20:10:19 +00:00

Author

Owner

Task 3 completed. In a 6 s repeat, ScaleFactor 0.5 ended at retained framebuffer 2502x1326 (ServerInit still says 5004x2652; VM display settings were untouched). Full Zlib rectangles: native 5,065,408 B p50 / 5,065,419 B p99 and decode+pixel CPU 67.004/69.129 ms; scaled 1,525,880/1,526,114 B and 24.546/29.054 ms. That's ~70% fewer wire bytes and ~63% lower median decode/conversion CPU. Logs: /home/kayg/.cache/codex-scratch/spike-vnc-c/runs/poll50.log and apple-scale-half.log.

kayg commented

2026-10-06 20:19:49 +00:00

Author

Owner

Task 4 completed in 4166e9a: one persistent ffmpeg/libx264 pipe emits an initial browser-decodable frame and then P-frames in a live Chromium smoke (10 successive input frames, no follow-up IDR). For those 10 1280x800 4:2:0 pointer-marker frames, encode p50/p99 10.648/12.960 ms and payload p50/p99 649/19,696 B; JPEG encode was 0.706/0.895 ms. The initial H.264 join image was 73,036 B. Full 200-event runs will use the post-input RFB probe because incremental pointer-only updates stopped after 104 events.

Task 4 completed in 4166e9a: one persistent ffmpeg/libx264 pipe emits an initial browser-decodable frame and then P-frames in a live Chromium smoke (10 successive input frames, no follow-up IDR). For those 10 1280x800 4:2:0 pointer-marker frames, encode p50/p99 10.648/12.960 ms and payload p50/p99 649/19,696 B; JPEG encode was 0.706/0.895 ms. The initial H.264 join image was 73,036 B. Full 200-event runs will use the post-input RFB probe because incremental pointer-only updates stopped after 104 events.

kayg commented

2026-10-06 20:22:04 +00:00

Author

Owner

Task 5 completed; artifacts: /home/kayg/.cache/codex-scratch/spike-vnc-c/runs/jpeg-200.json and h264-200.json (200/200 each, ~41 s each). Stages below are independent p50/p99 ms; H.264 uses persistent x264 P-frames after its join IDR. JPEG total 175.5/212.6; RFB 156.4/188.3; scale 9.9/26.6; encode 1.0/5.0; WS 0.8/10.8; browser decode 1.8/10.2; draw 0.0/0.2; payload 800/6,334 B. H.264 total 184.0/235.8; RFB 156.1/182.8; scale 6.3/20.2; encode 13.4/45.0; WS 0.6/5.7; decode 1.0/9.6; draw 4.7/12.2; payload 174/12,134 B. Each input was followed by a forced 1x1 FBU while the full incremental request remained active, so this reaches the RFB response floor but is not proof of OS input application/native cursor rendering. A pointer-only run without this probe stalled at 104 events. Terminal test typed then Backspace; each produced a canvas update (317/218 ms).

kayg commented

2026-10-06 20:22:13 +00:00

Author

Owner

Task 5 completed; artifacts: /home/kayg/.cache/codex-scratch/spike-vnc-c/runs/jpeg-200.json and h264-200.json (200/200 each, about 41 s each). Stages are independent p50/p99 ms; H.264 uses persistent x264 P-frames after its join IDR. JPEG total 175.5/212.6; RFB 156.4/188.3; scale 9.9/26.6; encode 1.0/5.0; WS 0.8/10.8; browser decode 1.8/10.2; draw 0.0/0.2; payload 800/6,334 B. H.264 total 184.0/235.8; RFB 156.1/182.8; scale 6.3/20.2; encode 13.4/45.0; WS 0.6/5.7; decode 1.0/9.6; draw 4.7/12.2; payload 174/12,134 B. Each input was followed by a forced 1x1 FBU while the full incremental request remained active, so this reaches the RFB response floor but is not proof of OS input application or native cursor rendering. A pointer-only run without this probe stalled at 104 events. Terminal test typed p then Backspace; each produced a canvas update (317/218 ms).

kayg commented

2026-10-06 20:23:53 +00:00

Author

Owner

CopyRect follow-up: offered CopyRect plus Raw for 30 s while producing Terminal output with jot 100 and paging through it. macOS sent Raw only (no CopyRect), including 1758x996 and 516x54 rectangles; the run received 116,765,900 B including two 53,082,444 B full rectangles. Evidence: /home/kayg/.cache/codex-scratch/spike-vnc-c/runs/enc-copyrect-scroll.log. This VM and quality tier did not use CopyRect for the tested scroll.

kayg commented

2026-10-06 20:27:27 +00:00

Author

Owner

Task 5 final-code follow-up at 861ffe9; adjacent 200-event runs are in jpeg-200-final.json and h264-200-final.json. JPEG total 172.5/246.0 ms p50/p99; stages RFB 150.4/184.7, scale 16.0/46.5, encode 1.9/11.1, WS 1.3/11.9, browser decode 2.2/12.0, draw 0.0/0.2; payload 799/5,919 B. H.264 total 189.2/269.1; stages RFB 156.4/198.9, scale 9.5/34.4, encode 16.3/41.1, WS 0.7/8.4, decode 1.3/8.9, draw 5.8/15.8; payload 174/12,046 B; all 200 follow-up frames were P-frames. Dev host infraternal-dev has 8 CPUs and load average 11.72/6.97/5.91 with concurrent Rust/C compilation; treat these as exploratory, not a performance-budget result. The 1x1 post-input request proxy limitation from the prior comment still applies.

Task 5 final-code follow-up at 861ffe9; adjacent 200-event runs are in jpeg-200-final.json and h264-200-final.json. JPEG total 172.5/246.0 ms p50/p99; stages RFB 150.4/184.7, scale 16.0/46.5, encode 1.9/11.1, WS 1.3/11.9, browser decode 2.2/12.0, draw 0.0/0.2; payload 799/5,919 B. H.264 total 189.2/269.1; stages RFB 156.4/198.9, scale 9.5/34.4, encode 16.3/41.1, WS 0.7/8.4, decode 1.3/8.9, draw 5.8/15.8; payload 174/12,046 B; all 200 follow-up frames were P-frames. Dev host infraternal-dev has 8 CPUs and load average 11.72/6.97/5.91 with concurrent Rust/C compilation; treat these as exploratory, not a performance-budget result. The 1x1 post-input request proxy limitation from the prior comment still applies.

kayg commented

2026-10-06 20:31:01 +00:00

Author

Owner

Task 6 complete: research/vnc/SPIKE-C.md records a 158.706 ms p50 path baseline and a rough 175.4 ms path-plus-frame floor. Final 200-event marker proxy was JPEG 172.5/246.0 ms and persistent H.264 189.2/269.1 ms p50/p99; the forced 1x1 response is not proof of macOS input-to-pixel latency. Nightly rfb_message fuzz run: 92,532 executions in 11 s, no crash. Owner questions and remaining real-pixel/camera measurement are in the report.

kayg commented

2026-10-06 20:32:26 +00:00

Author

Owner

Task 4 capacity addendum: during a short 40-event H.264 sample, 100 ms process sampling saw FFmpeg max RSS 75,812 KiB and 37.5% peak CPU; holder max RSS 69,020 KiB and 3.5% CPU. One H.264 process is owned per WebSocket subscriber; the measured desktop had one subscriber, so it was one encoder for that run. Sharing across subscribers and late-join IDR recovery remain open owner questions in SPIKE-C.md.

kayg commented

2026-10-06 20:33:10 +00:00

Author

Owner

Finished: phase C report committed at research/vnc/SPIKE-C.md. Head SHA: 3a10a17472. Validation: cargo test 12 passed; release holder build passed; nightly rfb_message fuzz completed 92,532 executions in 11 s without a crash; final JPEG/H.264 browser runs each completed 200/200 events. Holder and SSH forward stopped, Notes restored, scratch directory removed. scripts/verify fast output (verbatim): zsh:1: no such file or directory: scripts/verify

Finished: phase C report committed at research/vnc/SPIKE-C.md. Head SHA: 3a10a174723e96aa1ae0cbedfd41d575f62d6998. Validation: cargo test 12 passed; release holder build passed; nightly rfb_message fuzz completed 92,532 executions in 11 s without a crash; final JPEG/H.264 browser runs each completed 200/200 events. Holder and SSH forward stopped, Notes restored, scratch directory removed. scripts/verify fast output (verbatim): zsh:1: no such file or directory: scripts/verify

kayg commented

2026-10-06 20:34:26 +00:00

Author

Owner

Orchestrator note on phase C (spike-vnc 3a10a17, SPIKE-C.md):

Input-to-canvas p50/p99 went from 382/1032 to 172.5/246 ms (JPEG) and from 704/953 to 189.2/269 ms (H.264, persistent x264 at 16.3 ms encode p50).
The network accounts for almost all of the rest: the dev box (Hetzner, Germany) ↔ the macOS VM (an Indian ISP) is a direct P2P NetBird path with 152 ms RTT (ping 152.9 ms avg). The pipeline itself adds about 20–35 ms above the RTT plus one display frame.
Implication for the product (D104): the holder runs as the infraternal agent on the target's own LAN, so the user-perceived latency is phone ↔ holder (through the server's relay path) plus about 20–35 ms. Where the server and relay sit relative to the owner and the targets will dominate. This is an owner question.

Owner questions for the morning (from SPIKE-B/C):

the expected number of concurrent remote desktop sessions (for the memory and CPU budget);
whether a phone gets a scaled picture (1280×800) or native pixels;
JPEG rects vs H.264 (similar latency here; H.264 is smoother in motion, JPEG is lossless-ish at rest and cheaper per subscriber);
whether to share one encoder across subscribers.

Orchestrator note on phase C (spike-vnc 3a10a17, SPIKE-C.md): - Input-to-canvas p50/p99 went from 382/1032 to **172.5/246 ms** (JPEG) and from 704/953 to **189.2/269 ms** (H.264, persistent x264 at 16.3 ms encode p50). - **The network accounts for almost all of the rest:** the dev box (Hetzner, Germany) ↔ the macOS VM (an Indian ISP) is a direct P2P NetBird path with 152 ms RTT (ping 152.9 ms avg). The pipeline itself adds about 20–35 ms above the RTT plus one display frame. - **Implication for the product (D104):** the holder runs as the infraternal agent on the target's own LAN, so the user-perceived latency is phone ↔ holder (through the server's relay path) plus about 20–35 ms. Where the server and relay sit relative to the owner and the targets will dominate. This is an owner question. **Owner questions for the morning (from SPIKE-B/C):** - the expected number of concurrent remote desktop sessions (for the memory and CPU budget); - whether a phone gets a scaled picture (1280×800) or native pixels; - JPEG rects vs H.264 (similar latency here; H.264 is smoother in motion, JPEG is lossless-ish at rest and cheaper per subscriber); - whether to share one encoder across subscribers.

kayg referenced this issue

2026-10-06 20:34:26 +00:00

Status board: work in flight (kept current) #24

kayg referenced this issue from a commit

2026-10-10 09:20:34 +00:00

research(vnc): keep the S4 spike reports A–C on main (#35)

kayg referenced this issue from a commit

2026-10-10 09:21:12 +00:00

spike: add browser subscribers for JPEG and H.264

kayg referenced this issue from a commit

2026-10-10 09:21:12 +00:00

spike: bound Zlib rectangles and fuzz Apple parsers