S1 lane: protocol codec (Rust + TS), fuzzing, golden vectors #15
Closed
opened 2026-10-06 10:10:48 +00:00 by kayg
·
19 comments
No Branch/Tag specified
main
j470b-server-mesh-rejoin
mesh-compat-382
j444b-d381-key-wins-flood
j461b-macos-utun-reader-thread
j475b-remove-host-motion
fix-reconnect-new-link
j481-d380
dev
j479-mesh-soak
fix/j476-host-facts
j475-motion-pill-split
j470-mac-node-join
root-upgrade-d376
j474-review-merge
latency-targets-120hz
nightly-suite-shards
gotatun-pin-j469
macos-utun-reader-thread
fix-flood-echo-stall
wip/j416
motion-fast-ease-out
mesh-addhost-j259
takeover-close-terminals-d368
kbd-instant-followups
fix-trust-key-double-click
nightly-suite-partial-report
appwin-icons-j207
rusqlite-slice1
swipe-content-first-d374
mesh-mtu-381
addhost-instant-open
glass-contrast-d365-mock
glass-contrast-d365
signout-followups-442
fix-startup-toast-red
research-gotatun-matrix-j317
themes-sync-j304
fix-load-flakes
fix-nix-zig-patch
fix-compute01-browser-baseline
ratchet-size-baseline-419
predict-auto-d130
revoke-coverage-d367
nightly-suite-queue-timeout
nightly-browser-suite
fix-rustsec-2026-0459
fonts-ioskeley-obel
log-retention-d375
fix-kbd-no-anim-427
fix-dev-red-specs-2
takeover-proof-305-j205
wip/j410-server-pump
ui-consistency-d372
fix-osc52-clipboard
libghostty-vt-1.0
forget-retired-host-d369
j404-merge-426
fix-setup-lock-race
fix-touch-select-426
revoke-access-report-ui
fix-predictive-echo-specs
fix-selinux-agent-exec
wip/j395-checkpoint2
wip/j395
j391-review-scratch
ui-consistency-j195-deferred-rebased
fix-rustc-paths-421
decisions-wording-341
fix-install-suite-red
fix-dev-rtt-specs
fix-pending-removal-align
fix-privileged-skip
fix-install-suite-404
review-j368
mesh-dns-j254
revoke-access-report-d356b
revoke-privilege-round5
fix-removal-confirm
review-j360
gotatun-rotation-367
deps-refresh-340
review-j354
review-J351
review-j350
addhost-takeover-d359
revoke-report-items-347
revoke-end-sessions-d362
review-j348
size-mesh-ssh-346
size-trim-337
revoke-access-diff-d356
j259-pre-rebase-b06c5a6
j305-ended-at-start-spec
addhost-sudo-consent
fix-layout-real-specs-411
fix-shell-open-411
fix-console-hosts-373
fix-shell-exit
fuzz-host-name-353
fix-host-details-410
macos-filter-j197
research-gotatun-tricks
revoke-p3-j216l
stale-state-pill
revoke-followups-j216k
revoke-progress-j216e
backup/macos01-20261106/lume/j197
backup/macos01-20261106/lume/appwin-icons-j207-stash0
backup/macos01-20261106/claude/stash-0
backup/macos01-20261106/claude/stash-1
backup/macos01-20261106/claude/J80-macos-s1
backup/macos01-20261106/claude/J80
backup/macos01-20261106/claude/J80-slice1-20261102
backup/macos01-20261106/claude/J16b-appwin-on-dev
backup/macos01-20261106/claude/appwin-launch-idle-146
backup/macos01-20261106/claude/J80-s1-run
wip/j259
trust-285-p3c
runtime-lock-j216f
mesh-ssh-p3-j175
tun-proof-p3b
trust-285-p3
terminal-polish-j271
tun-proof-p3
ssh-install-flakes-j216i
fuzz-fixes-j264
trust-285-a
trust-285-b
restart-oversized-fixture-j198
backup/macos01-20261106/lume/appwin-icons-j207
restart-bound-j216h
up-preflight-p3-j198
backup/macos01-20261106/lume/j251
up-preflight-j198
wip/up-preflight-j198-j245-p2
wip/up-preflight-j198-j245-p1
agent-fsync-offloop-j196
wip/up-preflight-j198-j245
settings-offscreen-j231
wip/up-preflight-j198-j226-inspection
wip/j103f
themes-j206
wip/up-preflight-j198-j226
flakes-j216c
backup/macos01-20261106/claude/j197
flakes-j216b
backup/macos01-20261106/claude/j197-red
wip/up-preflight-j198-review
flakes-j216
backup/macos01-20261106/claude/appwin-icons-j207
engineering-j208b
appwin-mock-j209
toast-stack-j165
engineering-j208
decisions-d351
decisions-d350
decisions-d349
appearance-presets-mock-j129
decisions-d348
docs-audit-spec
doc-coverage-baseline-raise
decisions-d347
decisions-d346b
decisions-d346
j88h-profile
ui-consistency-j195-deferred
ui-consistency-j195
engineering-practices-j194
preferred-login-ui-j166
signin-pools-263-j174
macos-node-265-s1
decisions-d345
decisions-d344
proxy-cgroup-j164
verify-quick-fuzz-329
proxy-start-io-j168
timing-flakes-j163
up-263-s1
mcp-events-j147
verify-quick-fuzz-j158
proxy-p3-j159
agent-resource-j157-detect
wip/j103d-final
remote-cancel-hang-j155
proxy-traefik-281-j62d
j149-verify-quick
proxy-traefik-281-j62b
j150-drop-unnamed
wip/j79d-red-r4
wip/j103d-d332
rootless-toast-j142
wip/j103d-head
wip/j103d
j125c-mcp
macos-build-j136
backup/macos01-20261106/claude/j136
wip/j79d-red-cli
wip/j79d-red-server
j121-link-liveness
j118c-tracer-failures
wip/j79c-head
release-desktop-j131
j124c-signed-macos
wip/j79c-red-p3
wip/j79c-red-p23
wip/j79c-red-p22
merge-j128-check
perf-deferred-j128
sidebar-toggle-260
j127-shortcut-changed-dot
j125-mcp
j118-tracer-failures
j126-account-summary
settings-sections-mock-j92
j120-pinch-lock
sidebar-292
backup/macos01-20261106/claude/j120
taste-2026-11-03
row-actions-242-215
backup/macos01-20261106/claude/stash-0-untracked
backup/macos01-20261106/claude/J80-265-s1-run
backup/macos01-20261106/claude/J80-macos-s1-20261102
j112-canvas-commands
ui-no-overshoot-scroll
fix-recovery-restart
backup/macos01-20261106/claude/stash-1-untracked
taste-2026-11-02
size-baseline-ratchet
machine-cards-structure-j108
j109-boot-copy-mockups
fix-dev-red
wip/up-263-s1
j95b-settings-rows
proxy-traefik-281
j95-option-rows
docs-human-j91
docs-keys-palette
fix-flood-1013-270-j38d
mesh-ssh-s1
mesh-ssh-s0
addhost-d291-tunnel
test-stability-load
mesh-relay-220
repro-296
shim-thp-off-24
mesh-pins-258-v2
mesh-dns
signin-d311-merged
docs-pages
ui-size-283
fix-ui-batch-specs
wip/j12b-edge-update
gate-on-arm
fix-rename-sync-293
trust-285-j60b
fix-focus-ring-clip
integ-ui-1101
edge-update-v2
signin-d311
taste-2026-11-01
layout-per-tab-250
ui-approved-splitopt-224
ui-approved-splitopt-224-on-dev
icon-text-centre-282
ui-approved-189-212-216
research-reverse-proxy
transfer-budget-320
review-mesh-pins-258-r2
sidebar-pill-259
review-mesh-relay-220
wip/j12-edge-update-v2
nix-package
status-crash-loop-237
fix-dev-test-failures
fix-fixture-socket-path
wip/mesh-relay-220-pre-dev-advance-rebase
fix-canvas-side-clip
research-head-to-head
frosted-real-256
review-mesh-pins-258
wip/mesh-relay-220-pre-final-rebase
review-layout-per-tab-250-r2
broad-suite-triage
nix-devshell
review-status-crash-loop-237-r3
wip/nix-devshell-pre-p3
taste-frosted-real
wip/code-serve-demo-2026-10-30
wip/code-serve-glass-2026-10-30
wip/code-serve-clusters-2026-10-30
wip/code-merge-release94-dev-2026-10-30
wip/rproxy-delta-scratch-2026-10-30
wip/stale-note-257-staged-2026-10-30
wip/d241-size-staged-2026-10-30
review-broad-suite-207
wip/mesh-pins-258-pre-split
wip/mesh-relay-220-pre-rebase
mesh-measure-220
plan-211-update
review-edge-autoupdate-r3
perf24-reconnect
review-stale-note-r2
release-prod-live
d241-shell-marker
review-mesh-pins-server-258
review-revocation-r3
review-setup-token-r2
review-sync-journal-r3
wip/review-mesh-pins-258-prune-review-1995fee46
frosted-scroll-256
sync-journal-r3
revocation-per-account-wip
review-tokio-asyncfd-0386
review-sync-p3s
wip/stash-8by64-stash0-2026-10-30
mesh-drag-r2
setup-token-data-wip
release-9-4
auth-busy-1013
mesh-view-overlap
sync-p3s
review-auth-busy-wake
review-readd-r8-wip
mesh-drag-bounds
audit-auth-93
app-windows-stream-start
review-authority-cap-r3
app-windows-r7-measure
readd-minimal-r6
readd-minimal-r5
app-windows-r7-soak
readd-minimal-r4
app-windows-r7
app-windows-r6
readd-minimal-r3
readd-minimal-r2
readd-minimal
merge-release8-dev
review-busy-page-liveness-r2
review-busy-page-liveness
mesh-view-trim
review-authority-read-retry-r2
review-remove-uninstall-r2
release-retry-reattach
release-retry
review-remove-uninstall
review-authority-read-retry
app-windows-ship
mesh-autoswitch-p2
mesh-ssh
mesh-relay
mesh-macos-node
screen-and-control
session-labels
review-real-logins-r5
review-real-logins-r4
review-real-logins-r3
shim-memory-r2
review-real-logins-r2
shim-memory
shadow-addhost-instant
shadow-real-logins
shadow-same-session-flood
shadow-prod-perf-fix
hosts-from-server
app-windows-real
app-windows-transport
release-prod-6w
mesh-policy-verify
quiet-close-toast
dev-merge-r4
review-mesh-agent-transport
review-app-windows-transport
settings-roundtrip
review-release-pipeline-r4
mesh-agent-transport
ended-strip-fix
review-audit-ca-update-d244
settings-rows
phone-width
signout-clear
fix-restart-tests
dev-merge-rp
signer-hardening
review-audit-ca-update
terminal-fidelity
review-audit-ca-update-wip
mesh-integrate
review-mesh-agent-transport-515d3712
review-mesh-dataplane-feec1988
mesh-dataplane
signout
research-canvas-node
wip/review-renderer-cost
wip/review-terminal-fidelity
padlock-glyph
review-release-pipeline-2e95b689
phone-polish
sidebar-polish
inbox-generic
review-mesh-dataplane-378d4055
prod-batch
ui-inbox
ui-ca-unlock
ui-machine-sections
review-mesh-dataplane-a5f744fb
tls-session-prod
session-bounds-prod
fix-dead-session
session-bounds
tls-session
palette-polish
shortcuts-d222
research-appmeta
phone-gutter
ui-centre-button
wip/ui-centre-button
wip/tls-session
review-addhost-breakit
ui-platform
renderer-cost
remote-agents
mesh-api
ssh-bootstrap
prod-hotfix2
prod-fixes
review-remote-agents
review-mesh-dataplane
session-life
research-browser
review-mesh-api
research-capture
review-mesh-api-round3
review-mesh-api-round1
gate-stability
review-gate-stability
review-relay-direct
review-mesh-api-findings
proto-sessions
font-picker
proto-addhost
proto-windows
mesh-core
latency-gate
research-bsd
research-discovery
keytray-mods
mock-core
ui-tweaks
fix-motion-sel
fix-motion
integrate2
fix-robust
t-web
t-server
vt-ghostty
fix-snap-echo
t-web-auth
review-vt2
t-agent
lane-protocol
lane-shim
lane-auth
s1-base
s1-contracts
s1-workspace
No results found.
No labels
launch-qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set
Reference
kayg/infraternal#15
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Implement
contracts/wire.md§2–§4 ininfraternal-protocol(Rust) andpackages/protocol(TS), with shared golden test vectors, the fuzz targetwire_frame_decode, and Criterion and JS benchmarks against the #12 numbers. Base:s1-base386acfd. Part of #9.Starting protocol codec work on branch lane-protocol, based on
386acfd3dc. Finding: contracts/wire.md §2-§3 says LEB128 varint but does not define maximum width or whether non-minimal encodings are valid. Proposed implementation profile: unsigned u32 LEB128, shortest-form only; frame lengths are bounded to 4 MiB before any allocation, and stream IDs / nested string lengths use the same u32 representation. I will implement this consistently in Rust and TypeScript and flag the added constraint in the result.Finding: while implementing vectors, the initial Close encoder incorrectly added a varint reason length. contracts/wire.md §4 specifies u16 code followed by UTF-8 reason to the end of the body; the §2 length-prefixed string rule applies only where the table says varint+utf8. The shared golden test exposed the extra 0x04 byte in close.bin. Fixed Rust and TypeScript encoders and regenerated contracts/vectors/wire/close.bin; bun test and the Rust protocol tests now pass all 17 frame vectors.
Performance finding (measured in crates/infraternal-protocol/benches/wire_codec.rs): Criterion used the spike's 20 samples, 200 ms warmup and 500 ms measurement. Sequence/payload summary medians were 10.635 ns for 1-byte Input and 10.952 ns for 64-byte Output; full typed-frame medians were 14.378 ns and 15.275 ns. The #12 spike reported 4.63 ns and 4.95 ns for its simpler four-frame decoder with fixed-width length framing. The full §4 typed decoder is slower, and I'm recording both summary and full-path results while completing the requested fuzz and checks.
Finding from review: crates/infraternal-protocol/src/lib.rs::StreamDecoder::buffer_bytes currently uses try_reserve_exact for each received chunk. The streaming harness feeds 3-byte chunks (fuzz/fuzz_targets/wire_stream_decode.rs); for a 4 KiB partial frame, exact growth can request about 1,365 reallocations and copy roughly 2.8 MiB cumulatively. I am changing growth to geometric capacity bounded by the validated frame length, then restarting the streaming fuzz run against that implementation.
Finding: Bun 1.4.2's default TextDecoder strips an initial UTF-8 BOM () and returns , while Rust std::str::from_utf8 preserves it as U+FEFF. That would make the same Hello.client_info bytes decode differently across clients. I will set ignoreBOM: true in packages/protocol/src/codec.ts and change the shared Hello vector to begin with U+FEFF so both implementations prove parity.
Finding: Bun 1.4.2 default TextDecoder strips the initial UTF-8 BOM bytes EF BB BF and returns only the following A, while Rust std::str::from_utf8 preserves U+FEFF. The same Hello.client_info bytes would decode differently across clients. I will set ignoreBOM true in packages/protocol/src/codec.ts and change the shared Hello vector to begin with U+FEFF so both implementations prove parity.
Fuzz evidence on final Rust source: wire_frame_decode completed 601 s with 323,552,539 executions (538,356 exec/s average, 535 MiB peak RSS); wire_stream_decode completed 601 s with 13,342,465 executions (22,200 exec/s average, 510 MiB peak RSS). Neither target found a crash. The streaming run exercised the geometric bounded-growth buffer implementation.
Benchmark evidence: the lower-contention Criterion run (same 20 samples, 200 ms warmup, 500 ms measure as #12 on the i9-9900K) measured Rust sequence/payload decode at 10.635 ns for Input/1 B and 10.952 ns for Output/64 B; full typed decode at 14.378 ns and 15.275 ns; reusable encode at 14.972 ns and 10.043 ns. Compared with #12 specialized decode at 4.63 ns and 4.95 ns, the new full codec costs about 3.1x and 3.1x (summary-only decode about 2.3x and 2.2x), while remaining about 5x faster than the #12 prost comparison. Bun medians were 232.8 ns and 237.5 ns (about 4.04M exec/s), versus #12 Chrome 153 at 287.1 ns and 283.3 ns; runtime comparison is not apples-to-apples. A later retry was discarded as host-contended: mpstat showed 89% average CPU busy across all cores during concurrent OpenSSL builds, and Criterion intervals widened to 24.7–47.3 ns for Input encode and 35.8–46.0 ns for Input summary decode.
Finished S1 lane protocol codec on branch lane-protocol. Head: 407057272ce0e33c8b9e6dfed10a03c719887e77.
Both final-source cargo-fuzz runs completed 601 seconds without crashes: wire_frame_decode 323,552,539 executions (538,356/s average, 535 MiB peak RSS); wire_stream_decode 13,342,465 executions (22,200/s average, 510 MiB peak RSS).
Benchmark results and the host-contention caveat are in the preceding issue comment. Bun and Rust golden tests pass against the same 17-frame manifest. The contract clarifies the previously unspecified varint rule as shortest-form unsigned u32 LEB128.
Required gate command exited 0. Gate output (verbatim):
Independent read-only review started on lane-protocol, base
386acfd3dc, head 407057272ce0e33c8b9e6dfed10a03c719887e77. Reviewing safety, Rust/TypeScript parity, fuzzing, performance, API and history.P1 blocker reproduced in both runtimes: incremental mux header refill copies up to 10 bytes without respecting the newly completed frame boundary, then clears the entire buffer (Rust src/lib.rs around 489-531; TS src/codec.ts around 213-242). Feed hex 0005140100000000030f0000 in chunks [first byte, remainder]: only Credit is delivered, then Rust EmptyFrame / TS empty-frame. Two valid Close envelopes 00030f000000030f0000 split after byte 1 silently deliver only one callback. Add all-split multi-frame equivalence tests; consume only the header or preserve buffered surplus.
Review evidence: both Rust and TS fail 2/769 split-position cases over repeated golden vectors (close.bin split 1: one callback and 8 retained bytes; credit.bin split 1: one callback then EmptyFrame). Rust checks all 359 golden-vector truncations as Incomplete. Direct decoder differential: 24,504 deterministic mutated/golden/varint/mux cases + 3,450 cases from 119 frame and 1,606 stream corpus files, zero divergences (accepted 9,126 and 2,002 respectively); outputs and consumed lengths compared via re-encoding. Replay fuzz coverage: frame cov 280 / 963 counters, stream cov 472 / 1347 counters; no crash, but no semantic oracle and fixed 3-byte chunks do not exercise the found refill over-read. Evidence target/review--summary.json and target/review-fuzz-.log. History blocker: git log s1-base..lane-protocol contains only 4070572, violating the requested logical-step commit split. Fast gate passes; review verdict pending final report.
Independent read-only review finished. Verdict: MERGE AFTER FIXES.
Head: 407057272ce0e33c8b9e6dfed10a03c719887e77
Blockers:
Evidence: direct differential 24,504 generated cases plus 3,450 existing fuzz-corpus mode/case pairs, zero Rust/TS divergences; all 359 golden-vector truncations incomplete. Fuzz replay frame cov 280/963 counters and stream cov 472/1347, no crash; fixed 3-byte chunks and absent semantic assertions miss the data-loss bug. Full Rust decode central 12.040 ns input / 11.668 ns 64B output, approximately 0.18% one-core CPU at 10 MB/s with 64B chunks, not an end-to-end latency result. Borrowed direct decode works over Bytes, but internal-Vec streaming callbacks cannot transfer retained payload ownership zero-copy to async tasks. Strict TS compilation separately passed; add it to fast gate.
Findings appended to /home/kayg/.local/state/codex-jobs/infraternal/review-protocol/findings.md. Repro/data/bench/fuzz/gate artifacts are under the lane worktree target/review-*. Tracked worktree remains clean. No code edits, commits, pushes or issue closure.
scripts/verify fast exited 0; complete output verbatim:
Starting independent-review fixes on lane-protocol, original head 407057272ce0e33c8b9e6dfed10a03c719887e77, base s1-base
386acfd3dc. Read CLAUDE.md, DECISIONS.md, docs/ARCHITECTURE.md and all findings. Will first demonstrate split-header data loss in both runtimes, add stream differential/oracle coverage, strengthen fuzzing, and tag/rewrite the lane as gated atomic steps. No push.Confirmed P1 with failing tests before edits: target/regression-rust-before.log reports left 1/right 2 callbacks; target/regression-ts-before.log reports the missing second Close. Fixed both header refill paths to recheck length after each header byte, retaining the direct fast path. packages/protocol/src/stream.test.ts plus crates/infraternal-protocol/examples/stream_oracle.rs compare scalar fields, payloads, order, re-encoding and final buffered state: 134 streams, 11,439 identical schedules, zero divergences. Every boundary, empty chunks, all 1–5-byte mux widths, all 17 vectors, minimum Close/Credit and seeded random multi-splits included. Added deterministic 4 MiB/oversized tests; strengthening fuzz oracles before timed runs.
Review follow-ups: added strict TypeScript@5.9.3 compiler invocation to scripts/verify.d/ts.sh (Bun transpilation alone did not enforce decision 40). Documented incremental callback/copy/async retention limitations in both decoder APIs; borrowed direct decoders remain available for owning transport buffers. contracts/wire.md now gives redundant/overflow varint examples and requires consumer checked u64 exclusive sequence ends; Snapshot length does not advance out_seq. Connection integration must still enforce Hello ordering, stream ownership, direction/version/caps, JSON authorization, admission/reclamation and frozen upgrade layouts before implementation; this lane makes no auth or end-to-end budget claim. Benchmark evidence target/bench-before.log and target/bench-after.log: mux split_header id=0 central 66.419 -> 64.585 ns (overlapping intervals); id=u32::MAX 75.747 -> 108.05 ns (~32.3 ns for repeated header checks). Correctness justifies that bounded fragmented-path cost; direct full typed Output 12.294 -> 12.936 ns with overlapping intervals. Benchmarks ran locally alongside fuzzing, not on the pinned perf VM. Added mux/fragmentation cases to the Rust benchmark rather than weakening strict validation or refactoring enums without end-to-end evidence.
Implementation complete on lane-protocol, head
6ac06c7f75; base s1-base386acfd3dc. Original reviewed head preserved as lane-protocol-v1 (407057272ce0e33c8b9e6dfed10a03c719887e77). No push/deploy; issue remains open.Review disposition:
nix shell nixpkgs#binutils nixpkgs#file -c scripts/verify faston its own tree. Logs target/atomic-gate-01.log through target/atomic-gate-11.log. Before TS integration, relevant TS steps additionally ran compiler/tests. Every commit body records choice/rejected alternatives/decisions and ends Refs: #15.Atomic commits:
Final-tree fast gate (exit 0), verbatim:
Timed post-fix fuzz results (both exit 0, max_len=65536):
These are libFuzzer edge/feature counts, not source-coverage percentages. Summary: target/fuzz-fixed-summary.json. Worktree clean at final head; generated corpora/artifacts are ignored. All review findings answered above; no issue closure.
Merged into
devas7174d6b. Gate on the merge result: all PASS, TS 7/7.Closed by the orchestrator during the 2026-10-28 handover (#254), on the owner's request to close finished issues. Evidence: Codec lane merged to dev as
7174d6b(verified ancestor of dev); P1 split-header finding fixed and fuzz clean. Reopen if anything is still missing.kayg referenced this issue2026-10-28 17:53:54 +00:00