Measure QUIC stacks: quinn+rustls+ring vs OpenSSL native QUIC #88
Open
opened 2026-10-08 05:28:16 +00:00 by kayg
·
16 comments
No Branch/Tag specified
main
j444b-d381-key-wins-flood
mesh-compat-382
j470b-server-mesh-rejoin
j461b-macos-utun-reader-thread
j475b-remove-host-motion
fix-reconnect-new-link
j481-d380
dev
j479-mesh-soak
fix/j476-host-facts
j475-motion-pill-split
j470-mac-node-join
root-upgrade-d376
j474-review-merge
latency-targets-120hz
nightly-suite-shards
gotatun-pin-j469
macos-utun-reader-thread
fix-flood-echo-stall
wip/j416
motion-fast-ease-out
mesh-addhost-j259
takeover-close-terminals-d368
kbd-instant-followups
fix-trust-key-double-click
nightly-suite-partial-report
appwin-icons-j207
rusqlite-slice1
swipe-content-first-d374
mesh-mtu-381
addhost-instant-open
glass-contrast-d365-mock
glass-contrast-d365
signout-followups-442
fix-startup-toast-red
research-gotatun-matrix-j317
themes-sync-j304
fix-load-flakes
fix-nix-zig-patch
fix-compute01-browser-baseline
ratchet-size-baseline-419
predict-auto-d130
revoke-coverage-d367
nightly-suite-queue-timeout
nightly-browser-suite
fix-rustsec-2026-0459
fonts-ioskeley-obel
log-retention-d375
fix-kbd-no-anim-427
fix-dev-red-specs-2
takeover-proof-305-j205
wip/j410-server-pump
ui-consistency-d372
fix-osc52-clipboard
libghostty-vt-1.0
forget-retired-host-d369
j404-merge-426
fix-setup-lock-race
fix-touch-select-426
revoke-access-report-ui
fix-predictive-echo-specs
fix-selinux-agent-exec
wip/j395-checkpoint2
wip/j395
j391-review-scratch
ui-consistency-j195-deferred-rebased
fix-rustc-paths-421
decisions-wording-341
fix-install-suite-red
fix-dev-rtt-specs
fix-pending-removal-align
fix-privileged-skip
fix-install-suite-404
review-j368
mesh-dns-j254
revoke-access-report-d356b
revoke-privilege-round5
fix-removal-confirm
review-j360
gotatun-rotation-367
deps-refresh-340
review-j354
review-J351
review-j350
addhost-takeover-d359
revoke-report-items-347
revoke-end-sessions-d362
review-j348
size-mesh-ssh-346
size-trim-337
revoke-access-diff-d356
j259-pre-rebase-b06c5a6
j305-ended-at-start-spec
addhost-sudo-consent
fix-layout-real-specs-411
fix-shell-open-411
fix-console-hosts-373
fix-shell-exit
fuzz-host-name-353
fix-host-details-410
macos-filter-j197
research-gotatun-tricks
revoke-p3-j216l
stale-state-pill
revoke-followups-j216k
revoke-progress-j216e
backup/macos01-20261106/lume/j197
backup/macos01-20261106/lume/appwin-icons-j207-stash0
backup/macos01-20261106/claude/stash-0
backup/macos01-20261106/claude/stash-1
backup/macos01-20261106/claude/J80-macos-s1
backup/macos01-20261106/claude/J80
backup/macos01-20261106/claude/J80-slice1-20261102
backup/macos01-20261106/claude/J16b-appwin-on-dev
backup/macos01-20261106/claude/appwin-launch-idle-146
backup/macos01-20261106/claude/J80-s1-run
wip/j259
trust-285-p3c
runtime-lock-j216f
mesh-ssh-p3-j175
tun-proof-p3b
trust-285-p3
terminal-polish-j271
tun-proof-p3
ssh-install-flakes-j216i
fuzz-fixes-j264
trust-285-a
trust-285-b
restart-oversized-fixture-j198
backup/macos01-20261106/lume/appwin-icons-j207
restart-bound-j216h
up-preflight-p3-j198
backup/macos01-20261106/lume/j251
up-preflight-j198
wip/up-preflight-j198-j245-p2
wip/up-preflight-j198-j245-p1
agent-fsync-offloop-j196
wip/up-preflight-j198-j245
settings-offscreen-j231
wip/up-preflight-j198-j226-inspection
wip/j103f
themes-j206
wip/up-preflight-j198-j226
flakes-j216c
backup/macos01-20261106/claude/j197
flakes-j216b
backup/macos01-20261106/claude/j197-red
wip/up-preflight-j198-review
flakes-j216
backup/macos01-20261106/claude/appwin-icons-j207
engineering-j208b
appwin-mock-j209
toast-stack-j165
engineering-j208
decisions-d351
decisions-d350
decisions-d349
appearance-presets-mock-j129
decisions-d348
docs-audit-spec
doc-coverage-baseline-raise
decisions-d347
decisions-d346b
decisions-d346
j88h-profile
ui-consistency-j195-deferred
ui-consistency-j195
engineering-practices-j194
preferred-login-ui-j166
signin-pools-263-j174
macos-node-265-s1
decisions-d345
decisions-d344
proxy-cgroup-j164
verify-quick-fuzz-329
proxy-start-io-j168
timing-flakes-j163
up-263-s1
mcp-events-j147
verify-quick-fuzz-j158
proxy-p3-j159
agent-resource-j157-detect
wip/j103d-final
remote-cancel-hang-j155
proxy-traefik-281-j62d
j149-verify-quick
proxy-traefik-281-j62b
j150-drop-unnamed
wip/j79d-red-r4
wip/j103d-d332
rootless-toast-j142
wip/j103d-head
wip/j103d
j125c-mcp
macos-build-j136
backup/macos01-20261106/claude/j136
wip/j79d-red-cli
wip/j79d-red-server
j121-link-liveness
j118c-tracer-failures
wip/j79c-head
release-desktop-j131
j124c-signed-macos
wip/j79c-red-p3
wip/j79c-red-p23
wip/j79c-red-p22
merge-j128-check
perf-deferred-j128
sidebar-toggle-260
j127-shortcut-changed-dot
j125-mcp
j118-tracer-failures
j126-account-summary
settings-sections-mock-j92
j120-pinch-lock
sidebar-292
backup/macos01-20261106/claude/j120
taste-2026-11-03
row-actions-242-215
backup/macos01-20261106/claude/stash-0-untracked
backup/macos01-20261106/claude/J80-265-s1-run
backup/macos01-20261106/claude/J80-macos-s1-20261102
j112-canvas-commands
ui-no-overshoot-scroll
fix-recovery-restart
backup/macos01-20261106/claude/stash-1-untracked
taste-2026-11-02
size-baseline-ratchet
machine-cards-structure-j108
j109-boot-copy-mockups
fix-dev-red
wip/up-263-s1
j95b-settings-rows
proxy-traefik-281
j95-option-rows
docs-human-j91
docs-keys-palette
fix-flood-1013-270-j38d
mesh-ssh-s1
mesh-ssh-s0
addhost-d291-tunnel
test-stability-load
mesh-relay-220
repro-296
shim-thp-off-24
mesh-pins-258-v2
mesh-dns
signin-d311-merged
docs-pages
ui-size-283
fix-ui-batch-specs
wip/j12b-edge-update
gate-on-arm
fix-rename-sync-293
trust-285-j60b
fix-focus-ring-clip
integ-ui-1101
edge-update-v2
signin-d311
taste-2026-11-01
layout-per-tab-250
ui-approved-splitopt-224
ui-approved-splitopt-224-on-dev
icon-text-centre-282
ui-approved-189-212-216
research-reverse-proxy
transfer-budget-320
review-mesh-pins-258-r2
sidebar-pill-259
review-mesh-relay-220
wip/j12-edge-update-v2
nix-package
status-crash-loop-237
fix-dev-test-failures
fix-fixture-socket-path
wip/mesh-relay-220-pre-dev-advance-rebase
fix-canvas-side-clip
research-head-to-head
frosted-real-256
review-mesh-pins-258
wip/mesh-relay-220-pre-final-rebase
review-layout-per-tab-250-r2
broad-suite-triage
nix-devshell
review-status-crash-loop-237-r3
wip/nix-devshell-pre-p3
taste-frosted-real
wip/code-serve-demo-2026-10-30
wip/code-serve-glass-2026-10-30
wip/code-serve-clusters-2026-10-30
wip/code-merge-release94-dev-2026-10-30
wip/rproxy-delta-scratch-2026-10-30
wip/stale-note-257-staged-2026-10-30
wip/d241-size-staged-2026-10-30
review-broad-suite-207
wip/mesh-pins-258-pre-split
wip/mesh-relay-220-pre-rebase
mesh-measure-220
plan-211-update
review-edge-autoupdate-r3
perf24-reconnect
review-stale-note-r2
release-prod-live
d241-shell-marker
review-mesh-pins-server-258
review-revocation-r3
review-setup-token-r2
review-sync-journal-r3
wip/review-mesh-pins-258-prune-review-1995fee46
frosted-scroll-256
sync-journal-r3
revocation-per-account-wip
review-tokio-asyncfd-0386
review-sync-p3s
wip/stash-8by64-stash0-2026-10-30
mesh-drag-r2
setup-token-data-wip
release-9-4
auth-busy-1013
mesh-view-overlap
sync-p3s
review-auth-busy-wake
review-readd-r8-wip
mesh-drag-bounds
audit-auth-93
app-windows-stream-start
review-authority-cap-r3
app-windows-r7-measure
readd-minimal-r6
readd-minimal-r5
app-windows-r7-soak
readd-minimal-r4
app-windows-r7
app-windows-r6
readd-minimal-r3
readd-minimal-r2
readd-minimal
merge-release8-dev
review-busy-page-liveness-r2
review-busy-page-liveness
mesh-view-trim
review-authority-read-retry-r2
review-remove-uninstall-r2
release-retry-reattach
release-retry
review-remove-uninstall
review-authority-read-retry
app-windows-ship
mesh-autoswitch-p2
mesh-ssh
mesh-relay
mesh-macos-node
screen-and-control
session-labels
review-real-logins-r5
review-real-logins-r4
review-real-logins-r3
shim-memory-r2
review-real-logins-r2
shim-memory
shadow-addhost-instant
shadow-real-logins
shadow-same-session-flood
shadow-prod-perf-fix
hosts-from-server
app-windows-real
app-windows-transport
release-prod-6w
mesh-policy-verify
quiet-close-toast
dev-merge-r4
review-mesh-agent-transport
review-app-windows-transport
settings-roundtrip
review-release-pipeline-r4
mesh-agent-transport
ended-strip-fix
review-audit-ca-update-d244
settings-rows
phone-width
signout-clear
fix-restart-tests
dev-merge-rp
signer-hardening
review-audit-ca-update
terminal-fidelity
review-audit-ca-update-wip
mesh-integrate
review-mesh-agent-transport-515d3712
review-mesh-dataplane-feec1988
mesh-dataplane
signout
research-canvas-node
wip/review-renderer-cost
wip/review-terminal-fidelity
padlock-glyph
review-release-pipeline-2e95b689
phone-polish
sidebar-polish
inbox-generic
review-mesh-dataplane-378d4055
prod-batch
ui-inbox
ui-ca-unlock
ui-machine-sections
review-mesh-dataplane-a5f744fb
tls-session-prod
session-bounds-prod
fix-dead-session
session-bounds
tls-session
palette-polish
shortcuts-d222
research-appmeta
phone-gutter
ui-centre-button
wip/ui-centre-button
wip/tls-session
review-addhost-breakit
ui-platform
renderer-cost
remote-agents
mesh-api
ssh-bootstrap
prod-hotfix2
prod-fixes
review-remote-agents
review-mesh-dataplane
session-life
research-browser
review-mesh-api
research-capture
review-mesh-api-round3
review-mesh-api-round1
gate-stability
review-gate-stability
review-relay-direct
review-mesh-api-findings
proto-sessions
font-picker
proto-addhost
proto-windows
mesh-core
latency-gate
research-bsd
research-discovery
keytray-mods
mock-core
ui-tweaks
fix-motion-sel
fix-motion
integrate2
fix-robust
t-web
t-server
vt-ghostty
fix-snap-echo
t-web-auth
review-vt2
t-agent
lane-protocol
lane-shim
lane-auth
s1-base
s1-contracts
s1-workspace
No results found.
No labels
launch-qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set
Reference
kayg/infraternal#88
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Owner (2026-10-08): measure which QUIC stack has the lower footprint and better performance, quinn+rustls+ring (current, D41/D173) or OpenSSL native QUIC (vendored OpenSSL 3.6; client and server), and whether rustls+ring has real benefits. Research only; the owner decides.
Started QUIC stack research on branch quic-research at base
5da658ec87(main; clean worktree). Read #88 (0 comments), the related transport history #78 and parent #5, plus CLAUDE.md, DECISIONS.md, docs/ARCHITECTURE.md and docs/PROCESS.md. Prior #78 measurements cover Quinn+rustls+ring and a slower rustls-OpenSSL provider; OpenSSL native QUIC was not measured. This lane will add research artifacts only under research/quic-stacks/.Step 1 finding (harness smoke paths), branch quic-research at base/head
5da658ec87: the standalone harness now passes mutual-TLS QUIC handshakes and a bidirectional echo for both Quinn+rustls+ring and OpenSSL 3.6.3 native QUIC. Both report ALPN quicbench, mutual_tls=true, bidirectional_stream=true. Sources are research/quic-stacks/harness/src/{test_identity.rs,quinn_stack.rs,openssl_native.rs}; no product code changed. OpenSSL native QUIC needed a thin local FFI for listener/connection APIs not exposed by openssl-sys.Finding: OpenSSL 3.6.3 native smoke passes mutual TLS, ALPN, and one bidirectional stream, but the four-stream harness is not yet measurable. After the server accepts stream ID 0,
SSL_get_accept_stream_queue_lenreports 3; subsequentSSL_accept_stream(..., SSL_ACCEPT_STREAM_NO_BLOCK)returns NULL and the transfer times out. Reproduction is inresearch/quic-stacks/harness/src/openssl_native.rs(bench-throughput openssl 4 524288). OpenSSL 3.6 docs say NO_BLOCK never blocks and NULL may still leave queued streams; investigating whether this is harness configuration or a 3.6.3 API defect before reporting it as a stack limitation.Update to the stream finding: setting
SSL_INCOMING_STREAM_POLICY_ACCEPTexplicitly after the handshake fixed the harness. The earlier four-stream stall was a harness configuration issue, not evidence of a native API defect. A single diagnostic run now accepts stream IDs 0, 4, 8, and 12 and transfers 524,288 bytes in 19,817 us (bench-throughput openssl 4 524288); this is a smoke result, not a five-repeat benchmark. The configured four-stream measurement is now viable.Size finding (static x86_64 musl, release/LTO, stripped; raw/research/quic-stacks/raw/size-binaries.csv): the OpenSSL TLS baseline is 5,190,400 B, Quinn candidate 6,987,368 B (+1,796,968 B), and OpenSSL-native candidate 5,192,096 B (+1,696 B). The OpenSSL delta is small because QUIC engine objects are already pulled into the baseline OpenSSL link; this is marginal size over the required baseline, not the total OpenSSL QUIC implementation size. The historical #78 Quinn delta used a different probe/dependency set and is not directly comparable. Full repeated performance and memory runs are still in progress.
Harness finding from the delayed Quinn handshake run: repetition 3 failed twice before one successful attempt. Both failure logs (
research/quic-stacks/raw/handshake-300ms-quinn-3-a1.logand-a2.log) say the server handshake was aborted by the peer; the client closed eachConnectionimmediately afterconnect()returned while the server was still inincoming.await(). The timing indicates a harness teardown race, so I will defer connection close until server acceptance is confirmed and rerun all five Quinn delayed handshakes. Failed logs remain as evidence and will be excluded from the reported numbers.Memory-probe finding: all three initial OpenSSL 1,000-connection attempts failed at
SSL_new(memory client)(raw/memory-1000-openssl-1-a1.logthrough-a3.log), before producing server RSS. The benchmark process had a 1,024 softRLIMIT_NOFILEwhile the harness opens one client UDP socket per connection; I infer descriptor pressure is the cause, but will confirm by rerunning with an 8,192 scope limit. These attempts are preserved and excluded from the comparison.The corrected OpenSSL memory probe now completes with 1,000 connections and one active stream per connection under an 8,192 soft file-descriptor limit. The one-off diagnostic measured 69,775 B idle and 6,447 B incremental stream RSS per connection (
research/quic-stacks/raw/diagnostics/memory-openssl-limit8192.csv); this is a harness check, not part of the final five-run summary. Client QUIC events now remain serviced while the server accepts streams.Corrected handshake/0-RTT step completed; final five-run CSVs are in
research/quic-stacks/raw/. Quinn fresh mTLS loopback median is 955 us per connection (five-run spread 948–1,020 us; median within-run p95 1,353 us) at 1,901 us CPU/connection. OpenSSL 3.6.3 is 4,883 us (4,605–6,296 us; p95 median 13,982 us) at 7,976 us CPU/connection. At 300 ms RTT, Quinn is 311,723 us (277,245–315,592 us) and OpenSSL is 886,331 us (867,227–888,265 us). Quinn's idempotent 0-RTT echo is 1,212 us on loopback and 603,050 us on the delayed path; OpenSSL native QUIC does not support 0-RTT. The measured mTLS server waits for authentication before acting on early data, so this is not an application-latency win under the current policy. Stream and memory repeats remain in progress.Five-run results are in
research/quic-stacks/REPORT.md; exact rows and qdisc logs are underresearch/quic-stacks/raw/.*OpenSSL's static TLS baseline already pulls QUIC implementation objects, so 1,696 B is marginal added size, not standalone engine size. Qdisc counters recorded Quinn loss at 0.16–0.26% and OpenSSL at 1.02–1.07% despite both being configured at 1%; impaired throughput is therefore not a matched-realized-loss comparison. Loopback throughput was comparable. Quinn's other measured benefits are faster handshake, lower connection RSS, explicit BBR selection, direct Tokio async integration, 0-RTT, and migration APIs. OpenSSL native QUIC lacks native 0-RTT and a public BBR selector; the tested OpenSSL 3.6.3 predates QUIC security fixes in 3.6.4/3.6.5.
Recommendation: retain Quinn + rustls + ring with BBR for now (moderate confidence); the image size advantage favors OpenSSL, but latency, memory, API fit, and observed delayed-path behavior favor Quinn. Repeat delayed throughput with matched deterministic loss and larger transfers before using it as a capacity estimate. No foundational decision changed.
Step complete: committed the reproducible static-musl comparison harness at
1371ebf(build + Quinn/OpenSSL mTLS smoke checks passed). It measures Quinn + rustls + ring and OpenSSL native QUIC against a TLS baseline; the full five-repeat matrix is now recorded and the report/raw-data commit is next.Step complete: report and raw data committed at
5c7d471. The five-run matrix shows 955 µs vs 4,883 µs loopback fresh mTLS, 43,978 B vs 69,767 B idle RSS per connection, and 0.515/0.536 vs 0.022 MiB/s on the configured delayed loss path. The report records that qdisc counters realized 0.16–0.26% loss for Quinn and 1.02–1.07% for OpenSSL, so the loss-path ratio is not a same-loss estimate. Full comparison table and files are inresearch/quic-stacks/REPORT.mdandresearch/quic-stacks/raw/.Verification note: the required command
scripts/verify fastcannot run in this checkout. Output verbatim:zsh:1: no such file or directory: scripts/verify.scripts/contains onlyperf-run, andmainhas the same contents. I left the repository gate unchanged because this job is restricted toresearch/quic-stacks/. Harness build/smoke, the five-repeat benchmark matrix,cargo fmt --check, shell syntax checks, andgit diff --checkpassed.Review fix complete:
00a9297gives each default benchmark run a fresh UTC timestamped output directory and refuses an existing explicit output path. The guard was exercised against the committed raw directory and exited 2 withrefusing to overwrite existing results directory: research/quic-stacks/raw; no measurements were run or overwritten. Reproduction instructions now describe the output behavior.Finished at HEAD
00a9297. The comparison table is posted above; report, harness, five-repeat raw numbers and logs are committed underresearch/quic-stacks/. Independent review completed with no findings after the rerun-output fix. Harness build, both real-stack mTLS smoke checks, the 80-invocation matrix,cargo fmt --check, shell syntax, output-directory guard, andgit diff --checkpassed.Required gate output verbatim:
zsh:1: no such file or directory: scripts/verify(exit 127).scripts/verifyis absent from this worktree andmain; I left the repository gate unchanged because this job permits writes only inresearch/quic-stacks/.Owner decision (2026-10-08): keep quinn + rustls + ring. Recorded as D176 on main. Follow-ups: rerun the delayed-path row with matched loss; measure BBR at 300 ms RTT after remote-agents lands (it owns the transport code).
kayg referenced this issue2026-10-28 17:53:56 +00:00