Measure QUIC stacks: quinn+rustls+ring vs OpenSSL native QUIC #88

Open
opened 2026-10-08 05:28:16 +00:00 by kayg · 16 comments
Owner

Owner (2026-10-08): measure which QUIC stack has the lower footprint and better performance, quinn+rustls+ring (current, D41/D173) or OpenSSL native QUIC (vendored OpenSSL 3.6; client and server), and whether rustls+ring has real benefits. Research only; the owner decides.

Owner (2026-10-08): measure which QUIC stack has the lower footprint and better performance, quinn+rustls+ring (current, D41/D173) or OpenSSL native QUIC (vendored OpenSSL 3.6; client and server), and whether rustls+ring has real benefits. Research only; the owner decides.
Author
Owner

Started QUIC stack research on branch quic-research at base 5da658ec87 (main; clean worktree). Read #88 (0 comments), the related transport history #78 and parent #5, plus CLAUDE.md, DECISIONS.md, docs/ARCHITECTURE.md and docs/PROCESS.md. Prior #78 measurements cover Quinn+rustls+ring and a slower rustls-OpenSSL provider; OpenSSL native QUIC was not measured. This lane will add research artifacts only under research/quic-stacks/.

Started QUIC stack research on branch quic-research at base 5da658ec87cd651fa4b30a9d17592383c9b9d6d9 (main; clean worktree). Read #88 (0 comments), the related transport history #78 and parent #5, plus CLAUDE.md, DECISIONS.md, docs/ARCHITECTURE.md and docs/PROCESS.md. Prior #78 measurements cover Quinn+rustls+ring and a slower rustls-OpenSSL provider; OpenSSL native QUIC was not measured. This lane will add research artifacts only under research/quic-stacks/.
Author
Owner

Step 1 finding (harness smoke paths), branch quic-research at base/head 5da658ec87: the standalone harness now passes mutual-TLS QUIC handshakes and a bidirectional echo for both Quinn+rustls+ring and OpenSSL 3.6.3 native QUIC. Both report ALPN quicbench, mutual_tls=true, bidirectional_stream=true. Sources are research/quic-stacks/harness/src/{test_identity.rs,quinn_stack.rs,openssl_native.rs}; no product code changed. OpenSSL native QUIC needed a thin local FFI for listener/connection APIs not exposed by openssl-sys.

Step 1 finding (harness smoke paths), branch quic-research at base/head 5da658ec87cd651fa4b30a9d17592383c9b9d6d9: the standalone harness now passes mutual-TLS QUIC handshakes and a bidirectional echo for both Quinn+rustls+ring and OpenSSL 3.6.3 native QUIC. Both report ALPN quicbench, mutual_tls=true, bidirectional_stream=true. Sources are research/quic-stacks/harness/src/{test_identity.rs,quinn_stack.rs,openssl_native.rs}; no product code changed. OpenSSL native QUIC needed a thin local FFI for listener/connection APIs not exposed by openssl-sys.
Author
Owner

Finding: OpenSSL 3.6.3 native smoke passes mutual TLS, ALPN, and one bidirectional stream, but the four-stream harness is not yet measurable. After the server accepts stream ID 0, SSL_get_accept_stream_queue_len reports 3; subsequent SSL_accept_stream(..., SSL_ACCEPT_STREAM_NO_BLOCK) returns NULL and the transfer times out. Reproduction is in research/quic-stacks/harness/src/openssl_native.rs (bench-throughput openssl 4 524288). OpenSSL 3.6 docs say NO_BLOCK never blocks and NULL may still leave queued streams; investigating whether this is harness configuration or a 3.6.3 API defect before reporting it as a stack limitation.

Finding: OpenSSL 3.6.3 native smoke passes mutual TLS, ALPN, and one bidirectional stream, but the four-stream harness is not yet measurable. After the server accepts stream ID 0, `SSL_get_accept_stream_queue_len` reports 3; subsequent `SSL_accept_stream(..., SSL_ACCEPT_STREAM_NO_BLOCK)` returns NULL and the transfer times out. Reproduction is in `research/quic-stacks/harness/src/openssl_native.rs` (`bench-throughput openssl 4 524288`). OpenSSL 3.6 docs say NO_BLOCK never blocks and NULL may still leave queued streams; investigating whether this is harness configuration or a 3.6.3 API defect before reporting it as a stack limitation.
Author
Owner

Update to the stream finding: setting SSL_INCOMING_STREAM_POLICY_ACCEPT explicitly after the handshake fixed the harness. The earlier four-stream stall was a harness configuration issue, not evidence of a native API defect. A single diagnostic run now accepts stream IDs 0, 4, 8, and 12 and transfers 524,288 bytes in 19,817 us (bench-throughput openssl 4 524288); this is a smoke result, not a five-repeat benchmark. The configured four-stream measurement is now viable.

Update to the stream finding: setting `SSL_INCOMING_STREAM_POLICY_ACCEPT` explicitly after the handshake fixed the harness. The earlier four-stream stall was a harness configuration issue, not evidence of a native API defect. A single diagnostic run now accepts stream IDs 0, 4, 8, and 12 and transfers 524,288 bytes in 19,817 us (`bench-throughput openssl 4 524288`); this is a smoke result, not a five-repeat benchmark. The configured four-stream measurement is now viable.
Author
Owner

Size finding (static x86_64 musl, release/LTO, stripped; raw/research/quic-stacks/raw/size-binaries.csv): the OpenSSL TLS baseline is 5,190,400 B, Quinn candidate 6,987,368 B (+1,796,968 B), and OpenSSL-native candidate 5,192,096 B (+1,696 B). The OpenSSL delta is small because QUIC engine objects are already pulled into the baseline OpenSSL link; this is marginal size over the required baseline, not the total OpenSSL QUIC implementation size. The historical #78 Quinn delta used a different probe/dependency set and is not directly comparable. Full repeated performance and memory runs are still in progress.

Size finding (static x86_64 musl, release/LTO, stripped; raw/research/quic-stacks/raw/size-binaries.csv): the OpenSSL TLS baseline is 5,190,400 B, Quinn candidate 6,987,368 B (+1,796,968 B), and OpenSSL-native candidate 5,192,096 B (+1,696 B). The OpenSSL delta is small because QUIC engine objects are already pulled into the baseline OpenSSL link; this is marginal size over the required baseline, not the total OpenSSL QUIC implementation size. The historical #78 Quinn delta used a different probe/dependency set and is not directly comparable. Full repeated performance and memory runs are still in progress.
Author
Owner

Harness finding from the delayed Quinn handshake run: repetition 3 failed twice before one successful attempt. Both failure logs (research/quic-stacks/raw/handshake-300ms-quinn-3-a1.log and -a2.log) say the server handshake was aborted by the peer; the client closed each Connection immediately after connect() returned while the server was still in incoming.await(). The timing indicates a harness teardown race, so I will defer connection close until server acceptance is confirmed and rerun all five Quinn delayed handshakes. Failed logs remain as evidence and will be excluded from the reported numbers.

Harness finding from the delayed Quinn handshake run: repetition 3 failed twice before one successful attempt. Both failure logs (`research/quic-stacks/raw/handshake-300ms-quinn-3-a1.log` and `-a2.log`) say the server handshake was aborted by the peer; the client closed each `Connection` immediately after `connect()` returned while the server was still in `incoming.await()`. The timing indicates a harness teardown race, so I will defer connection close until server acceptance is confirmed and rerun all five Quinn delayed handshakes. Failed logs remain as evidence and will be excluded from the reported numbers.
Author
Owner

Memory-probe finding: all three initial OpenSSL 1,000-connection attempts failed at SSL_new(memory client) (raw/memory-1000-openssl-1-a1.log through -a3.log), before producing server RSS. The benchmark process had a 1,024 soft RLIMIT_NOFILE while the harness opens one client UDP socket per connection; I infer descriptor pressure is the cause, but will confirm by rerunning with an 8,192 scope limit. These attempts are preserved and excluded from the comparison.

Memory-probe finding: all three initial OpenSSL 1,000-connection attempts failed at `SSL_new(memory client)` (`raw/memory-1000-openssl-1-a1.log` through `-a3.log`), before producing server RSS. The benchmark process had a 1,024 soft `RLIMIT_NOFILE` while the harness opens one client UDP socket per connection; I infer descriptor pressure is the cause, but will confirm by rerunning with an 8,192 scope limit. These attempts are preserved and excluded from the comparison.
Author
Owner

The corrected OpenSSL memory probe now completes with 1,000 connections and one active stream per connection under an 8,192 soft file-descriptor limit. The one-off diagnostic measured 69,775 B idle and 6,447 B incremental stream RSS per connection (research/quic-stacks/raw/diagnostics/memory-openssl-limit8192.csv); this is a harness check, not part of the final five-run summary. Client QUIC events now remain serviced while the server accepts streams.

The corrected OpenSSL memory probe now completes with 1,000 connections and one active stream per connection under an 8,192 soft file-descriptor limit. The one-off diagnostic measured 69,775 B idle and 6,447 B incremental stream RSS per connection (`research/quic-stacks/raw/diagnostics/memory-openssl-limit8192.csv`); this is a harness check, not part of the final five-run summary. Client QUIC events now remain serviced while the server accepts streams.
Author
Owner

Corrected handshake/0-RTT step completed; final five-run CSVs are in research/quic-stacks/raw/. Quinn fresh mTLS loopback median is 955 us per connection (five-run spread 948–1,020 us; median within-run p95 1,353 us) at 1,901 us CPU/connection. OpenSSL 3.6.3 is 4,883 us (4,605–6,296 us; p95 median 13,982 us) at 7,976 us CPU/connection. At 300 ms RTT, Quinn is 311,723 us (277,245–315,592 us) and OpenSSL is 886,331 us (867,227–888,265 us). Quinn's idempotent 0-RTT echo is 1,212 us on loopback and 603,050 us on the delayed path; OpenSSL native QUIC does not support 0-RTT. The measured mTLS server waits for authentication before acting on early data, so this is not an application-latency win under the current policy. Stream and memory repeats remain in progress.

Corrected handshake/0-RTT step completed; final five-run CSVs are in `research/quic-stacks/raw/`. Quinn fresh mTLS loopback median is 955 us per connection (five-run spread 948–1,020 us; median within-run p95 1,353 us) at 1,901 us CPU/connection. OpenSSL 3.6.3 is 4,883 us (4,605–6,296 us; p95 median 13,982 us) at 7,976 us CPU/connection. At 300 ms RTT, Quinn is 311,723 us (277,245–315,592 us) and OpenSSL is 886,331 us (867,227–888,265 us). Quinn's idempotent 0-RTT echo is 1,212 us on loopback and 603,050 us on the delayed path; OpenSSL native QUIC does not support 0-RTT. The measured mTLS server waits for authentication before acting on early data, so this is not an application-latency win under the current policy. Stream and memory repeats remain in progress.
Author
Owner

Five-run results are in research/quic-stacks/REPORT.md; exact rows and qdisc logs are under research/quic-stacks/raw/.

Measure Quinn + rustls + ring OpenSSL native QUIC 3.6.3
Added stripped static binary over OpenSSL TLS baseline 1,796,968 B 1,696 B*
Fresh mTLS handshake, loopback 955 µs; 1.901 ms CPU/conn 4,883 µs; 7.977 ms CPU/conn
Fresh mTLS handshake, 300 ms RTT 311.723 ms; 2.352 ms CPU 886.331 ms; 5.528 ms CPU
1 stream, loopback 56.370 MiB/s; 2.896 CPU s/Gbit 55.509 MiB/s; 3.376 CPU s/Gbit
4 streams, loopback 52.587 MiB/s; 3.368 CPU s/Gbit 56.808 MiB/s; 3.633 CPU s/Gbit
1 stream, delayed path / configured 1% loss 0.515 MiB/s; 5.076 CPU s/Gbit 0.022 MiB/s; 39.969 CPU s/Gbit
4 streams, delayed path / configured 1% loss 0.536 MiB/s; 5.573 CPU s/Gbit 0.022 MiB/s; 76.693 CPU s/Gbit
1,000 idle server connections 43,978 B/conn; 41.94 MiB total 69,767 B/conn; 66.53 MiB total
Added RSS per active stream × 1,000 647 B/conn; 0.62 MiB total 6,447 B/conn; 6.15 MiB total

*OpenSSL's static TLS baseline already pulls QUIC implementation objects, so 1,696 B is marginal added size, not standalone engine size. Qdisc counters recorded Quinn loss at 0.16–0.26% and OpenSSL at 1.02–1.07% despite both being configured at 1%; impaired throughput is therefore not a matched-realized-loss comparison. Loopback throughput was comparable. Quinn's other measured benefits are faster handshake, lower connection RSS, explicit BBR selection, direct Tokio async integration, 0-RTT, and migration APIs. OpenSSL native QUIC lacks native 0-RTT and a public BBR selector; the tested OpenSSL 3.6.3 predates QUIC security fixes in 3.6.4/3.6.5.

Recommendation: retain Quinn + rustls + ring with BBR for now (moderate confidence); the image size advantage favors OpenSSL, but latency, memory, API fit, and observed delayed-path behavior favor Quinn. Repeat delayed throughput with matched deterministic loss and larger transfers before using it as a capacity estimate. No foundational decision changed.

Five-run results are in `research/quic-stacks/REPORT.md`; exact rows and qdisc logs are under `research/quic-stacks/raw/`. | Measure | Quinn + rustls + ring | OpenSSL native QUIC 3.6.3 | |---|---:|---:| | Added stripped static binary over OpenSSL TLS baseline | 1,796,968 B | 1,696 B* | | Fresh mTLS handshake, loopback | 955 µs; 1.901 ms CPU/conn | 4,883 µs; 7.977 ms CPU/conn | | Fresh mTLS handshake, 300 ms RTT | 311.723 ms; 2.352 ms CPU | 886.331 ms; 5.528 ms CPU | | 1 stream, loopback | 56.370 MiB/s; 2.896 CPU s/Gbit | 55.509 MiB/s; 3.376 CPU s/Gbit | | 4 streams, loopback | 52.587 MiB/s; 3.368 CPU s/Gbit | 56.808 MiB/s; 3.633 CPU s/Gbit | | 1 stream, delayed path / configured 1% loss | 0.515 MiB/s; 5.076 CPU s/Gbit | 0.022 MiB/s; 39.969 CPU s/Gbit | | 4 streams, delayed path / configured 1% loss | 0.536 MiB/s; 5.573 CPU s/Gbit | 0.022 MiB/s; 76.693 CPU s/Gbit | | 1,000 idle server connections | 43,978 B/conn; 41.94 MiB total | 69,767 B/conn; 66.53 MiB total | | Added RSS per active stream × 1,000 | 647 B/conn; 0.62 MiB total | 6,447 B/conn; 6.15 MiB total | *OpenSSL's static TLS baseline already pulls QUIC implementation objects, so 1,696 B is marginal added size, not standalone engine size. Qdisc counters recorded Quinn loss at 0.16–0.26% and OpenSSL at 1.02–1.07% despite both being configured at 1%; impaired throughput is therefore not a matched-realized-loss comparison. Loopback throughput was comparable. Quinn's other measured benefits are faster handshake, lower connection RSS, explicit BBR selection, direct Tokio async integration, 0-RTT, and migration APIs. OpenSSL native QUIC lacks native 0-RTT and a public BBR selector; the tested OpenSSL 3.6.3 predates QUIC security fixes in 3.6.4/3.6.5. Recommendation: retain Quinn + rustls + ring with BBR for now (moderate confidence); the image size advantage favors OpenSSL, but latency, memory, API fit, and observed delayed-path behavior favor Quinn. Repeat delayed throughput with matched deterministic loss and larger transfers before using it as a capacity estimate. No foundational decision changed.
Author
Owner

Step complete: committed the reproducible static-musl comparison harness at 1371ebf (build + Quinn/OpenSSL mTLS smoke checks passed). It measures Quinn + rustls + ring and OpenSSL native QUIC against a TLS baseline; the full five-repeat matrix is now recorded and the report/raw-data commit is next.

Step complete: committed the reproducible static-musl comparison harness at `1371ebf` (build + Quinn/OpenSSL mTLS smoke checks passed). It measures Quinn + rustls + ring and OpenSSL native QUIC against a TLS baseline; the full five-repeat matrix is now recorded and the report/raw-data commit is next.
Author
Owner

Step complete: report and raw data committed at 5c7d471. The five-run matrix shows 955 µs vs 4,883 µs loopback fresh mTLS, 43,978 B vs 69,767 B idle RSS per connection, and 0.515/0.536 vs 0.022 MiB/s on the configured delayed loss path. The report records that qdisc counters realized 0.16–0.26% loss for Quinn and 1.02–1.07% for OpenSSL, so the loss-path ratio is not a same-loss estimate. Full comparison table and files are in research/quic-stacks/REPORT.md and research/quic-stacks/raw/.

Step complete: report and raw data committed at `5c7d471`. The five-run matrix shows 955 µs vs 4,883 µs loopback fresh mTLS, 43,978 B vs 69,767 B idle RSS per connection, and 0.515/0.536 vs 0.022 MiB/s on the configured delayed loss path. The report records that qdisc counters realized 0.16–0.26% loss for Quinn and 1.02–1.07% for OpenSSL, so the loss-path ratio is not a same-loss estimate. Full comparison table and files are in `research/quic-stacks/REPORT.md` and `research/quic-stacks/raw/`.
Author
Owner

Verification note: the required command scripts/verify fast cannot run in this checkout. Output verbatim: zsh:1: no such file or directory: scripts/verify. scripts/ contains only perf-run, and main has the same contents. I left the repository gate unchanged because this job is restricted to research/quic-stacks/. Harness build/smoke, the five-repeat benchmark matrix, cargo fmt --check, shell syntax checks, and git diff --check passed.

Verification note: the required command `scripts/verify fast` cannot run in this checkout. Output verbatim: `zsh:1: no such file or directory: scripts/verify`. `scripts/` contains only `perf-run`, and `main` has the same contents. I left the repository gate unchanged because this job is restricted to `research/quic-stacks/`. Harness build/smoke, the five-repeat benchmark matrix, `cargo fmt --check`, shell syntax checks, and `git diff --check` passed.
Author
Owner

Review fix complete: 00a9297 gives each default benchmark run a fresh UTC timestamped output directory and refuses an existing explicit output path. The guard was exercised against the committed raw directory and exited 2 with refusing to overwrite existing results directory: research/quic-stacks/raw; no measurements were run or overwritten. Reproduction instructions now describe the output behavior.

Review fix complete: `00a9297` gives each default benchmark run a fresh UTC timestamped output directory and refuses an existing explicit output path. The guard was exercised against the committed raw directory and exited 2 with `refusing to overwrite existing results directory: research/quic-stacks/raw`; no measurements were run or overwritten. Reproduction instructions now describe the output behavior.
Author
Owner

Finished at HEAD 00a9297. The comparison table is posted above; report, harness, five-repeat raw numbers and logs are committed under research/quic-stacks/. Independent review completed with no findings after the rerun-output fix. Harness build, both real-stack mTLS smoke checks, the 80-invocation matrix, cargo fmt --check, shell syntax, output-directory guard, and git diff --check passed.

Required gate output verbatim: zsh:1: no such file or directory: scripts/verify (exit 127). scripts/verify is absent from this worktree and main; I left the repository gate unchanged because this job permits writes only in research/quic-stacks/.

Finished at HEAD `00a9297`. The comparison table is posted above; report, harness, five-repeat raw numbers and logs are committed under `research/quic-stacks/`. Independent review completed with no findings after the rerun-output fix. Harness build, both real-stack mTLS smoke checks, the 80-invocation matrix, `cargo fmt --check`, shell syntax, output-directory guard, and `git diff --check` passed. Required gate output verbatim: `zsh:1: no such file or directory: scripts/verify` (exit 127). `scripts/verify` is absent from this worktree and `main`; I left the repository gate unchanged because this job permits writes only in `research/quic-stacks/`.
Author
Owner

Owner decision (2026-10-08): keep quinn + rustls + ring. Recorded as D176 on main. Follow-ups: rerun the delayed-path row with matched loss; measure BBR at 300 ms RTT after remote-agents lands (it owns the transport code).

Owner decision (2026-10-08): keep quinn + rustls + ring. Recorded as D176 on main. Follow-ups: rerun the delayed-path row with matched loss; measure BBR at 300 ms RTT after remote-agents lands (it owns the transport code).
Sign in to join this conversation.
No labels
launch-qa
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/infraternal#88
No description provided.