Reverse proxy on every client: embeddable engine candidates, measured (D316) #261

New issue

Open

opened 2026-10-30 16:12:54 +00:00 by kayg · 31 comments

kayg commented 2026-10-30 16:12:54 +00:00

Owner

Copy link

Research job (measure, do not decide). D316 says: integrate Traefik for the reverse proxy, never write our own proxy; certificates use DNS-01 by default. Traefik is a Go program and cannot be embedded in the single static Rust binary (D7, D243), so this issue measures the embeddable Rust alternatives against Traefik as a managed sidecar. The owner decides.

Candidates: Pingora (Apache-2.0), Sōzu (sozu-lib), rpxy, and an ACME DNS-01 client (instant-acme), with Traefik as the sidecar reference.

For each candidate, in a throwaway crate outside the workspace (never added to the product's Cargo.toml/Cargo.lock): an HTTPS listener with SNI routing of two hostnames to two local backends (HTTP/1.1, HTTP/2, WebSocket upgrade, gRPC/h2c where supported), route hot reload without dropping connections, and TLS from a certificate loaded at runtime. Recorded: (1) stripped static size delta, x86_64 and aarch64, plus cargo tree -e features dependency count; (2) idle RSS and RSS at 1,000 keep-alive connections; (3) requests per second and p50/p99 at concurrency 1/64/512; (4) TLS crypto stack, and whether rustls + ring (D176) is possible; (5) route reload behaviour; (6) licence compatibility with AGPL-3.0-only; (7) maintenance signals. ACME client: DNS-01 against a local Pebble with a stub DNS provider, plus size delta.

Deliverables: results table on this issue (timings labelled "build host aarch64"), an x86 relay kit for the perf VM, and research/reverse-proxy/README.md on branch research-reverse-proxy (pushed, not merged). Refs D316, D219, D198, D243, D176.

Label: research-only. Nothing here ships.

Research job (measure, do not decide). D316 says: integrate Traefik for the reverse proxy, never write our own proxy; certificates use DNS-01 by default. Traefik is a Go program and cannot be embedded in the single static Rust binary (D7, D243), so this issue measures the embeddable Rust alternatives against Traefik as a managed sidecar. The owner decides. Candidates: Pingora (Apache-2.0), Sōzu (`sozu-lib`), rpxy, and an ACME DNS-01 client (`instant-acme`), with Traefik as the sidecar reference. For each candidate, in a throwaway crate outside the workspace (never added to the product's `Cargo.toml`/`Cargo.lock`): an HTTPS listener with SNI routing of two hostnames to two local backends (HTTP/1.1, HTTP/2, WebSocket upgrade, gRPC/h2c where supported), route hot reload without dropping connections, and TLS from a certificate loaded at runtime. Recorded: (1) stripped static size delta, x86_64 and aarch64, plus `cargo tree -e features` dependency count; (2) idle RSS and RSS at 1,000 keep-alive connections; (3) requests per second and p50/p99 at concurrency 1/64/512; (4) TLS crypto stack, and whether rustls + ring (D176) is possible; (5) route reload behaviour; (6) licence compatibility with AGPL-3.0-only; (7) maintenance signals. ACME client: DNS-01 against a local Pebble with a stub DNS provider, plus size delta. Deliverables: results table on this issue (timings labelled "build host aarch64"), an x86 relay kit for the perf VM, and `research/reverse-proxy/README.md` on branch `research-reverse-proxy` (pushed, not merged). Refs D316, D219, D198, D243, D176. Label: research-only. Nothing here ships.

kayg commented 2026-10-30 16:13:01 +00:00

Author

Owner

Copy link

Started. Branch research-reverse-proxy, base a4f15f8f7d7049e1ee6504c5ae3643c829c9bdc9 (origin/dev). Research only; nothing ships. Plan: throwaway crates under scratch (outside the workspace) for Pingora, Sozu, rpxy, instant-acme+Pebble, Traefik sidecar reference; same release profile as the product (opt 3, fat LTO, cgu 1, strip, panic unwind, musl crt-static, pinned Zig as the C compiler). Timings come from the aarch64 build host (labelled as such, not evidence per the perf-VM rule); an x86 relay kit for the perf VM follows.

Started. Branch `research-reverse-proxy`, base `a4f15f8f7d7049e1ee6504c5ae3643c829c9bdc9` (origin/dev). Research only; nothing ships. Plan: throwaway crates under scratch (outside the workspace) for Pingora, Sozu, rpxy, instant-acme+Pebble, Traefik sidecar reference; same release profile as the product (opt 3, fat LTO, cgu 1, strip, panic unwind, musl crt-static, pinned Zig as the C compiler). Timings come from the aarch64 build host (labelled as such, not evidence per the perf-VM rule); an x86 relay kit for the perf VM follows.

kayg commented 2026-10-30 17:03:28 +00:00

Author

Owner

Copy link

Progress (research only; no timings yet, they come after the size builds finish so the host is quiet).

Branch research-reverse-proxy is pushed with a shared helper tool, the per-case measurement script, and demos for Pingora 0.9.0 (rustls + ring), Sozu 2.3.3 (sozu-lib, ring), rpxy 0.14.0 (rpxy-lib + rpxy-certs) and instant-acme 0.8.5. Each proxy demo passed the same 9-case functional matrix against Traefik 3.7.17 as the reference: SNI picks the right certificate for each name, HTTP/1.1, HTTP/2, h2c upstream, WebSocket upgrade over TLS, gRPC-shaped trailers, and keep-alive reuse. Live reload (new certificate and route while an old keep-alive connection, a WebSocket and an in-flight request are open) completes in all four; how it behaves differs (below).

Findings so far, each from the demos or the upstream manifests:

• Pingora 0.9.0 panics on tokio ≥ 1.54: it builds BufStream::with_capacity(0, 0, ..) per accepted connection and tokio 1.54.1 asserts capacity > 0, so the first TLS connection kills the listener thread. Works on the product's locked tokio 1.53.3. Pingora has no route or certificate reload of its own (only process-level graceful upgrade); the demo swaps an ArcSwap route table and a rustls certificate resolver. Server::run_forever takes over the process and exits it; SIGTERM is a slow graceful drain.
• Sozu licensing: sozu-lib is AGPL-3.0 ("version 3.0 and above" in its README), sozu-command-lib (a required dependency) is LGPL-3.0. Neither is in deny.toml's allow-list, and the project README adds a request to give back contributions or contact the vendor for a "business based on Sōzu". Our core is AGPL-3.0-only, so the library is combinable; what ee/ (D31, commercial) would mean inside the same binary is a question for the owner. kawa (also required) has a non-SPDX Apache-2.0 license-file. Building sozu-command-lib needs protoc (prost-build), a new build prerequisite next to the pinned Zig.
• Sozu embedding: the supported shape is a supervisor process plus worker processes; in-process workers use https::testing::start_https_worker, which upstream labels "available for example and testing purposes". Hot reconfiguration (AddCertificate, ReplaceCertificate, AddBackend, RemoveBackend) is native.
• rpxy: not on crates.io (publish = false), so a git dependency (deny.toml has unknown-git = "deny"). rpxy-certs hard-wires rustls with aws-lc-rs and installs it as the process default, so rustls + ring is not a switch. Its pinned-Zig static build fails to link on aarch64 (__ubsan_handle_shift_out_of_bounds from aws-lc's jitterentropy); being retried with -fno-sanitize=undefined. max_clients defaults to 512 connections process-wide (1,000 held connections failed 488 times until raised). A route change restarts its entrypoint (as rpxy's own binary does on a config change): existing connections, WebSocket and in-flight request survive, but new connections can be refused during the restart (0 and 168 of about 600 attempts in two runs); certificates reload by 1 s polling.
• instant-acme 0.8.5 needs default-features = false, features = ["ring", "hyper-rustls", "rcgen"] to avoid aws-lc-rs. DNS-01 against Pebble 2.10.1 with a stub DNS provider (two methods, one HTTP call per record, no extra dependency) issued 2 names in 1.7 s (about 845 ms each, Pebble's validation wait switched off).
• Traefik 3.7.17 (reference): 89 published GitHub advisories, 64 of them in the last 12 months (2 critical, 36 high); Pingora has 8 published, all patched in 0.9.0; Sozu, rpxy and instant-acme have none published on GitHub.
• Product baseline (merge base a4f15f8f7, same recipe): stripped static infraternal is 19,601,832 B aarch64 and 21,437,872 B x86_64.

Progress (research only; no timings yet, they come after the size builds finish so the host is quiet). Branch `research-reverse-proxy` is pushed with a shared helper tool, the per-case measurement script, and demos for Pingora 0.9.0 (rustls + ring), Sozu 2.3.3 (`sozu-lib`, ring), rpxy 0.14.0 (`rpxy-lib` + `rpxy-certs`) and `instant-acme` 0.8.5. Each proxy demo passed the same 9-case functional matrix against Traefik 3.7.17 as the reference: SNI picks the right certificate for each name, HTTP/1.1, HTTP/2, h2c upstream, WebSocket upgrade over TLS, gRPC-shaped trailers, and keep-alive reuse. Live reload (new certificate and route while an old keep-alive connection, a WebSocket and an in-flight request are open) completes in all four; how it behaves differs (below). Findings so far, each from the demos or the upstream manifests: - **Pingora 0.9.0 panics on tokio ≥ 1.54**: it builds `BufStream::with_capacity(0, 0, ..)` per accepted connection and tokio 1.54.1 asserts `capacity > 0`, so the first TLS connection kills the listener thread. Works on the product's locked tokio 1.53.3. Pingora has no route or certificate reload of its own (only process-level graceful upgrade); the demo swaps an `ArcSwap` route table and a rustls certificate resolver. `Server::run_forever` takes over the process and exits it; SIGTERM is a slow graceful drain. - **Sozu licensing**: `sozu-lib` is `AGPL-3.0` ("version 3.0 and above" in its README), `sozu-command-lib` (a required dependency) is `LGPL-3.0`. Neither is in `deny.toml`'s allow-list, and the project README adds a request to give back contributions or contact the vendor for a "business based on Sōzu". Our core is AGPL-3.0-only, so the library is combinable; what `ee/` (D31, commercial) would mean inside the same binary is a question for the owner. `kawa` (also required) has a non-SPDX Apache-2.0 `license-file`. Building `sozu-command-lib` needs `protoc` (prost-build), a new build prerequisite next to the pinned Zig. - **Sozu embedding**: the supported shape is a supervisor process plus worker processes; in-process workers use `https::testing::start_https_worker`, which upstream labels "available for example and testing purposes". Hot reconfiguration (`AddCertificate`, `ReplaceCertificate`, `AddBackend`, `RemoveBackend`) is native. - **rpxy**: not on crates.io (`publish = false`), so a git dependency (`deny.toml` has `unknown-git = "deny"`). `rpxy-certs` hard-wires rustls with **aws-lc-rs** and installs it as the process default, so rustls + ring is not a switch. Its pinned-Zig static build fails to link on aarch64 (`__ubsan_handle_shift_out_of_bounds` from aws-lc's jitterentropy); being retried with `-fno-sanitize=undefined`. `max_clients` defaults to 512 connections process-wide (1,000 held connections failed 488 times until raised). A route change restarts its entrypoint (as rpxy's own binary does on a config change): existing connections, WebSocket and in-flight request survive, but new connections can be refused during the restart (0 and 168 of about 600 attempts in two runs); certificates reload by 1 s polling. - **instant-acme 0.8.5** needs `default-features = false, features = ["ring", "hyper-rustls", "rcgen"]` to avoid aws-lc-rs. DNS-01 against Pebble 2.10.1 with a stub DNS provider (two methods, one HTTP call per record, no extra dependency) issued 2 names in 1.7 s (about 845 ms each, Pebble's validation wait switched off). - **Traefik 3.7.17** (reference): 89 published GitHub advisories, 64 of them in the last 12 months (2 critical, 36 high); Pingora has 8 published, all patched in 0.9.0; Sozu, rpxy and instant-acme have none published on GitHub. - Product baseline (merge base `a4f15f8f7`, same recipe): stripped static `infraternal` is 19,601,832 B aarch64 and 21,437,872 B x86_64.

kayg commented 2026-10-30 17:14:44 +00:00

Author

Owner

Copy link

Paused for the VM reboot (per orchestrator). Everything running was stopped by PID; nothing of mine is left running. WIP commit pushed to research-reverse-proxy.

Done: demos for Pingora 0.9.0, Sozu 2.3.3, rpxy 0.14.0, instant-acme 0.8.5 (all pass the 9-case functional matrix; trial runs on aarch64 exist), measurement scripts and relay entry kit/run-all.sh, dependency/licence census, maintenance signals, product-wired static size builds for Pingora/Sozu/rpxy (both targets).

Stripped static sizes so far (product baseline: 19,601,832 B aarch64 / 21,437,872 B x86_64):

• Pingora: +4,763,968 aarch64 / +5,338,368 x86_64
• Sozu: +5,168,424 aarch64 / +6,383,648 x86_64
• rpxy: +3,532,576 aarch64 / +4,731,328 x86_64
  (instant-acme product-wired build was interrupted by the stop; standalone 2,530,768 aarch64 / 2,990,800 x86_64.)

Next: instant-acme wired size, full-duration aarch64 passes, kit tarball with sha256, write-up. Scratch state: ~/.cache/codex-scratch/rproxy.

Paused for the VM reboot (per orchestrator). Everything running was stopped by PID; nothing of mine is left running. WIP commit pushed to `research-reverse-proxy`. Done: demos for Pingora 0.9.0, Sozu 2.3.3, rpxy 0.14.0, instant-acme 0.8.5 (all pass the 9-case functional matrix; trial runs on aarch64 exist), measurement scripts and relay entry `kit/run-all.sh`, dependency/licence census, maintenance signals, product-wired static size builds for Pingora/Sozu/rpxy (both targets). Stripped static sizes so far (product baseline: 19,601,832 B aarch64 / 21,437,872 B x86_64): - Pingora: +4,763,968 aarch64 / +5,338,368 x86_64 - Sozu: +5,168,424 aarch64 / +6,383,648 x86_64 - rpxy: +3,532,576 aarch64 / +4,731,328 x86_64 (instant-acme product-wired build was interrupted by the stop; standalone 2,530,768 aarch64 / 2,990,800 x86_64.) Next: instant-acme wired size, full-duration aarch64 passes, kit tarball with sha256, write-up. Scratch state: ~/.cache/codex-scratch/rproxy.

kayg referenced this issue 2026-10-30 17:36:12 +00:00

Orchestrator handover 2026-10-30: all work paused; mesh-first roadmap; Claude-only #262

kayg commented 2026-10-31 16:12:35 +00:00

Author

Owner

Copy link

J20 resumed: branch research-reverse-proxy, base ad7daa89f. Isolated worktree on control host; x86 builds/tests on compute01; full-duration proxy/ACME measurements on backup01 strictly through perf-run. Research only; owner chooses the engine.

J20 resumed: branch research-reverse-proxy, base ad7daa89f. Isolated worktree on control host; x86 builds/tests on compute01; full-duration proxy/ACME measurements on backup01 strictly through perf-run. Research only; owner chooses the engine.

kayg commented 2026-10-31 16:17:04 +00:00

Author

Owner

Copy link

J20 infrastructure blocker: compute01 is unreachable (ssh: connect to host infraternal-dev-01-compute01 port 22: No route to host) after build launch PID 280013, log /home/claude/worktrees/j20-rproxy/build.log. Its completion and cleanup are not yet known. Scratch caches from the previous host are absent. No performance measurement has run. Preparing full TLS handshake and CPU instrumentation locally; no engine selected.

J20 infrastructure blocker: compute01 is unreachable (`ssh: connect to host infraternal-dev-01-compute01 port 22: No route to host`) after build launch PID 280013, log /home/claude/worktrees/j20-rproxy/build.log. Its completion and cleanup are not yet known. Scratch caches from the previous host are absent. No performance measurement has run. Preparing full TLS handshake and CPU instrumentation locally; no engine selected.

kayg commented 2026-10-31 17:12:24 +00:00

Author

Owner

Copy link

J20 progress: the Nix store repair unblocked the x86 build. labtool, Pingora and Sozu compiled. rpxy failed at static link on unresolved __ubsan_handle_type_mismatch_v1 / pointer_overflow / shift_out_of_bounds / divrem_overflow / out_of_bounds from aws-lc-sys 0.47.0 jitterentropy objects. Its native jitter sources intentionally compile at -O0, enabling Zig default UBSan without a C sanitizer runtime in the Rust static link. A research-only CFLAGS_x86_64_unknown_linux_musl=-fno-sanitize=undefined retry is queued through build-slot; not yet verified. This preserves the required unoptimized entropy code and the provider, with no product build-policy change. Full-duration direct and Traefik perf-run cases completed; Pingora is measuring now. Initial cleanup diffs contained only kernel worker churn (raw diffs retained); a revised snapshot excludes kthreadd children and the fresh direct case reports STATE DIFF EMPTY. Final comparison and evidence follow after the remaining cases.

J20 progress: the Nix store repair unblocked the x86 build. labtool, Pingora and Sozu compiled. rpxy failed at static link on unresolved __ubsan_handle_type_mismatch_v1 / pointer_overflow / shift_out_of_bounds / divrem_overflow / out_of_bounds from aws-lc-sys 0.47.0 jitterentropy objects. Its native jitter sources intentionally compile at -O0, enabling Zig default UBSan without a C sanitizer runtime in the Rust static link. A research-only CFLAGS_x86_64_unknown_linux_musl=-fno-sanitize=undefined retry is queued through build-slot; not yet verified. This preserves the required unoptimized entropy code and the provider, with no product build-policy change. Full-duration direct and Traefik perf-run cases completed; Pingora is measuring now. Initial cleanup diffs contained only kernel worker churn (raw diffs retained); a revised snapshot excludes kthreadd children and the fresh direct case reports STATE DIFF EMPTY. Final comparison and evidence follow after the remaining cases.

kayg commented 2026-10-31 17:49:20 +00:00

Author

Owner

Copy link

J20 full-duration reverse-proxy measurements on infraternal-perf-01-backup01 (x86_64), strictly through non-root perf-run. Engine choice remains with kayg.

All load phases use 5 s warmup + 30 s measurement. The proxy gets two of perf.slice CPUs 1–3; backend and generator share the third. One measured run per workload, with three reload trials. The direct HTTP reference reached 63,506 requests/s at h1 c64; these results include a client/backend ceiling and are not unconstrained engine capacity. CPU 100% means one core.

Engine	h1 c1: req/s / p99 ms	h1 c64	h1 c512	h2 c64
Pingora	7,140 / 0.328	45,372 / 2.981	34,965 / 38.203	35,246 / 3.673
Sozu	11,489 / 0.140	50,052 / 3.161	38,217 / 45.940	29,444 / 4.482
rpxy	10,394 / 0.151	52,315 / 2.120	38,928 / 45.097	35,733 / 3.119
Traefik	6,418 / 0.224	32,956 / 7.296	25,711 / 39.075	25,289 / 7.393

All HTTP load phases above returned only 200 responses, with no request errors. The functional matrix passed 9/9 before and after reload for every engine.

Engine	Idle RSS MiB	Warm RSS MiB	RSS at 1,000 connections MiB	Approx. added KiB/connection	VmHWM after HTTP MiB
Pingora	6.66	8.09	64.01	57.26	63.88
Sozu	8.19	10.04	52.36	43.34	52.10
rpxy	6.14	6.68	44.25	38.47	59.39
Traefik	93.30	95.77	150.27	55.81	265.84

RSS is the standalone research process, not the incremental embedded product cost. Each holder opened 1,000/1,000 connections successfully. The per-connection estimate is (held RSS − warm RSS) / 1000; it does not establish scaling beyond that point. Linux RSS/HWM counters are sampled and can differ slightly.

Engine	Idle CPU %	h1 c64 CPU %	h2 c64 CPU %	TLS c64 CPU %
Pingora	0.20	190.29	171.23	102.04
Sozu	0.00	141.73	132.02	101.39
rpxy	0.07	166.68	152.76	98.56
Traefik	0.00	193.73	195.31	159.19

Engine	Full TLS/s c1	p50 / p99 ms	Full TLS/s c64	p50 / p99 ms	TLS failures
Pingora	2,670	0.340 / 0.673	4,885	13.062 / 16.062	0
Sozu	2,745	0.343 / 0.443	5,187	12.346 / 15.702	0
rpxy	2,944	0.321 / 0.407	5,259	12.209 / 14.760	0
Traefik	2,262	0.419 / 0.559	5,624	11.371 / 16.314	0

TLS phases are 30 s each, fresh TCP + full TLS only, with resumption disabled and no HTTP. A common client verifies handshake signatures but skips disposable certificate-chain validation. Client work shares the single generator CPU, so these are end-to-end rates under this allocation.

Engine	Reload failed requests in trials 1 / 2 / 3	Route switch ms	Certificate switch ms	Old HTTP keep-alive, next request
Pingora	0 / 0 / 0	95 / 38 / 191	94 / 49 / 191	alive; backend C
Sozu	0 / 0 / 0	109 / 50 / 203	108 / 49 / 203	alive; backend C
rpxy	229 / 40 / 24	59 / 141 / 94	949 / 174 / 665	alive; backend A
Traefik	0 / 0 / 0	27 / 561 / 1041	27 / 560 / 1041	alive; backend C

rpxy restarts its demo entrypoint on route changes: reload lost 229/40/24 requests, while old keep-alive, WebSocket, h2 and in-flight work survived. Old keep-alive connections remained on backend A; the others switched them to backend C. The rpxy case exits nonzero for the reload failures; its final basic matrix still passes.

DNS-01 via instant-acme 0.8.5 passed: two certificates in 1,789 ms total (896 / 847 ms per name), client peak RSS 4,988 KiB, executable 2,990,800 B. Local Pebble CA/stub DNS, validation sleep disabled; this excludes public DNS propagation, real provider authentication and public-CA latency.

Cleanup: final direct, Pingora, Sozu, rpxy and ACME snapshots are empty. Initial direct/Traefik snapshot failures consisted exclusively of kernel-worker name/count churn; all sockets, kit and temporary-file diffs were empty. Raw diffs are preserved. The snapshot now excludes kthreadd children; the repeated direct case validated the change. Initial ACME staging lacked executable bits on official archive members; the successful retry used the kit’s intended executable modes.

Build: both x86 kits compiled on compute01 through build-slot. rpxy’s aws-lc jitter sources require -O0; Zig default UBSan produced unresolved _ubsan_handle* symbols at static link. The research-only C flag -fno-sanitize=undefined resolves this without disabling jitter entropy, switching providers or changing product build policy. The archive SHA-256 is c2f9e75aae2e7457df102c533f9d3aa83bbfb7880befe22ae3ba5eb8d8b889d2; perf kit members pass its MANIFEST.

Prior measured product deltas and integration constraints remain those recorded earlier on this issue: Pingora +5,338,368 B (locked tokio 1.53.3; 1.54.1 panic), Sozu +6,383,648 B (AGPL/LGPL required crates outside deny allow-list; supervisor/worker integration), rpxy +4,731,328 B (git-only; aws-lc process default differs from D176). The Traefik sidecar executable here is 182,120,608 B. The owner must weigh those constraints and reload correctness alongside these timings; J20 selects no engine.

Required repository gates are running; final branch head and verbatim gate result will follow.

J20 full-duration reverse-proxy measurements on `infraternal-perf-01-backup01` (x86_64), strictly through non-root `perf-run`. Engine choice remains with kayg. All load phases use 5 s warmup + 30 s measurement. The proxy gets two of perf.slice CPUs 1–3; backend and generator share the third. One measured run per workload, with three reload trials. The direct HTTP reference reached 63,506 requests/s at h1 c64; these results include a client/backend ceiling and are not unconstrained engine capacity. CPU 100% means one core. | Engine | h1 c1: req/s / p99 ms | h1 c64 | h1 c512 | h2 c64 | |---|---:|---:|---:|---:| | Pingora | 7,140 / 0.328 | 45,372 / 2.981 | 34,965 / 38.203 | 35,246 / 3.673 | | Sozu | 11,489 / 0.140 | 50,052 / 3.161 | 38,217 / 45.940 | 29,444 / 4.482 | | rpxy | 10,394 / 0.151 | 52,315 / 2.120 | 38,928 / 45.097 | 35,733 / 3.119 | | Traefik | 6,418 / 0.224 | 32,956 / 7.296 | 25,711 / 39.075 | 25,289 / 7.393 | All HTTP load phases above returned only 200 responses, with no request errors. The functional matrix passed 9/9 before and after reload for every engine. | Engine | Idle RSS MiB | Warm RSS MiB | RSS at 1,000 connections MiB | Approx. added KiB/connection | VmHWM after HTTP MiB | |---|---:|---:|---:|---:|---:| | Pingora | 6.66 | 8.09 | 64.01 | 57.26 | 63.88 | | Sozu | 8.19 | 10.04 | 52.36 | 43.34 | 52.10 | | rpxy | 6.14 | 6.68 | 44.25 | 38.47 | 59.39 | | Traefik | 93.30 | 95.77 | 150.27 | 55.81 | 265.84 | RSS is the standalone research process, not the incremental embedded product cost. Each holder opened 1,000/1,000 connections successfully. The per-connection estimate is `(held RSS − warm RSS) / 1000`; it does not establish scaling beyond that point. Linux RSS/HWM counters are sampled and can differ slightly. | Engine | Idle CPU % | h1 c64 CPU % | h2 c64 CPU % | TLS c64 CPU % | |---|---:|---:|---:|---:| | Pingora | 0.20 | 190.29 | 171.23 | 102.04 | | Sozu | 0.00 | 141.73 | 132.02 | 101.39 | | rpxy | 0.07 | 166.68 | 152.76 | 98.56 | | Traefik | 0.00 | 193.73 | 195.31 | 159.19 | | Engine | Full TLS/s c1 | p50 / p99 ms | Full TLS/s c64 | p50 / p99 ms | TLS failures | |---|---:|---:|---:|---:|---:| | Pingora | 2,670 | 0.340 / 0.673 | 4,885 | 13.062 / 16.062 | 0 | | Sozu | 2,745 | 0.343 / 0.443 | 5,187 | 12.346 / 15.702 | 0 | | rpxy | 2,944 | 0.321 / 0.407 | 5,259 | 12.209 / 14.760 | 0 | | Traefik | 2,262 | 0.419 / 0.559 | 5,624 | 11.371 / 16.314 | 0 | TLS phases are 30 s each, fresh TCP + full TLS only, with resumption disabled and no HTTP. A common client verifies handshake signatures but skips disposable certificate-chain validation. Client work shares the single generator CPU, so these are end-to-end rates under this allocation. | Engine | Reload failed requests in trials 1 / 2 / 3 | Route switch ms | Certificate switch ms | Old HTTP keep-alive, next request | |---|---:|---|---|---| | Pingora | 0 / 0 / 0 | 95 / 38 / 191 | 94 / 49 / 191 | alive; backend C | | Sozu | 0 / 0 / 0 | 109 / 50 / 203 | 108 / 49 / 203 | alive; backend C | | rpxy | 229 / 40 / 24 | 59 / 141 / 94 | 949 / 174 / 665 | alive; backend A | | Traefik | 0 / 0 / 0 | 27 / 561 / 1041 | 27 / 560 / 1041 | alive; backend C | rpxy restarts its demo entrypoint on route changes: reload lost 229/40/24 requests, while old keep-alive, WebSocket, h2 and in-flight work survived. Old keep-alive connections remained on backend A; the others switched them to backend C. The rpxy case exits nonzero for the reload failures; its final basic matrix still passes. DNS-01 via instant-acme 0.8.5 passed: two certificates in 1,789 ms total (896 / 847 ms per name), client peak RSS 4,988 KiB, executable 2,990,800 B. Local Pebble CA/stub DNS, validation sleep disabled; this excludes public DNS propagation, real provider authentication and public-CA latency. Cleanup: final direct, Pingora, Sozu, rpxy and ACME snapshots are empty. Initial direct/Traefik snapshot failures consisted exclusively of kernel-worker name/count churn; all sockets, kit and temporary-file diffs were empty. Raw diffs are preserved. The snapshot now excludes kthreadd children; the repeated direct case validated the change. Initial ACME staging lacked executable bits on official archive members; the successful retry used the kit’s intended executable modes. Build: both x86 kits compiled on compute01 through build-slot. rpxy’s aws-lc jitter sources require -O0; Zig default UBSan produced unresolved __ubsan_handle_* symbols at static link. The research-only C flag -fno-sanitize=undefined resolves this without disabling jitter entropy, switching providers or changing product build policy. The archive SHA-256 is c2f9e75aae2e7457df102c533f9d3aa83bbfb7880befe22ae3ba5eb8d8b889d2; perf kit members pass its MANIFEST. Prior measured product deltas and integration constraints remain those recorded earlier on this issue: Pingora +5,338,368 B (locked tokio 1.53.3; 1.54.1 panic), Sozu +6,383,648 B (AGPL/LGPL required crates outside deny allow-list; supervisor/worker integration), rpxy +4,731,328 B (git-only; aws-lc process default differs from D176). The Traefik sidecar executable here is 182,120,608 B. The owner must weigh those constraints and reload correctness alongside these timings; J20 selects no engine. Required repository gates are running; final branch head and verbatim gate result will follow.

metrics.txt

12 KiB

kit-manifest.txt

1.3 KiB

rproxy-reference.log

7.4 KiB

rproxy-pingora.log

4.3 KiB

rproxy-sozu.log

3.3 KiB

rproxy-final.log

3.8 KiB

rproxy-acme-retry.log

923 B

kayg referenced this issue 2026-10-31 18:12:30 +00:00

Move production from monitor01 to ssh #274

kayg referenced this issue 2026-10-31 18:12:31 +00:00

Move production from monitor01 to ssh #274

kayg commented 2026-10-31 18:27:24 +00:00

Author

Owner

Copy link

Gate failure cause identified: the background/nohup launcher inherited SigIgn: 0000000000000007 (SIGHUP/SIGINT/SIGQUIT ignored) into the gate. infraternal_shim::exit_on_interrupt deliberately preserves an inherited SIG_IGN disposition, so the CLI interruption test's SIGINT was ignored. Both branches failed that same test under this launcher. This is not evidence of a product regression or an origin/dev baseline failure.

The replacement launcher explicitly resets those three dispositions before exec of build-slot/Nix/verify. Both complete scripts/verify fast gates will rerun; no product/test code or timeout is changed. Original logs are retained.

Gate failure cause identified: the background/nohup launcher inherited `SigIgn: 0000000000000007` (SIGHUP/SIGINT/SIGQUIT ignored) into the gate. `infraternal_shim::exit_on_interrupt` deliberately preserves an inherited SIG_IGN disposition, so the CLI interruption test's SIGINT was ignored. Both branches failed that same test under this launcher. This is not evidence of a product regression or an origin/dev baseline failure. The replacement launcher explicitly resets those three dispositions before exec of build-slot/Nix/verify. Both complete `scripts/verify fast` gates will rerun; no product/test code or timeout is changed. Original logs are retained.

kayg commented 2026-10-31 18:28:18 +00:00

Author

Owner

Copy link

Supplemental complete raw evidence for the measured comparison above: per-load HTTP/TLS JSON, functional/reload records, state diffs, perf-run before/after receipts, source/member manifest, and run logs. The validated retry rows and ACME permission-staging correction remain identified in the comparison comment; initial failures are retained rather than silently overwritten in the write-up. No engine is selected by this job.

Supplemental complete raw evidence for the measured comparison above: per-load HTTP/TLS JSON, functional/reload records, state diffs, perf-run before/after receipts, source/member manifest, and run logs. The validated retry rows and ACME permission-staging correction remain identified in the comparison comment; initial failures are retained rather than silently overwritten in the write-up. No engine is selected by this job.

rproxy-raw-evidence.zip

63 KiB

kayg commented 2026-10-31 19:03:03 +00:00

Author

Owner

Copy link

The existing CLI interruption test passes when launched with SIGINT's default disposition restored. This used the already-built test executable, with no compilation or repository change:

running 1 test
test ctrl_c_in_json_mode_ends_with_an_interrupted_envelope_and_exit_130 ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 16 filtered out; finished in 0.84s

This confirms the background-launcher diagnosis. Both complete fast gates remain queued through build-slot; this focused check does not replace them.

The existing CLI interruption test passes when launched with SIGINT's default disposition restored. This used the already-built test executable, with no compilation or repository change: ```text running 1 test test ctrl_c_in_json_mode_ends_with_an_interrupted_envelope_and_exit_130 ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 16 filtered out; finished in 0.84s ``` This confirms the background-launcher diagnosis. Both complete fast gates remain queued through build-slot; this focused check does not replace them.

kayg commented 2026-10-31 19:31:22 +00:00

Author

Owner

Copy link

J20 scheduling blocker: all measurements, comparisons, browser triage, source checks and perf cleanup are complete. The corrected full fast gates and final archive repack are waiting through the mandatory compute01 build-slot helper. Current queued PID: 2045649; log: ~/.cache/codex-scratch/j20/verify-both-signal-reset.log. This launcher alone has waited over 30 minutes; all four slot holders have changed while it has remained queued. The helper polls locks and has no FIFO queue.

Orchestrator scheduling help is needed if a prompt handoff is required: reserve a slot for this already-queued command. I am leaving it queued and will finish the two gates, final packages, commits and authorized branch pushes once it acquires a slot. No product work or perf run remains, and no gate is waived. The earlier focused SIGINT test passes with the launcher correction; the two complete gates still need to run.

J20 scheduling blocker: all measurements, comparisons, browser triage, source checks and perf cleanup are complete. The corrected full fast gates and final archive repack are waiting through the mandatory compute01 build-slot helper. Current queued PID: 2045649; log: ~/.cache/codex-scratch/j20/verify-both-signal-reset.log. This launcher alone has waited over 30 minutes; all four slot holders have changed while it has remained queued. The helper polls locks and has no FIFO queue. Orchestrator scheduling help is needed if a prompt handoff is required: reserve a slot for this already-queued command. I am leaving it queued and will finish the two gates, final packages, commits and authorized branch pushes once it acquires a slot. No product work or perf run remains, and no gate is waived. The earlier focused SIGINT test passes with the launcher correction; the two complete gates still need to run.

kayg commented 2026-10-31 20:01:27 +00:00

Author

Owner

Copy link

J20 acquired compute01 slot 4. Full run now passes the CLI SIGINT test, confirming the launcher correction. The next existing replay-size fixture failed before product startup: fixture_state('rr257') rejects its default/fallback Unix-socket path at 109 bytes once host PIDs have seven digits.

The helper already supports INFRATERNAL_AGENT_TEST_STATE_ROOT / INFRATERNAL_TEST_STATE_ROOT. Final complete gates use a fresh short /tmp/j20.XXXXXX root through those variables, removed on exit. Repository/product/test code is unchanged. Both full gates will rerun inside the same held build slot, so this will not re-enter the unfair queue. Original attempts are retained; no gate is waived. Final authoritative logs are verify-fast-short-root.log and .exit under each branch's J20 cache.

J20 acquired compute01 slot 4. Full run now passes the CLI SIGINT test, confirming the launcher correction. The next existing replay-size fixture failed before product startup: `fixture_state('rr257')` rejects its default/fallback Unix-socket path at 109 bytes once host PIDs have seven digits. The helper already supports `INFRATERNAL_AGENT_TEST_STATE_ROOT` / `INFRATERNAL_TEST_STATE_ROOT`. Final complete gates use a fresh short `/tmp/j20.XXXXXX` root through those variables, removed on exit. Repository/product/test code is unchanged. Both full gates will rerun inside the same held build slot, so this will not re-enter the unfair queue. Original attempts are retained; no gate is waived. Final authoritative logs are `verify-fast-short-root.log` and `.exit` under each branch's J20 cache.

kayg commented 2026-10-31 20:23:28 +00:00

Author

Owner

Copy link

J20 measurements and raw evidence are complete; research source is now preserved on the named branch as a WIP commit pending the final fast gate. No product code or verification code changed.

Both static target builds pass at the checked-in size caps. Full Rust gate attempts exposed fixture-environment requirements: the default fallback socket path reaches 109 bytes with seven-digit host PIDs; configuring /tmp fixes that length but is unsuitable for restart fixtures (world-writable ancestor admission refusal and cross-device hard links). The next full run uses a fresh short 0700 directory directly in claude's home, on the same filesystem as the test binary. Failed logs will be attached with the final run. No gate is waived.

The branches' non-research workspace trees are identical. The final gate runs unchanged scripts/verify fast in the h2h canonical worktree with both research slices present; provenance records both source manifests and common workspace hash. This is one shared full workspace gate, not two independent passes. Individual standalone kits already have separate build/perf evidence.

compute01 logs: ~/.cache/codex-scratch/j20/verify-common-home-coordinator.log, verify-fast-common.log, and retained verify-fast-common-tmp-root.log. The retry is queued through build-slot. Remaining: final gate, final kit manifest/binary identity checks, replace WIP history with atomic commits and post final heads/output. No perf/browser background jobs remain.

J20 measurements and raw evidence are complete; research source is now preserved on the named branch as a WIP commit pending the final fast gate. No product code or verification code changed. Both static target builds pass at the checked-in size caps. Full Rust gate attempts exposed fixture-environment requirements: the default fallback socket path reaches 109 bytes with seven-digit host PIDs; configuring `/tmp` fixes that length but is unsuitable for restart fixtures (world-writable ancestor admission refusal and cross-device hard links). The next full run uses a fresh short 0700 directory directly in claude's home, on the same filesystem as the test binary. Failed logs will be attached with the final run. No gate is waived. The branches' non-research workspace trees are identical. The final gate runs unchanged `scripts/verify fast` in the h2h canonical worktree with both research slices present; provenance records both source manifests and common workspace hash. This is one shared full workspace gate, not two independent passes. Individual standalone kits already have separate build/perf evidence. compute01 logs: `~/.cache/codex-scratch/j20/verify-common-home-coordinator.log`, `verify-fast-common.log`, and retained `verify-fast-common-tmp-root.log`. The retry is queued through build-slot. Remaining: final gate, final kit manifest/binary identity checks, replace WIP history with atomic commits and post final heads/output. No perf/browser background jobs remain.

kayg commented 2026-10-31 20:52:32 +00:00

Author

Owner

Copy link

J20 final-gate retry has waited over 30 minutes for compute01 build-slot (coordinator PID 3109436; ~/.cache/codex-scratch/j20/verify-common-home-coordinator.log). All four slots have active builds from other jobs; no J20 process retains a stale slot. No gate was bypassed.

Research is preserved at WIP heads research-head-to-head 9c29e5b75d and research-reverse-proxy ccdc7e6376. No source changes since that push. Two relevant restart checks executed in the existing workspace test binary under Nix with a private home root: each reports 1 passed; 0 failed. The full gate remains outstanding. Measurements/raw evidence are complete; perf/browser workloads are stopped and cleaned. The finished std02 target cache is removed per the brief.

J20 final-gate retry has waited over 30 minutes for compute01 build-slot (coordinator PID 3109436; `~/.cache/codex-scratch/j20/verify-common-home-coordinator.log`). All four slots have active builds from other jobs; no J20 process retains a stale slot. No gate was bypassed. Research is preserved at WIP heads `research-head-to-head` 9c29e5b75d87e1fa2f9692f6d8329de2b05028f5 and `research-reverse-proxy` ccdc7e6376836a3307cd81c504e30434d38a55fc. No source changes since that push. Two relevant restart checks executed in the existing workspace test binary under Nix with a private home root: each reports `1 passed; 0 failed`. The full gate remains outstanding. Measurements/raw evidence are complete; perf/browser workloads are stopped and cleaned. The finished std02 target cache is removed per the brief.

kayg referenced this issue 2026-10-31 21:22:36 +00:00

Head-to-head: infraternal mesh vs NetBird/Tailscale, infraternal sessions vs tmux/Zellij #260

kayg commented 2026-10-31 21:22:36 +00:00

Author

Owner

Copy link

J20 final verification is blocked on compute01 capacity: coordinator PID 3109436 has waited over an hour in build-slot. Log remains ~/.cache/codex-scratch/j20/verify-common-home-coordinator.log; no slot has been acquired. The queued command will run automatically if one becomes free. If the orchestrator can reserve the next slot for this final cached gate, that would unblock completion. No other job has been interrupted and no gate has been bypassed.

No source changes since the pushed WIP heads 9c29e5b75 (#260) and ccdc7e637 (#261). Browser triage, x86 kit builds, full perf tables/raw bundles and perf/browser cleanup are complete. Both static targets passed at their caps. Private-home-root focused restart checks each executed one test and passed. Remaining work is the full shared gate, final manifest/binary identity packaging, atomic commit finalization and final head/output comments. This is not a completed-gate claim.

J20 final verification is blocked on compute01 capacity: coordinator PID 3109436 has waited over an hour in build-slot. Log remains `~/.cache/codex-scratch/j20/verify-common-home-coordinator.log`; no slot has been acquired. The queued command will run automatically if one becomes free. If the orchestrator can reserve the next slot for this final cached gate, that would unblock completion. No other job has been interrupted and no gate has been bypassed. No source changes since the pushed WIP heads 9c29e5b75 (#260) and ccdc7e637 (#261). Browser triage, x86 kit builds, full perf tables/raw bundles and perf/browser cleanup are complete. Both static targets passed at their caps. Private-home-root focused restart checks each executed one test and passed. Remaining work is the full shared gate, final manifest/binary identity packaging, atomic commit finalization and final head/output comments. This is not a completed-gate claim.

kayg commented 2026-10-31 21:48:24 +00:00

Author

Owner

Copy link

J20 finished and pushed research-reverse-proxy at 7aa0c42036.

Full comparison table, CPU/RSS, TLS handshakes, reload findings and DNS-01: #261 (comment); complete raw bundle: #261 (comment). All proxy/ACME measurements ran through non-root perf-run. Steady HTTP/TLS had zero errors; rpxy reload failures are retained as nonzero measurement exits. The direct reference exposes the generator/backend ceiling. DNS-01 used local Pebble/stub DNS, excluding public propagation/provider latency. Prior size/licensing/provider constraints remain documented. The owner chooses the engine; no product integration or foundation choice is included.

Final reverse-proxy kit SHA-256: 75b62ca197b0ee01541550e5441c4d80bf5891d946f1f20eaf87a7ea574cae39.

Final verification ran unchanged scripts/verify fast through compute01 build-slot with jobs/threads limited to four, both research slices present, and a fresh private short home-directory fixture root. The branches have identical workspace tree entries outside research/ (SHA-256 f2adf3aefe5585f305a67e14ecacbe13c99f9f3dd822aa71e57f44ab6bb01418). This is one shared full workspace gate, not two independent passes. The attached bundle records both baseline heads, every research file hash, and proof that both final commits exactly match those tested sources. Individual standalone builds and perf evidence are separate.

Earlier failed attempts and launchers are retained in the bundle: inherited ignored SIGINT, an overlong default socket path, and an unsuitable /tmp fixture root. The final run changed only the supported fixture-root environment; no product/test/gate code was changed or test skipped. Static sizes are exactly 21,431,472 B x86_64 and 19,605,896 B aarch64 (both delta +0).

Verbatim final gate output excerpt (full output and exit file attached):

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+

 39 pass
 0 fail
 50325 expect() calls
Ran 39 tests across 10 files. [7.68s]
J20 common final gate exit=0

Final kits are on compute01 under ~/.cache/codex-scratch/j20/{h2h,rproxy}/final-kits/ and copied to perf VM ~/j20/final-kits/; both destination SHA-256 checks report OK. Their names retain the original baseline heads; SOURCE_DIFF.patch and the attached final-commit provenance identify the final harness sources. All 19 measured executables are byte-identical to the original measured kits; manifests verified. Build caches on compute01/std02 are removed, the temporary source union is removed, and no J20 background build/browser/perf work remains. Perf namespaces/qdiscs and modules were cleaned; original cleanup diffs remain in the measurement evidence.

J20 finished and pushed `research-reverse-proxy` at **7aa0c42036b312e88fea8768ae8540b38d740e02**. Full comparison table, CPU/RSS, TLS handshakes, reload findings and DNS-01: https://git.kayg.org/kayg/infraternal/issues/261#issuecomment-71101; complete raw bundle: https://git.kayg.org/kayg/infraternal/issues/261#issuecomment-71215. All proxy/ACME measurements ran through non-root `perf-run`. Steady HTTP/TLS had zero errors; rpxy reload failures are retained as nonzero measurement exits. The direct reference exposes the generator/backend ceiling. DNS-01 used local Pebble/stub DNS, excluding public propagation/provider latency. Prior size/licensing/provider constraints remain documented. The owner chooses the engine; no product integration or foundation choice is included. Final reverse-proxy kit SHA-256: `75b62ca197b0ee01541550e5441c4d80bf5891d946f1f20eaf87a7ea574cae39`. Final verification ran unchanged `scripts/verify fast` through compute01 build-slot with jobs/threads limited to four, both research slices present, and a fresh private short home-directory fixture root. The branches have identical workspace tree entries outside `research/` (SHA-256 `f2adf3aefe5585f305a67e14ecacbe13c99f9f3dd822aa71e57f44ab6bb01418`). This is **one shared full workspace gate**, not two independent passes. The attached bundle records both baseline heads, every research file hash, and proof that both final commits exactly match those tested sources. Individual standalone builds and perf evidence are separate. Earlier failed attempts and launchers are retained in the bundle: inherited ignored SIGINT, an overlong default socket path, and an unsuitable `/tmp` fixture root. The final run changed only the supported fixture-root environment; no product/test/gate code was changed or test skipped. Static sizes are exactly 21,431,472 B x86_64 and 19,605,896 B aarch64 (both delta +0). Verbatim final gate output excerpt (full output and exit file attached): ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | static builds | PASS | +---------------------------+--------------+ 39 pass 0 fail 50325 expect() calls Ran 39 tests across 10 files. [7.68s] J20 common final gate exit=0 ``` Final kits are on compute01 under `~/.cache/codex-scratch/j20/{h2h,rproxy}/final-kits/` and copied to perf VM `~/j20/final-kits/`; both destination SHA-256 checks report OK. Their names retain the original baseline heads; `SOURCE_DIFF.patch` and the attached final-commit provenance identify the final harness sources. All 19 measured executables are byte-identical to the original measured kits; manifests verified. Build caches on compute01/std02 are removed, the temporary source union is removed, and no J20 background build/browser/perf work remains. Perf namespaces/qdiscs and modules were cleaned; original cleanup diffs remain in the measurement evidence.

j20-final-gates.zip

256 KiB

kayg commented 2026-11-01 07:46:10 +00:00

Author

Owner

Copy link

R2 research started. Read-only research branch research-r2, base f4b61cd6c3f52078cad7573c5367d7a6e75f84f0 (origin/dev); also inspecting origin/signin-d311 for sign-in credentials. No implementation or new engine decision. Evidence and owner questions will follow.

R2 research started. Read-only research branch `research-r2`, base `f4b61cd6c3f52078cad7573c5367d7a6e75f84f0` (origin/dev); also inspecting `origin/signin-d311` for sign-in credentials. No implementation or new engine decision. Evidence and owner questions will follow.

kayg commented 2026-11-01 07:50:05 +00:00

Author

Owner

Copy link

R2 source finding: Sōzu 2.3.3 does not include an ACME client. The archived sozu-acme README says it was incorporated, but the versioned CHANGELOG records removal of the acme command on 2023-05-04; doc/upgrade/1.x-to-2.0.md still lists ACME automation as follow-up (HTTP-01 first, DNS-01 later). Evidence: github.com/sozu-proxy/sozu@106d3c8277 and https://github.com/sozu-proxy/sozu/blob/2.3.3/doc/upgrade/1.x-to-2.0.md . J20 comment 71101 establishes a separate instant-acme DNS-01 path, not built-in Sōzu ACME. Full maturity/UI/sidecar comparison follows.

R2 source finding: Sōzu 2.3.3 does not include an ACME client. The archived sozu-acme README says it was incorporated, but the versioned CHANGELOG records removal of the acme command on 2023-05-04; doc/upgrade/1.x-to-2.0.md still lists ACME automation as follow-up (HTTP-01 first, DNS-01 later). Evidence: https://github.com/sozu-proxy/sozu/commit/106d3c8277ae25b6f3f09d3e5a7f81c07a6ae612 and https://github.com/sozu-proxy/sozu/blob/2.3.3/doc/upgrade/1.x-to-2.0.md . J20 comment 71101 establishes a separate instant-acme DNS-01 path, not built-in Sōzu ACME. Full maturity/UI/sidecar comparison follows.

kayg commented 2026-11-01 07:57:56 +00:00

Author

Owner

Copy link

R2: recommendation and evidence for the proxy decision

Recommendation: keep Traefik as the engine, downloaded at a pinned version and supervised only on nodes exposing services. Build our own UI and configuration model over its file provider. Sozu is a credible production proxy and substantially leaner in J20's workload. It is a good alternative if always-present embedding and minimum memory outweigh the extra integration, certificate automation and middleware work. Pingora is the least aligned with D316: it supplies an engine framework, but leaves considerably more proxy-product behavior to us. This is a recommendation for kayg, not a foundation decision or implementation.

Research snapshot: 2026-11-01. Product source: origin/dev at f4b61cd6c3f52078cad7573c5367d7a6e75f84f0. Candidate versions match J20: Sozu 2.3.3, Pingora 0.9.0 and Traefik 3.7.17. No new benchmark was run. Sources below distinguish vendor production claims from our measurements.

Maturity, ownership and demonstrated scale

	Sozu	Pingora	Traefik Proxy
Age and ownership	Clever Cloud's Rust proxy; public repository created January 2017, copyright starts 2015. About a decade of lineage, with a much smaller ecosystem.	Cloudflare's Rust networking framework, described publicly in 2022 and open-sourced February 2024; public repository created May 2023.	Started by Emile Vauge in 2015, maintained by Traefik Labs and community. About eleven years of public product evolution.
Recent releases	2.1.0 June 5, 2.2.0 July 16, 2.2.1 August 28; 2.3.0 through 2.3.3 October 14–19, 2026. Active, but clustered rather than a fixed monthly promise.	0.7.0 January 30, 0.8.0 March 2, 0.8.1 June 4, 0.9.0 September 9, 2026. Fewer releases and still pre-1.0.	Stable 3.7.14 October 6, .15 October 15, .16 October 20, .17 October 30, 2026; parallel 2.11 maintenance releases. Frequent patches, including security work. 3.8.0-rc.1 is not the proposed pin.
Production evidence	Clever Cloud says it fronts every application on its platform, with 2.0 having had a long production run on cleverapps.io load balancers before release. This establishes real PaaS use; I found no independently reproducible fleet throughput or connection-count figure to quote.	Upstream reports over 40 million Internet requests/s across Cloudflare's deployment. That is fleet scale with Cloudflare's integration, not a promise for one embedded listener or our ingress configuration.	A vendor case study reports billions of requests/month at Rocket.Chat. This is evidence of broad production suitability, not an apples-to-apples performance comparison with Cloudflare.
Practical maturity assessment	Mature operational core, native hot configuration and upgrades; concentrated stewardship and fewer ready integrations.	Exceptional production engineering at its principal user; comparatively young public framework and substantial integration responsibilities.	Most complete ready-to-operate product of the three, with the broadest relevant provider/middleware surface.

Evidence: repository metadata and releases for Sozu, Sozu releases, Pingora releases, Traefik releases; Clever Cloud's 2.0 production account; Pingora 0.9.0 README, open-source announcement; Traefik founder's history, Rocket.Chat case study.

Security history, not a safety ranking

The GitHub advisory APIs returned 0 Sozu, 8 Pingora and 89 Traefik advisories in this snapshot. These are published repository advisories, not all vulnerabilities in transitive dependencies, and counts cannot rank safety: scope, exposure and disclosure practices differ.

• Sozu's zero published GitHub advisories is a visibility limit. Its 2.x changelog documents parser, command-channel, routing-regex, certificate-rotation and HTTP/2 flood hardening. The current release has signed checksum manifests and provenance for standalone binaries. Neither that nor Rust's memory safety proves parser correctness. Review the exact embedded dependency closure and follow upstream fixes.
• Pingora: CVE-2025-4366 request smuggling was fixed in 0.5.0; CVE-2025-8671 HTTP/2 MadeYouReset in 0.6.0. The March 2026 upgrade/HTTP-1.0 smuggling and cache-key advisories were fixed in 0.8.0. October disclosures cover HTTP/2 memory exhaustion (fixed 0.8.1), authority/Host disagreement and Connection-header forwarding (fixed 0.9.0). Cloudflare explicitly says its deployment position avoided some ingress-specific cases. Its fleet record does not remove our ingress review obligation.
• Traefik's history includes CVE-2024-45410 header manipulation, later HTTP/parser and authentication issues, and provider-specific authorization problems. October 2026 includes ForwardAuth connection-bound identity leakage, patched in 3.7.16, plus Kubernetes-provider and FastProxy findings. File-provider-only deployment avoids enabling unrelated providers, but does not make core HTTP or ForwardAuth findings irrelevant. A promptly serviced pin is essential.

Sources: Sozu versioned changelog, Sozu advisory API, signed release description; Pingora advisories, Cloudflare's ingress caveat, 2025 smuggling account; Traefik advisories, ForwardAuth fix, security policy.

Capabilities and the UI control surface

Area	Sozu 2.3.3	Pingora 0.9.0	Traefik 3.7.17
Configuration model	TOML listeners, clusters, frontends, backends and certificates; protobuf command types through a private Unix command socket, queries/events and saved state.	Rust service construction and callbacks, not a complete declarative proxy model or management API.	Static install configuration: entrypoints, providers, resolvers, observability. Dynamic configuration: HTTP/TCP/UDP routers, services, middlewares, TLS certificates/options/transports. YAML/TOML and discovery providers.
Dynamic changes	Native Add/RemoveBackend, frontend/listener operations and Add/ReplaceCertificate. Need desired/observed reconciliation and persistence; a batch is not automatically an all-or-nothing product transaction.	We supply route-table swaps, TLS resolvers, versioning and reconciliation. J20 implemented these in its demo. Process graceful upgrade is not a route management API.	File provider watches a directory; other providers include Docker, Kubernetes, HTTP and KV stores. Its dashboard/API exposes configuration/status/errors for inspection, not a CRUD write API for our UI. We write through a provider. Some static changes require restart.
Auth and policy	Built-in Basic auth, frontend header mutations/HSTS/redirects and connection protection. Not a Traefik-style catalog of composable middleware or a demonstrated general ForwardAuth integration. Its per-source connection cap is not an equivalent to a configurable HTTP request-rate policy.	Auth/header/rate decisions can be implemented with hooks and helper crates such as pingora-limits. This leaves security-sensitive glue and middleware behavior to us.	Built-in BasicAuth/DigestAuth, ForwardAuth, RateLimit, InFlightReq, IPAllowList, headers/CORS, redirects, path rewriting, retry and middleware chains. Use the OSS capability set; do not assume every feature shown in shared Hub/Enterprise docs is in MIT Proxy. Plugins add choices but also compatibility, license and supply-chain review.
Protocols	HTTP/1.1, HTTP/2, TLS/SNI, WebSocket upgrade, TCP, and UDP since 2.1. J20's h2c/gRPC-shaped/WebSocket matrix passed.	HTTP/1 and HTTP/2, WebSocket proxying, TCP/network primitives. Do not assume a turnkey UDP-router product. Our required h2c/gRPC-shaped/WebSocket matrix passed in J20.	HTTP/1, HTTP/2, WebSockets without a special middleware, gRPC/h2c, TCP/TLS and UDP routing.
Observability	Worker/backend metrics queried over the command socket, StatsD emission, structured access logs/events and optional TUI. Its OpenTelemetry feature passes W3C trace context; it does not create spans or export OTLP. A Prometheus surface needs an adapter/exporter.	Prometheus integration, logging and framework instrumentation; our service decides labels, access logs and management diagnostics.	Access/application logs, Prometheus and other metric exporters, tracing, health/ping, API/dashboard. Our UI should expose filtered operational state through the infraternal server, never a public insecure dashboard.

Sources: Sozu configuration, CLI and command operations, library/supervisor split; Pingora framework, proxy callback interface; Traefik file provider, API endpoints, middleware reference, observability.

ACME correction: Sozu 2.3.3 has no integrated ACME issuance/renewal client. The archived sozu-acme README describes an earlier CLI integration; upstream removed that command on 2023-05-04, and the 2.x migration document lists ACME automation as follow-up, HTTP-01 first and DNS-01 later. Loading an ACME-generated fullchain is not issuance support. Pingora also leaves issuance orchestration to the embedding application. Traefik has built-in ACME HTTP-01, TLS-ALPN-01 and DNS-01, provider integrations and renewal. ACME documentation.

J20's separate instant-acme 0.8.5 DNS-01 test issued two certificates in 1,789 ms, client peak RSS 4,988 KiB, standalone x86 executable 2,990,800 B. This used local Pebble/stub DNS with validation sleep disabled. It proves the client protocol path, not real provider credential handling, DNS propagation or production renewal reliability. Embedded Sozu would need that lifecycle and provider integration, but not our own proxy engine.

A nice UI is feasible with either engine. Our product model should describe a published service: node, hostnames/path match, upstream targets, TLS policy, auth policy, request/connection limits, header rules, enabled state and desired revision. The infraternal server authorizes and audits changes, then the node's infraternal agent reconciles them. The UI shows pending/applied/failed revision, certificate and renewal health, upstream health, traffic/errors and an explicit exposure control. Sozu gets translated socket commands; Traefik gets generated provider configuration; Pingora needs an application-owned implementation of that mapping. An engine's schema should not become our public API. This is a backend/control-plane sketch, not a visual design approval.

Managed, on-demand Traefik sketch and costs

1. Acquisition and trust. Fetch only an approved tuple (version, OS, architecture, archive SHA-256, executable SHA-256) from official GitHub release assets, or an explicitly configured verified offline cache. The inspected 3.7.17 assets contain platform archives and a checksum text file, but no detached checksum signature/certificate asset. A signed Git tag or GitHub verified commit is not a signature over the downloaded tarball. Therefore pin the reviewed hashes in an infraternal-signed component manifest and verify that manifest with our release trust, then verify archive and extracted executable before use. Do not trust a checksum fetched beside an arbitrary binary as independent authenticity. This extends our update trust model; it is not already implemented. The current product release verifier also caps payloads at 128 MiB (crates/infraternal-server/src/release_verify.rs), below J20’s 182,120,608-byte Traefik binary. Reuse its trust/verification principles, but do not claim its existing payload adapter accepts this sidecar unchanged.
2. Private install. Use a private component directory beneath the existing state/data root, versioned installs and a current pointer, 0700 directories and appropriately restricted config/secret files. Bound download/archive size and extraction paths, reject links/path traversal, stage on the destination filesystem, fsync and rename atomically. The first exposure operation shows downloading/verification/startup progress and remains unapplied on failure. No download at ordinary CLI/browser startup. Keep an offline import path.
3. Supervision. The local infraternal agent starts an owned child process, without a container engine dependency, passes a path to static config, captures sanitized logs, bounds restart backoff, checks process/ping readiness and restores the last known good config. Detect port conflicts and rootless low-port restrictions before claiming exposure succeeded. Linux privilege delegation and macOS service behavior need explicit integration work; do not grant broad root access just to bind a port. Supervision must decide whether exposure survives an infraternal agent restart; a child alone does not guarantee that.
4. Config. Enable only the file provider initially, watching its directory. Generate a complete dynamic YAML/TOML snapshot in a temporary file outside the watched directory, validate our supported schema, then atomically replace one published file. Observe provider errors and probe the intended routing revision before declaring success; rename is not an engine acknowledgement. Keep static config separate because entrypoint/resolver/provider changes can need a supervised restart. Do not mount a Docker socket or enable an insecure public API by default. Traefik's provider debounce means changes are asynchronous; J20 observed route changes up to 1,041 ms.
5. Certificates and secrets. Let Traefik own its ACME account and automatic DNS-01 renewal if that is the owner-approved custody model; keep acme.json private and backed up. Supply scoped DNS credentials through supported _FILE secret options, never argv or logs. Expose expiry/renewal failure in our UI. Credentials that require environment variables need an explicit custody choice, not accidental inheritance. Issuance retries must respect public CA/DNS limits. External certificate files remain an alternative if we deliberately centralize issuance.
6. Updates. Ship an approved component version/hash manifest, never resolve latest on the node. Separate download from activation, validate release notes/advisories and config compatibility, stage then restart/drain in a maintenance policy, retain the previous binary/config and roll back on failed readiness. File hot reload is not zero-downtime binary upgrade; one process cannot be assumed to hand listening sockets to the next. Emergency engine fixes need a maintained release path. Normal settings UI should show version/update status, not package-installation mechanics.

Cost	Traefik on demand	Embedded Sozu
Product binary/distribution	No embedded engine bytes, but downloader/supervisor/config-adapter code has an unmeasured product delta. J20's separate executable is 182,120,608 B, about 173.68 MiB, before keeping a rollback copy. Download compression/latency were not measured.	J20 measured +6,383,648 B x86, about 6.09 MiB, and +5,168,424 B ARM64. Every distributed infraternal binary pays this even on nodes without exposed services.
Idle/held RSS	93.30 MiB idle; 150.27 MiB at 1,000 connections per active proxy node. Zero proxy-process RSS on a node where it is not running.	Standalone research process: 8.19 MiB idle; 52.36 MiB at 1,000 connections. These are not measured incremental embedding RSS.
Fleet planning	With 100 installed nodes and 10 exposing services, 10 × 93.30 = 933 MiB idle proxy RSS; if all 100 expose, 9,330 MiB. Arithmetic, not a new scale test.	10 active instances × 8.19 = 81.9 MiB standalone reference; 100 = 819 MiB. Embedded distribution adds about 609 MiB across 100 x86 installations before A/B copies.
Latency/throughput	h1 c64: 32,956 req/s, p99 7.296 ms; h2 c64: 25,289 req/s, p99 7.393 ms.	h1 c64: 50,052 req/s, p99 3.161 ms; h2 c64: 29,444 req/s, p99 4.482 ms.
Reload	Zero failed requests in three trials; route/certificate changes 27 / 561 / 1,041 ms.	Zero failed requests in three trials; route 109 / 50 / 203 ms, certificate 108 / 49 / 203 ms.
License/platform fit	MIT. Official Linux amd64/arm64 and Darwin amd64/arm64 assets. Linux/macOS delivery is straightforward in principle; signing/quarantine, supervision and low-port behavior still need platform tests.	sozu-lib AGPL-3.0, command library LGPL-3.0; compatible with AGPL core in principle, but the commercial ee/ combination and static-link compliance need review. Published binary matrix is Linux; macOS embedding must be verified, not assumed from Rust portability.

All performance figures above come from J20 comment 71101: x86 perf VM, 5 s warmup + 30 s samples, two proxy CPUs and one shared backend/generator CPU. The direct h1 c64 ceiling was 63,506 req/s. One run/workload and three reload trials establish this workload, not universal capacity or multi-node scaling. Pingora's corresponding idle/1,000 RSS was 6.66/64.01 MiB, h1 c64 45,372 req/s/p99 2.981 ms, product delta +5,338,368 B. Its J20 integration also hit a Tokio 1.54.1 panic while the product's locked 1.53.3 worked. Sozu's supported supervisor/worker shape needs integration care; its testing worker entrypoint is not an established embedding lifecycle contract.

License evidence: Traefik MIT license, Sozu exact license and process structure, Pingora Apache-2.0.

Research verification and handoff

Read-only branch research-r2, base/head f4b61cd6c3f52078cad7573c5367d7a6e75f84f0; no repository files changed, no commit, push, deployment or new measurement. Reports live in issue comments and control-host evidence directory /home/kayg/.local/state/infraternal-orchestrator/r2/. Referenced branch signin-d311 was read at 16b728e212a783a34c38e229a7a077dc64bac7f6.

A VM scripts/verify fast request was submitted through compute01 build-slot, but the gate itself never started. Verbatim scheduler output:

build-slot: [08:50:06] all 4 slots busy; queue position 9, waiting...

The scheduler printed UTC (14:20:06 IST). I canceled my queued wrapper before handoff; no R2 target directory/build/server remains. No passing fast gate is claimed; repository verification remains outstanding. Evidence here is code/source inspection and J20's cited measurements, not new runtime verification. Reports were checked for consistent table columns and source pinning.

Owner grill

Recommended choice: approve pinned, verified, on-demand Traefik with our UI/model and a managed child on service-exposing nodes. Does D316's “every client runs it” mean every installation has the ability to expose services, with the process present only while exposure is configured? That accepts roughly 93 MiB idle RSS per active proxy node and a separately managed binary in exchange for mature ACME/providers/middleware. Or do you mean an engine must be physically embedded in every binary and available without any component download, making Sozu plus separately managed DNS-01 the preferred trade-off despite additional integration and ee/ licensing review? If the footprint is the deciding factor, how many nodes should expose services simultaneously and what idle proxy RSS budget should we enforce?

## R2: recommendation and evidence for the proxy decision **Recommendation: keep Traefik as the engine, downloaded at a pinned version and supervised only on nodes exposing services. Build our own UI and configuration model over its file provider.** Sozu is a credible production proxy and substantially leaner in J20's workload. It is a good alternative if always-present embedding and minimum memory outweigh the extra integration, certificate automation and middleware work. Pingora is the least aligned with D316: it supplies an engine framework, but leaves considerably more proxy-product behavior to us. This is a recommendation for kayg, not a foundation decision or implementation. Research snapshot: 2026-11-01. Product source: `origin/dev` at `f4b61cd6c3f52078cad7573c5367d7a6e75f84f0`. Candidate versions match J20: Sozu 2.3.3, Pingora 0.9.0 and Traefik 3.7.17. No new benchmark was run. Sources below distinguish vendor production claims from our measurements. ### Maturity, ownership and demonstrated scale | | Sozu | Pingora | Traefik Proxy | |---|---|---|---| | Age and ownership | Clever Cloud's Rust proxy; public repository created January 2017, copyright starts 2015. About a decade of lineage, with a much smaller ecosystem. | Cloudflare's Rust networking framework, described publicly in 2022 and open-sourced February 2024; public repository created May 2023. | Started by Emile Vauge in 2015, maintained by Traefik Labs and community. About eleven years of public product evolution. | | Recent releases | 2.1.0 June 5, 2.2.0 July 16, 2.2.1 August 28; 2.3.0 through 2.3.3 October 14–19, 2026. Active, but clustered rather than a fixed monthly promise. | 0.7.0 January 30, 0.8.0 March 2, 0.8.1 June 4, 0.9.0 September 9, 2026. Fewer releases and still pre-1.0. | Stable 3.7.14 October 6, .15 October 15, .16 October 20, .17 October 30, 2026; parallel 2.11 maintenance releases. Frequent patches, including security work. 3.8.0-rc.1 is not the proposed pin. | | Production evidence | Clever Cloud says it fronts every application on its platform, with 2.0 having had a long production run on cleverapps.io load balancers before release. This establishes real PaaS use; I found no independently reproducible fleet throughput or connection-count figure to quote. | Upstream reports over 40 million Internet requests/s across Cloudflare's deployment. That is fleet scale with Cloudflare's integration, not a promise for one embedded listener or our ingress configuration. | A vendor case study reports billions of requests/month at Rocket.Chat. This is evidence of broad production suitability, not an apples-to-apples performance comparison with Cloudflare. | | Practical maturity assessment | Mature operational core, native hot configuration and upgrades; concentrated stewardship and fewer ready integrations. | Exceptional production engineering at its principal user; comparatively young public framework and substantial integration responsibilities. | Most complete ready-to-operate product of the three, with the broadest relevant provider/middleware surface. | Evidence: repository metadata and releases for [Sozu](https://api.github.com/repos/sozu-proxy/sozu), [Sozu releases](https://github.com/sozu-proxy/sozu/releases), [Pingora releases](https://github.com/cloudflare/pingora/releases), [Traefik releases](https://github.com/traefik/traefik/releases); Clever Cloud's [2.0 production account](https://www.clever.cloud/blog/engineering/2026/05/29/sozu-2-0-reverse-proxy-programmable-edge/); [Pingora 0.9.0 README](https://github.com/cloudflare/pingora/blob/0.9.0/README.md), [open-source announcement](https://blog.cloudflare.com/pingora-open-source/); [Traefik founder's history](https://community.traefik.io/t/traefik-labs-incubating-the-future-of-cloud-native-networking/7816), [Rocket.Chat case study](https://traefik.io/blog/case-study-rocket-chat-deploys-traefik-to-manage-unified-communications-at-scale). ### Security history, not a safety ranking The GitHub advisory APIs returned **0 Sozu, 8 Pingora and 89 Traefik advisories** in this snapshot. These are published repository advisories, not all vulnerabilities in transitive dependencies, and counts cannot rank safety: scope, exposure and disclosure practices differ. * Sozu's zero published GitHub advisories is a visibility limit. Its 2.x changelog documents parser, command-channel, routing-regex, certificate-rotation and HTTP/2 flood hardening. The current release has signed checksum manifests and provenance for standalone binaries. Neither that nor Rust's memory safety proves parser correctness. Review the exact embedded dependency closure and follow upstream fixes. * Pingora: CVE-2025-4366 request smuggling was fixed in 0.5.0; CVE-2025-8671 HTTP/2 MadeYouReset in 0.6.0. The March 2026 upgrade/HTTP-1.0 smuggling and cache-key advisories were fixed in 0.8.0. October disclosures cover HTTP/2 memory exhaustion (fixed 0.8.1), authority/Host disagreement and Connection-header forwarding (fixed 0.9.0). Cloudflare explicitly says its deployment position avoided some ingress-specific cases. Its fleet record does not remove our ingress review obligation. * Traefik's history includes CVE-2024-45410 header manipulation, later HTTP/parser and authentication issues, and provider-specific authorization problems. October 2026 includes ForwardAuth connection-bound identity leakage, patched in 3.7.16, plus Kubernetes-provider and FastProxy findings. File-provider-only deployment avoids enabling unrelated providers, but does not make core HTTP or ForwardAuth findings irrelevant. A promptly serviced pin is essential. Sources: [Sozu versioned changelog](https://github.com/sozu-proxy/sozu/blob/2.3.3/CHANGELOG.md), [Sozu advisory API](https://api.github.com/repos/sozu-proxy/sozu/security-advisories), [signed release description](https://github.com/sozu-proxy/sozu/blob/2.3.3/README.md); [Pingora advisories](https://github.com/cloudflare/pingora/security/advisories), [Cloudflare's ingress caveat](https://blog.cloudflare.com/pingora-oss-smuggling-vulnerabilities/), [2025 smuggling account](https://blog.cloudflare.com/resolving-a-request-smuggling-vulnerability-in-pingora/); [Traefik advisories](https://github.com/traefik/traefik/security/advisories), [ForwardAuth fix](https://github.com/traefik/traefik/security/advisories/GHSA-hrpp-p4fc-wwf8), [security policy](https://traefik.io/security). ### Capabilities and the UI control surface | Area | Sozu 2.3.3 | Pingora 0.9.0 | Traefik 3.7.17 | |---|---|---|---| | Configuration model | TOML listeners, clusters, frontends, backends and certificates; protobuf command types through a private Unix command socket, queries/events and saved state. | Rust service construction and callbacks, not a complete declarative proxy model or management API. | Static install configuration: entrypoints, providers, resolvers, observability. Dynamic configuration: HTTP/TCP/UDP routers, services, middlewares, TLS certificates/options/transports. YAML/TOML and discovery providers. | | Dynamic changes | Native Add/RemoveBackend, frontend/listener operations and Add/ReplaceCertificate. Need desired/observed reconciliation and persistence; a batch is not automatically an all-or-nothing product transaction. | We supply route-table swaps, TLS resolvers, versioning and reconciliation. J20 implemented these in its demo. Process graceful upgrade is not a route management API. | File provider watches a directory; other providers include Docker, Kubernetes, HTTP and KV stores. Its dashboard/API exposes configuration/status/errors for inspection, **not a CRUD write API for our UI**. We write through a provider. Some static changes require restart. | | Auth and policy | Built-in Basic auth, frontend header mutations/HSTS/redirects and connection protection. Not a Traefik-style catalog of composable middleware or a demonstrated general ForwardAuth integration. Its per-source connection cap is not an equivalent to a configurable HTTP request-rate policy. | Auth/header/rate decisions can be implemented with hooks and helper crates such as pingora-limits. This leaves security-sensitive glue and middleware behavior to us. | Built-in BasicAuth/DigestAuth, ForwardAuth, RateLimit, InFlightReq, IPAllowList, headers/CORS, redirects, path rewriting, retry and middleware chains. Use the OSS capability set; do not assume every feature shown in shared Hub/Enterprise docs is in MIT Proxy. Plugins add choices but also compatibility, license and supply-chain review. | | Protocols | HTTP/1.1, HTTP/2, TLS/SNI, WebSocket upgrade, TCP, and UDP since 2.1. J20's h2c/gRPC-shaped/WebSocket matrix passed. | HTTP/1 and HTTP/2, WebSocket proxying, TCP/network primitives. Do not assume a turnkey UDP-router product. Our required h2c/gRPC-shaped/WebSocket matrix passed in J20. | HTTP/1, HTTP/2, WebSockets without a special middleware, gRPC/h2c, TCP/TLS and UDP routing. | | Observability | Worker/backend metrics queried over the command socket, StatsD emission, structured access logs/events and optional TUI. Its OpenTelemetry feature passes W3C trace context; it does **not** create spans or export OTLP. A Prometheus surface needs an adapter/exporter. | Prometheus integration, logging and framework instrumentation; our service decides labels, access logs and management diagnostics. | Access/application logs, Prometheus and other metric exporters, tracing, health/ping, API/dashboard. Our UI should expose filtered operational state through the infraternal server, never a public insecure dashboard. | Sources: [Sozu configuration](https://github.com/sozu-proxy/sozu/blob/2.3.3/doc/configure.md), [CLI and command operations](https://github.com/sozu-proxy/sozu/blob/2.3.3/doc/configure_cli.md), [library/supervisor split](https://github.com/sozu-proxy/sozu/blob/2.3.3/README.md); [Pingora framework](https://github.com/cloudflare/pingora/tree/0.9.0), [proxy callback interface](https://docs.rs/pingora-proxy/0.9.0/pingora_proxy/trait.ProxyHttp.html); Traefik [file provider](https://doc.traefik.io/traefik/reference/install-configuration/providers/others/file/), [API endpoints](https://doc.traefik.io/traefik/reference/install-configuration/api-dashboard/), [middleware reference](https://github.com/traefik/traefik/tree/v3.7.17/pkg/middlewares), [observability](https://doc.traefik.io/traefik/observe/overview/). **ACME correction:** Sozu 2.3.3 has no integrated ACME issuance/renewal client. The archived sozu-acme README describes an earlier CLI integration; upstream [removed that command on 2023-05-04](https://github.com/sozu-proxy/sozu/commit/106d3c8277ae25b6f3f09d3e5a7f81c07a6ae612), and the [2.x migration document](https://github.com/sozu-proxy/sozu/blob/2.3.3/doc/upgrade/1.x-to-2.0.md) lists ACME automation as follow-up, HTTP-01 first and DNS-01 later. Loading an ACME-generated fullchain is not issuance support. Pingora also leaves issuance orchestration to the embedding application. Traefik has built-in ACME HTTP-01, TLS-ALPN-01 and DNS-01, provider integrations and renewal. [ACME documentation](https://doc.traefik.io/traefik/v3.7/reference/install-configuration/tls/certificate-resolvers/acme/). J20's separate `instant-acme` 0.8.5 DNS-01 test issued two certificates in **1,789 ms**, client peak RSS **4,988 KiB**, standalone x86 executable **2,990,800 B**. This used local Pebble/stub DNS with validation sleep disabled. It proves the client protocol path, not real provider credential handling, DNS propagation or production renewal reliability. Embedded Sozu would need that lifecycle and provider integration, but **not our own proxy engine**. **A nice UI is feasible with either engine.** Our product model should describe a published service: node, hostnames/path match, upstream targets, TLS policy, auth policy, request/connection limits, header rules, enabled state and desired revision. The infraternal server authorizes and audits changes, then the node's infraternal agent reconciles them. The UI shows pending/applied/failed revision, certificate and renewal health, upstream health, traffic/errors and an explicit exposure control. Sozu gets translated socket commands; Traefik gets generated provider configuration; Pingora needs an application-owned implementation of that mapping. An engine's schema should not become our public API. This is a backend/control-plane sketch, not a visual design approval. ### Managed, on-demand Traefik sketch and costs 1. **Acquisition and trust.** Fetch only an approved tuple `(version, OS, architecture, archive SHA-256, executable SHA-256)` from official [GitHub release assets](https://github.com/traefik/traefik/releases/tag/v3.7.17), or an explicitly configured verified offline cache. The inspected 3.7.17 assets contain platform archives and a checksum text file, but no detached checksum signature/certificate asset. A signed Git tag or GitHub verified commit is not a signature over the downloaded tarball. Therefore pin the reviewed hashes in an infraternal-signed component manifest and verify that manifest with our release trust, then verify archive and extracted executable before use. Do not trust a checksum fetched beside an arbitrary binary as independent authenticity. This extends our update trust model; it is not already implemented. The current product release verifier also caps payloads at 128 MiB (`crates/infraternal-server/src/release_verify.rs`), below J20’s 182,120,608-byte Traefik binary. Reuse its trust/verification principles, but do not claim its existing payload adapter accepts this sidecar unchanged. 2. **Private install.** Use a private component directory beneath the existing state/data root, versioned installs and a current pointer, 0700 directories and appropriately restricted config/secret files. Bound download/archive size and extraction paths, reject links/path traversal, stage on the destination filesystem, fsync and rename atomically. The first exposure operation shows downloading/verification/startup progress and remains unapplied on failure. No download at ordinary CLI/browser startup. Keep an offline import path. 3. **Supervision.** The local infraternal agent starts an owned child process, without a container engine dependency, passes a path to static config, captures sanitized logs, bounds restart backoff, checks process/ping readiness and restores the last known good config. Detect port conflicts and rootless low-port restrictions before claiming exposure succeeded. Linux privilege delegation and macOS service behavior need explicit integration work; do not grant broad root access just to bind a port. Supervision must decide whether exposure survives an infraternal agent restart; a child alone does not guarantee that. 4. **Config.** Enable only the file provider initially, watching its directory. Generate a complete dynamic YAML/TOML snapshot in a temporary file outside the watched directory, validate our supported schema, then atomically replace one published file. Observe provider errors and probe the intended routing revision before declaring success; rename is not an engine acknowledgement. Keep static config separate because entrypoint/resolver/provider changes can need a supervised restart. Do not mount a Docker socket or enable an insecure public API by default. Traefik's provider debounce means changes are asynchronous; J20 observed route changes up to 1,041 ms. 5. **Certificates and secrets.** Let Traefik own its ACME account and automatic DNS-01 renewal if that is the owner-approved custody model; keep `acme.json` private and backed up. Supply scoped DNS credentials through supported `_FILE` secret options, never argv or logs. Expose expiry/renewal failure in our UI. Credentials that require environment variables need an explicit custody choice, not accidental inheritance. Issuance retries must respect public CA/DNS limits. External certificate files remain an alternative if we deliberately centralize issuance. 6. **Updates.** Ship an approved component version/hash manifest, never resolve `latest` on the node. Separate download from activation, validate release notes/advisories and config compatibility, stage then restart/drain in a maintenance policy, retain the previous binary/config and roll back on failed readiness. File hot reload is not zero-downtime binary upgrade; one process cannot be assumed to hand listening sockets to the next. Emergency engine fixes need a maintained release path. Normal settings UI should show version/update status, not package-installation mechanics. | Cost | Traefik on demand | Embedded Sozu | |---|---|---| | Product binary/distribution | No embedded engine bytes, but downloader/supervisor/config-adapter code has an **unmeasured** product delta. J20's separate executable is 182,120,608 B, about 173.68 MiB, before keeping a rollback copy. Download compression/latency were not measured. | J20 measured +6,383,648 B x86, about 6.09 MiB, and +5,168,424 B ARM64. Every distributed infraternal binary pays this even on nodes without exposed services. | | Idle/held RSS | 93.30 MiB idle; 150.27 MiB at 1,000 connections per active proxy node. Zero proxy-process RSS on a node where it is not running. | Standalone research process: 8.19 MiB idle; 52.36 MiB at 1,000 connections. These are not measured incremental embedding RSS. | | Fleet planning | With 100 installed nodes and 10 exposing services, 10 × 93.30 = 933 MiB idle proxy RSS; if all 100 expose, 9,330 MiB. Arithmetic, not a new scale test. | 10 active instances × 8.19 = 81.9 MiB standalone reference; 100 = 819 MiB. Embedded distribution adds about 609 MiB across 100 x86 installations before A/B copies. | | Latency/throughput | h1 c64: 32,956 req/s, p99 7.296 ms; h2 c64: 25,289 req/s, p99 7.393 ms. | h1 c64: 50,052 req/s, p99 3.161 ms; h2 c64: 29,444 req/s, p99 4.482 ms. | | Reload | Zero failed requests in three trials; route/certificate changes 27 / 561 / 1,041 ms. | Zero failed requests in three trials; route 109 / 50 / 203 ms, certificate 108 / 49 / 203 ms. | | License/platform fit | MIT. Official Linux amd64/arm64 and Darwin amd64/arm64 assets. Linux/macOS delivery is straightforward in principle; signing/quarantine, supervision and low-port behavior still need platform tests. | sozu-lib AGPL-3.0, command library LGPL-3.0; compatible with AGPL core in principle, but the commercial `ee/` combination and static-link compliance need review. Published binary matrix is Linux; macOS embedding must be verified, not assumed from Rust portability. | All performance figures above come from [J20 comment 71101](https://git.kayg.org/kayg/infraternal/issues/261#issuecomment-71101): x86 perf VM, 5 s warmup + 30 s samples, two proxy CPUs and one shared backend/generator CPU. The direct h1 c64 ceiling was 63,506 req/s. One run/workload and three reload trials establish this workload, not universal capacity or multi-node scaling. Pingora's corresponding idle/1,000 RSS was 6.66/64.01 MiB, h1 c64 45,372 req/s/p99 2.981 ms, product delta +5,338,368 B. Its J20 integration also hit a Tokio 1.54.1 panic while the product's locked 1.53.3 worked. Sozu's supported supervisor/worker shape needs integration care; its testing worker entrypoint is not an established embedding lifecycle contract. License evidence: [Traefik MIT license](https://github.com/traefik/traefik/blob/v3.7.17/LICENSE.md), [Sozu exact license and process structure](https://github.com/sozu-proxy/sozu/blob/2.3.3/README.md), [Pingora Apache-2.0](https://github.com/cloudflare/pingora/blob/0.9.0/LICENSE). ### Research verification and handoff Read-only branch `research-r2`, base/head `f4b61cd6c3f52078cad7573c5367d7a6e75f84f0`; no repository files changed, no commit, push, deployment or new measurement. Reports live in issue comments and control-host evidence directory `/home/kayg/.local/state/infraternal-orchestrator/r2/`. Referenced branch `signin-d311` was read at `16b728e212a783a34c38e229a7a077dc64bac7f6`. A VM `scripts/verify fast` request was submitted through compute01 `build-slot`, but the gate itself never started. Verbatim scheduler output: ```text build-slot: [08:50:06] all 4 slots busy; queue position 9, waiting... ``` The scheduler printed UTC (14:20:06 IST). I canceled my queued wrapper before handoff; no R2 target directory/build/server remains. **No passing fast gate is claimed; repository verification remains outstanding.** Evidence here is code/source inspection and J20's cited measurements, not new runtime verification. Reports were checked for consistent table columns and source pinning. ### Owner grill **Recommended choice: approve pinned, verified, on-demand Traefik with our UI/model and a managed child on service-exposing nodes.** Does D316's “every client runs it” mean every installation has the ability to expose services, with the process present only while exposure is configured? That accepts roughly 93 MiB idle RSS per active proxy node and a separately managed binary in exchange for mature ACME/providers/middleware. Or do you mean an engine must be physically embedded in every binary and available without any component download, making Sozu plus separately managed DNS-01 the preferred trade-off despite additional integration and `ee/` licensing review? If the footprint is the deciding factor, how many nodes should expose services simultaneously and what idle proxy RSS budget should we enforce?

kayg referenced this issue 2026-11-01 07:57:56 +00:00

infraternal sign-in <server> (D311) #269

kayg commented 2026-11-01 08:00:22 +00:00

Author

Owner

Copy link

Owner decision (kayg, 2026-11-01): reverse-proxy engine = Traefik, downloaded on demand (R2 recommendation, comment 72433): a pinned, checksum/signature-verified Traefik binary, run as a supervised child only on nodes that expose services, configured through our own model and UI (Traefik file provider), DNS-01 via the Traefik ACME resolver. Owner: "I feel like this would be the most long-term option as well because there's already a large community around it and we can build around it." Rejected: embedding Sozu (no built-in ACME, smaller ecosystem), Pingora (pre-1.0 framework), rpxy (drops requests on reload). Next: an implementation plan slice (download/verify/update, supervision, config model, UI) for owner critique before code.

**Owner decision (kayg, 2026-11-01): reverse-proxy engine = Traefik, downloaded on demand** (R2 recommendation, comment 72433): a pinned, checksum/signature-verified Traefik binary, run as a supervised child only on nodes that expose services, configured through our own model and UI (Traefik file provider), DNS-01 via the Traefik ACME resolver. Owner: "I feel like this would be the most long-term option as well because there's already a large community around it and we can build around it." Rejected: embedding Sozu (no built-in ACME, smaller ecosystem), Pingora (pre-1.0 framework), rpxy (drops requests on reload). Next: an implementation plan slice (download/verify/update, supervision, config model, UI) for owner critique before code.

kayg commented 2026-11-01 08:06:50 +00:00

Author

Owner

Copy link

R3 started: Caddy versus Traefik paired measurement, branch research-reverse-proxy, base 7aa0c42036b312e88fea8768ae8540b38d740e02. Reusing J20 binaries and scenarios, all measurements through perf-run. Stock Caddy and a pinned Cloudflare-module build will each get idle/warm/1,000-connection RSS rows to verify the owner’s 17 MiB question. Compute builds through build-slot; no foundation decision or product integration. Evidence: control host ~/.local/state/infraternal-orchestrator/r3/, compute ~/worktrees/r3, perf ~/r3. Will push harness changes, retain raw receipts, and remove owned perf copies/targets.

R3 started: Caddy versus Traefik paired measurement, branch `research-reverse-proxy`, base `7aa0c42036b312e88fea8768ae8540b38d740e02`. Reusing J20 binaries and scenarios, all measurements through perf-run. Stock Caddy and a pinned Cloudflare-module build will each get idle/warm/1,000-connection RSS rows to verify the owner’s 17 MiB question. Compute builds through build-slot; no foundation decision or product integration. Evidence: control host `~/.local/state/infraternal-orchestrator/r3/`, compute `~/worktrees/r3`, perf `~/r3`. Will push harness changes, retain raw receipts, and remove owned perf copies/targets.

kayg referenced this issue 2026-11-01 08:11:39 +00:00

Reverse proxy: Traefik on demand — implementation plan #281

kayg referenced this issue 2026-11-01 08:11:39 +00:00

Reverse proxy: Traefik on demand — implementation plan #281

kayg commented 2026-11-01 08:12:41 +00:00

Author

Owner

Copy link

R3 first finding, provisional until paired runs finish: pinned stock Caddy 2.11.7 measured 50,152 KiB (48.98 MiB) idle, 50,860 KiB (49.67 MiB) warm, 91,720 KiB (89.57 MiB) at 1,000/1,000 successful TLS keep-alive connections, through perf-run on backup01 using J20’s exact binaries/scenarios and 2 proxy CPUs + 1 shared backend/generator CPU. The 17 MiB figure is not reproduced for this release/workload. Stock executable is 51,953,826 B. Basic matrix including WebSocket/h2/h2c/trailers is 9/9. Full load/reload phases are in progress; these are not final paired conclusions.

Caddy custom build pins: 2.11.7, xcaddy 0.4.7, Cloudflare module 0.2.4. Two clean build directories will test reproducibility; a third separate test-only binary adds Pebble TXT management, keeping that test module out of the Cloudflare memory rows. Official release checksum manifests use SHA-512 (not SHA-256); release assets include Sigstore signatures and certificates. Caddy’s reverse_proxy documentation requires stream_close_delay to preserve WebSockets across config unload, so the managed configuration uses 5 minutes and will be compared with the default behavior.

Compute01 custom build is queued behind 9 tickets (wrapper PID 484565, ~/worktrees/r3/build.log). The unchanged required product gate is prepared for the same acquired slot, using one target directory ~/worktrees/r3/target and private short home fixture roots. No queue bypass, alternate build host, product implementation or foundation decision. Raw current perf evidence: ~/r3/rproxy-kit-x86_64/results/ on backup01. Sources and exact pins will accompany the final side-by-side table.

R3 first finding, provisional until paired runs finish: pinned stock Caddy 2.11.7 measured **50,152 KiB (48.98 MiB) idle**, **50,860 KiB (49.67 MiB) warm**, **91,720 KiB (89.57 MiB) at 1,000/1,000 successful TLS keep-alive connections**, through perf-run on backup01 using J20’s exact binaries/scenarios and 2 proxy CPUs + 1 shared backend/generator CPU. The 17 MiB figure is not reproduced for this release/workload. Stock executable is 51,953,826 B. Basic matrix including WebSocket/h2/h2c/trailers is 9/9. Full load/reload phases are in progress; these are not final paired conclusions. Caddy custom build pins: 2.11.7, xcaddy 0.4.7, Cloudflare module 0.2.4. Two clean build directories will test reproducibility; a third separate test-only binary adds Pebble TXT management, keeping that test module out of the Cloudflare memory rows. Official release checksum manifests use SHA-512 (not SHA-256); release assets include Sigstore signatures and certificates. Caddy’s reverse_proxy documentation requires stream_close_delay to preserve WebSockets across config unload, so the managed configuration uses 5 minutes and will be compared with the default behavior. Compute01 custom build is queued behind 9 tickets (wrapper PID 484565, `~/worktrees/r3/build.log`). The unchanged required product gate is prepared for the same acquired slot, using one target directory `~/worktrees/r3/target` and private short home fixture roots. No queue bypass, alternate build host, product implementation or foundation decision. Raw current perf evidence: `~/r3/rproxy-kit-x86_64/results/` on backup01. Sources and exact pins will accompany the final side-by-side table.

kayg commented 2026-11-01 08:16:12 +00:00

Author

Owner

Copy link

R3 reload finding: stock Caddy 2.11.7 completed route + certificate switches at 25 / 16 / 13 ms, but the common checker observed 4 / 2 / 3 failed loop requests, and the existing h1 keep-alive and h2 connection were closed in all three trials. In-flight slow requests completed; WebSockets survived with explicit 5-minute stream_close_delay. The final basic matrix remained 9/9. This is not a zero-drop result, and it will be retained as a nonzero case exit. The local /load adapter returned 200; source and repeat trials will distinguish engine reload semantics from harness effects.

The initial staging run also caught a harness-operation mistake: I updated run-all.sh while its parent shell was still reading it, causing a post-measurement shell parse error and expected kit-file state diffs. I am preserving that attempt and will freeze kit sources before final alternating runs, then repeat stock Caddy. Its workload data is provisional and will not silently replace a clean final run. No product code changed. Custom build is still queued through compute01 build-slot.

R3 reload finding: stock Caddy 2.11.7 completed route + certificate switches at **25 / 16 / 13 ms**, but the common checker observed **4 / 2 / 3 failed loop requests**, and the existing h1 keep-alive and h2 connection were closed in all three trials. In-flight slow requests completed; WebSockets survived with explicit 5-minute `stream_close_delay`. The final basic matrix remained 9/9. This is not a zero-drop result, and it will be retained as a nonzero case exit. The local /load adapter returned 200; source and repeat trials will distinguish engine reload semantics from harness effects. The initial staging run also caught a harness-operation mistake: I updated run-all.sh while its parent shell was still reading it, causing a post-measurement shell parse error and expected kit-file state diffs. I am preserving that attempt and will freeze kit sources before final alternating runs, then repeat stock Caddy. Its workload data is provisional and will not silently replace a clean final run. No product code changed. Custom build is still queued through compute01 build-slot.

kayg commented 2026-11-01 08:24:26 +00:00

Author

Owner

Copy link

R3 additional runtime evidence: POST /load invalid-module input returned 400 and retained the exact exported previous config and live route; valid JSON changed the route; administration over a Unix socket below a 0700 directory worked; admin.disabled: true removed both admin endpoints while the HTTP listener kept serving. Perf wrapper exit 0, STATE DIFF EMPTY. Default Caddy stream policy was separately checked: all three reload trials closed the WebSocket (as documented), while the configured 5-minute stream delay preserved it. Neither case preserved old h1/h2 connections in the initial trials. A clean, frozen-source full stock/Traefik pair is running now, after a new direct baseline (h1 c64 56,451 req/s vs J20’s historical 63,506, so old timing rows will be labelled historical).

Custom builds remain queued, PID 484565, ~/worktrees/r3/build.log. Go was changed to 1.26.8, matching stock Caddy’s inspected build info, rather than introducing a Go-version confound into module footprint. No real DNS credential was supplied. WIP harness pushed at ddeca1887; pending report and final source updates are in the isolated control worktree ~/Developer/Worktrees/infraternal-r3.

R3 additional runtime evidence: `POST /load` invalid-module input returned 400 and retained the exact exported previous config and live route; valid JSON changed the route; administration over a Unix socket below a 0700 directory worked; `admin.disabled: true` removed both admin endpoints while the HTTP listener kept serving. Perf wrapper exit 0, STATE DIFF EMPTY. Default Caddy stream policy was separately checked: all three reload trials closed the WebSocket (as documented), while the configured 5-minute stream delay preserved it. Neither case preserved old h1/h2 connections in the initial trials. A clean, frozen-source full stock/Traefik pair is running now, after a new direct baseline (h1 c64 56,451 req/s vs J20’s historical 63,506, so old timing rows will be labelled historical). Custom builds remain queued, PID 484565, `~/worktrees/r3/build.log`. Go was changed to **1.26.8**, matching stock Caddy’s inspected build info, rather than introducing a Go-version confound into module footprint. No real DNS credential was supplied. WIP harness pushed at `ddeca1887`; pending report and final source updates are in the isolated control worktree `~/Developer/Worktrees/infraternal-r3`.

kayg commented 2026-11-01 08:29:04 +00:00

Author

Owner

Copy link

R3 clean stock repeat: 49,308 KiB idle; the old 17 MiB estimate remains unconfirmed. API/default-WebSocket diagnostics passed their intended checks with empty cleanup diffs; default WebSocket-close behavior is confirmed. The first frozen stock/Traefik pair is finishing/queued on the perf lock. I moved the Caddy route-to-API research watcher to reload phase only, pinned to the shared backend/generator CPU, and will use the repeated final pair from that source so steady CPU/RSS/load have no helper-process confound. Original runs remain preserved.

Custom build wrapper PID 484565 has queued about 20 minutes; four earlier tickets remain in front. It will build Cloudflare Caddy twice with Go 1.26.8 (matching stock), compare hashes, then build the separate Pebble adapter binary and run the required repository fast gate in the same held slot. No queue bypass or product code change. Target: ~/worktrees/r3/target. Caddy source confirms its reload stop path invokes Go HTTP Server.Shutdown; shutdown_delay does not preserve reused listeners’ old idle HTTP connections. Final report will distinguish this graceful behavior from an actual listener outage, and will not hide common-checker failures with retries.

R3 clean stock repeat: 49,308 KiB idle; the old 17 MiB estimate remains unconfirmed. API/default-WebSocket diagnostics passed their intended checks with empty cleanup diffs; default WebSocket-close behavior is confirmed. The first frozen stock/Traefik pair is finishing/queued on the perf lock. I moved the Caddy route-to-API research watcher to **reload phase only**, pinned to the shared backend/generator CPU, and will use the repeated final pair from that source so steady CPU/RSS/load have no helper-process confound. Original runs remain preserved. Custom build wrapper PID 484565 has queued about 20 minutes; four earlier tickets remain in front. It will build Cloudflare Caddy twice with Go 1.26.8 (matching stock), compare hashes, then build the separate Pebble adapter binary and run the required repository fast gate in the same held slot. No queue bypass or product code change. Target: `~/worktrees/r3/target`. Caddy source confirms its reload stop path invokes Go HTTP Server.Shutdown; shutdown_delay does not preserve reused listeners’ old idle HTTP connections. Final report will distinguish this graceful behavior from an actual listener outage, and will not hide common-checker failures with retries.

kayg commented 2026-11-01 08:41:40 +00:00

Author

Owner

Copy link

R3 build finding, caught before custom perf measurement: xcaddy 0.4.7 treats a nonempty XCADDY_GO_BUILD_FLAGS as a replacement for its normal build flags. My initial -trimpath -buildvcs=false override therefore omitted -ldflags '-w -s' and nobadger,nomysql,nopgx, producing an 82,007,670-byte unstripped/extra-driver binary. That artifact has not been measured and is not evidence of Cloudflare-module overhead. The corrected build explicitly sets the full stock-style flag set, retains Go 1.26.8, and will compare two independent compilation-cache builds. Initial metadata/logs remain in ~/worktrees/r3/unstripped-metadata/; corrected log is ~/worktrees/r3/corrected-build.log. The corrected build and product gate remain within the slot acquired at 14:01:56 IST (VM logs use CET).

Source: https://github.com/caddyserver/xcaddy/blob/v0.4.7/builder.go (the default flag block runs only when buildFlags is empty). This is a reproducibility/acquisition detail for the owner, not a product or engine defect. The final paired harness now starts its reload adapter only during reload and records workload exit separately from global cleanup-diff status. An earlier clean Traefik workload passed all checks but its global state diff recorded another queued job disappearing; raw diffs are retained.

R3 build finding, caught before custom perf measurement: xcaddy 0.4.7 treats a nonempty XCADDY_GO_BUILD_FLAGS as a replacement for its normal build flags. My initial `-trimpath -buildvcs=false` override therefore omitted `-ldflags '-w -s'` and `nobadger,nomysql,nopgx`, producing an 82,007,670-byte unstripped/extra-driver binary. That artifact has not been measured and is not evidence of Cloudflare-module overhead. The corrected build explicitly sets the full stock-style flag set, retains Go 1.26.8, and will compare two independent compilation-cache builds. Initial metadata/logs remain in `~/worktrees/r3/unstripped-metadata/`; corrected log is `~/worktrees/r3/corrected-build.log`. The corrected build and product gate remain within the slot acquired at 14:01:56 IST (VM logs use CET). Source: https://github.com/caddyserver/xcaddy/blob/v0.4.7/builder.go (the default flag block runs only when buildFlags is empty). This is a reproducibility/acquisition detail for the owner, not a product or engine defect. The final paired harness now starts its reload adapter only during reload and records workload exit separately from global cleanup-diff status. An earlier clean Traefik workload passed all checks but its global state diff recorded another queued job disappearing; raw diffs are retained.

kayg commented 2026-11-01 08:46:03 +00:00

Author

Owner

Copy link

R3 corrected build complete. Two compilation-cache-isolated Cloudflare builds are byte-identical: SHA-256 d46beb55fce5a7f7790674c6821ac96f0742e1f60990518483ed5713b9db8f2c. Stock executable 51,953,826 B; Cloudflare executable 52,019,362 B, exactly +65,536 B (64 KiB). Module-list diff is exactly one added dns.providers.cloudflare v0.2.4; both use Caddy 2.11.7 and Go 1.26.8. The initial 82 MB result came from my omitted strip/exclusion flags and is excluded from measurements, with metadata retained. Corrected source: WIP branch head d7274028c, pending final evidence/report commit.

The separate Pebble-adapter Caddy binary also built. The required full scripts/verify fast is now compiling on compute01 in the same held build-slot; ~/worktrees/r3/verify.log, sole Rust target ~/worktrees/r3/target. Final stock/Traefik pair with no reload helper in steady phases is running on backup01; Cloudflare/Traefik second pair and native DNS-01 probes follow. No foundation choice, merge or deployment.

R3 corrected build complete. Two compilation-cache-isolated Cloudflare builds are byte-identical: SHA-256 `d46beb55fce5a7f7790674c6821ac96f0742e1f60990518483ed5713b9db8f2c`. Stock executable **51,953,826 B**; Cloudflare executable **52,019,362 B**, exactly **+65,536 B (64 KiB)**. Module-list diff is exactly one added `dns.providers.cloudflare v0.2.4`; both use Caddy 2.11.7 and Go 1.26.8. The initial 82 MB result came from my omitted strip/exclusion flags and is excluded from measurements, with metadata retained. Corrected source: WIP branch head `d7274028c`, pending final evidence/report commit. The separate Pebble-adapter Caddy binary also built. The required full `scripts/verify fast` is now compiling on compute01 in the same held build-slot; `~/worktrees/r3/verify.log`, sole Rust target `~/worktrees/r3/target`. Final stock/Traefik pair with no reload helper in steady phases is running on backup01; Cloudflare/Traefik second pair and native DNS-01 probes follow. No foundation choice, merge or deployment.

kayg commented 2026-11-01 09:08:23 +00:00

Author

Owner

Copy link

R3 progress, 14:39 IST. Branch research-reverse-proxy, WIP f86ac71b4 pushed for crash safety.

The corrected Cloudflare build is 52,019,362 B, only 65,536 B larger than stock, and two builds with separate compilation caches are byte-identical. Its clean frozen-kit run measured idle/warm/1,000-TLS-connection RSS of 47.93 / 48.78 / 94.25 MiB. The 17 MiB idle figure is not reproduced in this scenario. Stock's final fresh pair remains pending; earlier stock starts were around 47–49 MiB. Module presence is measured without a live credential, under the same static-certificate setup as Traefik; small RSS differences must not be attributed causally to the module.

Fresh Cloudflare Caddy → Traefik B: h1 c64 22,009 / 26,888 req/s, p99 8.776 / 8.895 ms; h1 c512 13,198 / 20,643 req/s. Caddy reload failed-request counts 3 / 2 / 2, old h1/h2 closed, WebSocket retained with explicit 5-minute stream-close delay, slow in-flight requests completed. Traefik 0 / 0 / 0 failures and all checked connections preserved. Caddy switches 13–16 ms, Traefik 69 / 528 / 1,083 ms. Listener availability and preservation of every existing connection are distinct findings.

The final DNS-01 probes are waiting for another job's perf lock; stock Caddy → Traefik A follows. No source/binary transfer or bulk hashing overlaps the four primary runs. The global state snapshots include other jobs entering the lock queue; those diffs are retained separately from workload exits and owned socket/kit/tmp cleanup. Traefik B's workload passed despite that unrelated global process diff.

scripts/verify fast continues in the held compute01 build slot; target is solely ~/worktrees/r3/target. Logs: compute01 ~/worktrees/r3/verify.log; backup01 ~/r3/measured, ~/r3/measured-custom.log. Build metadata is copied to the control-host R3 evidence folder. Remaining: DNS probes, stock pair, final tables/grill and evidence attachment, atomic final commit, gate output and owned-copy/target cleanup. No engine decision, product integration or deployment.

R3 progress, 14:39 IST. Branch `research-reverse-proxy`, WIP `f86ac71b4` pushed for crash safety. The corrected Cloudflare build is 52,019,362 B, only 65,536 B larger than stock, and two builds with separate compilation caches are byte-identical. Its clean frozen-kit run measured idle/warm/1,000-TLS-connection RSS of **47.93 / 48.78 / 94.25 MiB**. The 17 MiB idle figure is not reproduced in this scenario. Stock's final fresh pair remains pending; earlier stock starts were around 47–49 MiB. Module presence is measured without a live credential, under the same static-certificate setup as Traefik; small RSS differences must not be attributed causally to the module. Fresh Cloudflare Caddy → Traefik B: h1 c64 **22,009 / 26,888 req/s**, p99 **8.776 / 8.895 ms**; h1 c512 **13,198 / 20,643 req/s**. Caddy reload failed-request counts **3 / 2 / 2**, old h1/h2 closed, WebSocket retained with explicit 5-minute stream-close delay, slow in-flight requests completed. Traefik **0 / 0 / 0** failures and all checked connections preserved. Caddy switches **13–16 ms**, Traefik **69 / 528 / 1,083 ms**. Listener availability and preservation of every existing connection are distinct findings. The final DNS-01 probes are waiting for another job's perf lock; stock Caddy → Traefik A follows. No source/binary transfer or bulk hashing overlaps the four primary runs. The global state snapshots include other jobs entering the lock queue; those diffs are retained separately from workload exits and owned socket/kit/tmp cleanup. Traefik B's workload passed despite that unrelated global process diff. `scripts/verify fast` continues in the held compute01 build slot; target is solely `~/worktrees/r3/target`. Logs: compute01 `~/worktrees/r3/verify.log`; backup01 `~/r3/measured`, `~/r3/measured-custom.log`. Build metadata is copied to the control-host R3 evidence folder. Remaining: DNS probes, stock pair, final tables/grill and evidence attachment, atomic final commit, gate output and owned-copy/target cleanup. No engine decision, product integration or deployment.

kayg commented 2026-11-01 09:24:39 +00:00

Author

Owner

Copy link

R3 measurements complete, 14:55 IST. Full operational/source report and atomic branch head will follow after the still-running repository gate and compute-target cleanup. The perf VM's R3 kit/results/transfer archive are removed; original J20 kit preserved. Raw evidence was copied and SHA-256 matched before cleanup.

17 MiB idle is not reproduced: stock Caddy 47.54 MiB, Cloudflare build 47.93 MiB, versus fresh Traefik 93.41 / 93.97 MiB. Cloudflare adds 64 KiB executable size, with no established causal RSS increase in these variable samples. All runs use pinned Caddy 2.11.7, Cloudflare 0.2.4, Traefik 3.7.17, the unchanged J20 clients/scenarios, via perf-run. No binary/source transfer or bulk checksums overlap the four primary runs.

R3 same-session order: Cloudflare Caddy → Traefik B → stock Caddy → Traefik A. DNS-01 probes ran between the pairs.

Engine	Idle MiB	Warm MiB	1,000 TLS connections MiB	Added KiB/connection	HTTP VmHWM MiB	Executable B
Caddy stock	47.54	50.20	96.45	47.37	147.12	51,953,826
Traefik A	93.41	95.50	150.01	55.82	275.19	182,120,608
Caddy + Cloudflare	47.93	48.78	94.25	46.56	139.78	52,019,362
Traefik B	93.97	95.86	150.21	55.65	275.09	182,120,608

J20 historical reference	Idle MiB	Warm MiB	1,000 TLS connections MiB	Added KiB/connection	HTTP VmHWM MiB
Pingora	6.66	8.09	64.01	57.26	63.88
Sozu	8.19	10.04	52.36	43.34	52.10
rpxy	6.14	6.68	44.25	38.47	59.39
Traefik	93.30	95.77	150.27	55.81	265.84

Engine	h1 c1 req/s / p99 ms	h1 c64	h1 c512	h2 c64
Caddy stock	5,027 / 0.568	20,776 / 9.480	12,577 / 73.882	15,738 / 10.863
Traefik A	5,609 / 0.508	30,340 / 8.024	23,241 / 42.226	23,604 / 7.917
Caddy + Cloudflare	4,950 / 0.587	22,009 / 8.776	13,198 / 69.576	17,417 / 9.335
Traefik B	5,104 / 0.617	26,888 / 8.895	20,643 / 53.022	23,036 / 8.030

J20 historical reference	h1 c1 req/s / p99 ms	h1 c64	h1 c512	h2 c64
Pingora	7,140 / 0.328	45,372 / 2.981	34,965 / 38.203	35,246 / 3.673
Sozu	11,489 / 0.140	50,052 / 3.161	38,217 / 45.940	29,444 / 4.482
rpxy	10,394 / 0.151	52,315 / 2.120	38,928 / 45.097	35,733 / 3.119
Traefik	6,418 / 0.224	32,956 / 7.296	25,711 / 39.075	25,289 / 7.393

Engine	Idle CPU %	h1 c1 CPU %	h1 c64 CPU %	h1 c512 CPU %	h2 c64 CPU %	TLS c64 CPU %
Caddy stock	0.00	89.68	192.46	191.06	190.59	158.80
Traefik A	0.00	83.31	191.84	179.72	194.07	158.68
Caddy + Cloudflare	0.00	89.54	193.23	189.54	192.01	159.01
Traefik B	0.00	83.61	191.33	182.12	194.43	158.21

Engine	Full TLS/s c1	p50 / p99 ms	Full TLS/s c64	p50 / p99 ms	TLS failures
Caddy stock	1,830	0.461 / 1.698	5,049	12.570 / 19.026	0
Traefik A	2,058	0.446 / 0.856	5,251	12.201 / 17.476	0
Caddy + Cloudflare	2,127	0.426 / 0.872	5,243	12.253 / 16.466	0
Traefik B	2,067	0.444 / 0.845	5,274	12.154 / 17.402	0

Engine	Failed requests 1 / 2 / 3	Route switch ms	Certificate switch ms	Old h1	WebSocket	Old h2
Caddy stock	3 / 3 / 2	17 / 13 / 13	17 / 12 / 12	closed / closed / closed	alive / alive / alive	closed / closed / closed
Traefik A	0 / 0 / 0	13 / 579 / 1035	12 / 579 / 1035	alive / alive / alive	alive / alive / alive	alive / alive / alive
Caddy + Cloudflare	3 / 2 / 2	16 / 14 / 16	16 / 13 / 15	closed / closed / closed	alive / alive / alive	closed / closed / closed
Traefik B	0 / 0 / 0	69 / 528 / 1083	69 / 527 / 1083	alive / alive / alive	alive / alive / alive	alive / alive / alive

All steady HTTP/full-TLS phases had zero errors; all engines held 1,000 connections and passed 9/9 functional checks before and after reload. Caddy's strict zero-drop workload failed: existing h1/h2 connections closed, although slow in-flight requests completed and WebSockets survived with explicit 5-minute stream-close delay. Default Caddy WebSockets closed in all three separate diagnostic trials. Traefik preserved all checked connection types through all six reload trials.

Native local DNS-01: Caddy issued two Pebble certificates in 1,539 ms, Traefik in 3,258 ms; both served both names. No live Cloudflare credential/public propagation was tested. API runtime probes confirmed rollback, successful JSON load, private Unix socket load and admin disabled. The stock, Traefik A, API and Caddy DNS global diffs were empty. Other primary/global DNS diffs contained only other queued jobs or a new external SFTP process; all socket/kit/tmp diffs were empty. Workload exits and global wrapper exits remain separately preserved.

Recommendation/grill for kayg: retain the proposed managed Traefik if existing-connection preservation is a hard contract. Does saving about 46 MiB per active idle proxy justify Caddy's observed h1/h2 reconnect semantics, bounded WebSocket retention and a custom Cloudflare-module artifact pipeline? Caddy is a mature, credible lower-footprint alternative if those trade-offs are acceptable. This job does not select an engine or change D316.

Limits: one full run per Caddy artifact, two Traefik repetitions; hypervisor steal ~1–2%; historical engines are reference rows, not same-session comparisons. The direct h1 c64 ceiling was 56,451 req/s versus historical 63,506. Native ACME timings include engine scheduling/polling and local DNS, not an isolated-library comparison. scripts/verify fast is still running under build-slot; no final green gate claim yet.

R3 measurements complete, 14:55 IST. Full operational/source report and atomic branch head will follow after the still-running repository gate and compute-target cleanup. The perf VM's R3 kit/results/transfer archive are removed; original J20 kit preserved. Raw evidence was copied and SHA-256 matched before cleanup. **17 MiB idle is not reproduced:** stock Caddy 47.54 MiB, Cloudflare build 47.93 MiB, versus fresh Traefik 93.41 / 93.97 MiB. Cloudflare adds 64 KiB executable size, with no established causal RSS increase in these variable samples. All runs use pinned Caddy 2.11.7, Cloudflare 0.2.4, Traefik 3.7.17, the unchanged J20 clients/scenarios, via perf-run. No binary/source transfer or bulk checksums overlap the four primary runs. R3 same-session order: Cloudflare Caddy → Traefik B → stock Caddy → Traefik A. DNS-01 probes ran between the pairs. | Engine | Idle MiB | Warm MiB | 1,000 TLS connections MiB | Added KiB/connection | HTTP VmHWM MiB | Executable B | |---|---|---|---|---|---|---| | Caddy stock | 47.54 | 50.20 | 96.45 | 47.37 | 147.12 | 51,953,826 | | Traefik A | 93.41 | 95.50 | 150.01 | 55.82 | 275.19 | 182,120,608 | | Caddy + Cloudflare | 47.93 | 48.78 | 94.25 | 46.56 | 139.78 | 52,019,362 | | Traefik B | 93.97 | 95.86 | 150.21 | 55.65 | 275.09 | 182,120,608 | | J20 historical reference | Idle MiB | Warm MiB | 1,000 TLS connections MiB | Added KiB/connection | HTTP VmHWM MiB | |---|---|---|---|---|---| | Pingora | 6.66 | 8.09 | 64.01 | 57.26 | 63.88 | | Sozu | 8.19 | 10.04 | 52.36 | 43.34 | 52.10 | | rpxy | 6.14 | 6.68 | 44.25 | 38.47 | 59.39 | | Traefik | 93.30 | 95.77 | 150.27 | 55.81 | 265.84 | | Engine | h1 c1 req/s / p99 ms | h1 c64 | h1 c512 | h2 c64 | |---|---|---|---|---| | Caddy stock | 5,027 / 0.568 | 20,776 / 9.480 | 12,577 / 73.882 | 15,738 / 10.863 | | Traefik A | 5,609 / 0.508 | 30,340 / 8.024 | 23,241 / 42.226 | 23,604 / 7.917 | | Caddy + Cloudflare | 4,950 / 0.587 | 22,009 / 8.776 | 13,198 / 69.576 | 17,417 / 9.335 | | Traefik B | 5,104 / 0.617 | 26,888 / 8.895 | 20,643 / 53.022 | 23,036 / 8.030 | | J20 historical reference | h1 c1 req/s / p99 ms | h1 c64 | h1 c512 | h2 c64 | |---|---|---|---|---| | Pingora | 7,140 / 0.328 | 45,372 / 2.981 | 34,965 / 38.203 | 35,246 / 3.673 | | Sozu | 11,489 / 0.140 | 50,052 / 3.161 | 38,217 / 45.940 | 29,444 / 4.482 | | rpxy | 10,394 / 0.151 | 52,315 / 2.120 | 38,928 / 45.097 | 35,733 / 3.119 | | Traefik | 6,418 / 0.224 | 32,956 / 7.296 | 25,711 / 39.075 | 25,289 / 7.393 | | Engine | Idle CPU % | h1 c1 CPU % | h1 c64 CPU % | h1 c512 CPU % | h2 c64 CPU % | TLS c64 CPU % | |---|---|---|---|---|---|---| | Caddy stock | 0.00 | 89.68 | 192.46 | 191.06 | 190.59 | 158.80 | | Traefik A | 0.00 | 83.31 | 191.84 | 179.72 | 194.07 | 158.68 | | Caddy + Cloudflare | 0.00 | 89.54 | 193.23 | 189.54 | 192.01 | 159.01 | | Traefik B | 0.00 | 83.61 | 191.33 | 182.12 | 194.43 | 158.21 | | Engine | Full TLS/s c1 | p50 / p99 ms | Full TLS/s c64 | p50 / p99 ms | TLS failures | |---|---|---|---|---|---| | Caddy stock | 1,830 | 0.461 / 1.698 | 5,049 | 12.570 / 19.026 | 0 | | Traefik A | 2,058 | 0.446 / 0.856 | 5,251 | 12.201 / 17.476 | 0 | | Caddy + Cloudflare | 2,127 | 0.426 / 0.872 | 5,243 | 12.253 / 16.466 | 0 | | Traefik B | 2,067 | 0.444 / 0.845 | 5,274 | 12.154 / 17.402 | 0 | | Engine | Failed requests 1 / 2 / 3 | Route switch ms | Certificate switch ms | Old h1 | WebSocket | Old h2 | |---|---|---|---|---|---|---| | Caddy stock | 3 / 3 / 2 | 17 / 13 / 13 | 17 / 12 / 12 | closed / closed / closed | alive / alive / alive | closed / closed / closed | | Traefik A | 0 / 0 / 0 | 13 / 579 / 1035 | 12 / 579 / 1035 | alive / alive / alive | alive / alive / alive | alive / alive / alive | | Caddy + Cloudflare | 3 / 2 / 2 | 16 / 14 / 16 | 16 / 13 / 15 | closed / closed / closed | alive / alive / alive | closed / closed / closed | | Traefik B | 0 / 0 / 0 | 69 / 528 / 1083 | 69 / 527 / 1083 | alive / alive / alive | alive / alive / alive | alive / alive / alive | All steady HTTP/full-TLS phases had zero errors; all engines held 1,000 connections and passed 9/9 functional checks before and after reload. Caddy's strict zero-drop workload failed: existing h1/h2 connections closed, although slow in-flight requests completed and WebSockets survived with explicit 5-minute stream-close delay. Default Caddy WebSockets closed in all three separate diagnostic trials. Traefik preserved all checked connection types through all six reload trials. Native local DNS-01: Caddy issued two Pebble certificates in 1,539 ms, Traefik in 3,258 ms; both served both names. No live Cloudflare credential/public propagation was tested. API runtime probes confirmed rollback, successful JSON load, private Unix socket load and admin disabled. The stock, Traefik A, API and Caddy DNS global diffs were empty. Other primary/global DNS diffs contained only other queued jobs or a new external SFTP process; all socket/kit/tmp diffs were empty. Workload exits and global wrapper exits remain separately preserved. Recommendation/grill for kayg: **retain the proposed managed Traefik if existing-connection preservation is a hard contract. Does saving about 46 MiB per active idle proxy justify Caddy's observed h1/h2 reconnect semantics, bounded WebSocket retention and a custom Cloudflare-module artifact pipeline?** Caddy is a mature, credible lower-footprint alternative if those trade-offs are acceptable. This job does not select an engine or change D316. Limits: one full run per Caddy artifact, two Traefik repetitions; hypervisor steal ~1–2%; historical engines are reference rows, not same-session comparisons. The direct h1 c64 ceiling was 56,451 req/s versus historical 63,506. Native ACME timings include engine scheduling/polling and local DNS, not an isolated-library comparison. `scripts/verify fast` is still running under build-slot; no final green gate claim yet.

r3-perf-evidence.tar.gz

104 KiB

r3-final-evidence.tar.gz

720 KiB

kayg commented 2026-11-01 09:54:36 +00:00

Author

Owner

Copy link

R3 complete, 15:24 IST. Measure only; no product integration, engine decision, merge or deployment.

Branch research-reverse-proxy, final head 24a3cf8ce98aff5fc28e56ef1b8140aac43bb4f3, one atomic R3 commit atop J20 7aa0c42036b312e88fea8768ae8540b38d740e02. Harness and report pushed. Immutable report. The VM worktree is on that head and clean. A fresh final-head check confirmed all 18 archived harness files match and the owned target/build directories are absent. No R3 background work remains. Issue remains open for kayg's decision.

Both evidence archives are attached to the full measurement tables; checksums, full gate output, cleanup and sources follow.

Caddy comparison, R3

Research only. D316's engine choice remains with kayg; no product dependency,
configuration, deployment or decision is changed here. Baseline: J20
7aa0c42036b312e88fea8768ae8540b38d740e02. Prior findings:
J20,
R2,
decision reopened for this comparison.

The owner's 17 MiB idle figure is not reproduced. Stock Caddy is
47.54 MiB idle and the Cloudflare build 47.93 MiB, versus fresh
Traefik 93.41 / 93.97 MiB. Caddy roughly halves idle memory in this
scenario, while Traefik preserves all checked connections through reload.
The owner still chooses that trade-off; the complete tables are below.

Pins and measurement contract

Latest stable Caddy at the 2026-11-01 research snapshot:
2.11.7, published 2026-10-03.
caddy/pins.env pins that release, xcaddy 0.4.7, Cloudflare module 0.2.4,
and Go 1.26.8, matching the official stock binary’s build info. Official archive checksums are SHA-512;
Go archive and measured executable identities use SHA-256. Stock means the
official GitHub Linux amd64 release, not the website's custom-download API.

Build on compute01 through build-slot bash caddy/build OUT. Two different
xcaddy working directories test byte identity with cmp. Saved generated
go.mod/go.sum, build info, module inventory, toolchain and executable
hashes identify the full closure. These are two builds on one machine with
separate compilation caches and shared downloaded modules, not independent-builder attestation. The corrected two Cloudflare builds were byte-identical (SHA-256
d46beb55fce5a7f7790674c6821ac96f0742e1f60990518483ed5713b9db8f2c).
Stock is 51,953,826 B and Cloudflare is 52,019,362 B: +65,536 B. The
module inventory adds exactly dns.providers.cloudflare v0.2.4 to 135 standard
modules. Both builds use stripping and nobadger,nomysql,nopgx tags. A nonempty
xcaddy build-flags override replaces its defaults; an initial unstripped
82,007,670 B artifact was caught before measurement and excluded. Its logs
and metadata remain with the evidence.

The module is installed but not provisioned with a live Cloudflare credential
in the common static-certificate workload. A separate
caddy-dnsstub executable adds the local Pebble adapter; it is excluded from
the stock/Cloudflare footprint rows.

Use J20's final kit on backup01 and verify its measured executable hashes.
The saved original-manifest check marks run-case.sh failed because it
was deliberately adapted for Caddy; every original executable matches.
The final nine changed/added kit source files were compared byte-for-byte
with the reviewable repository copies after measurement and matched.
Copy the new Caddy executables and kit scripts before starting any run;
freeze the kit throughout the measurements. Run all workloads via
bash run-all.sh CASE, which calls perf-run and explicitly passes overrides
inside its sanitized environment. Use a distinct OUT for each repetition.
Preserve nonzero exits and before/after state diffs, including failed staging
attempts. Never edit a shell script while a running shell is reading it.

The scenario and client code are J20's unchanged binaries: 5 s warmup +
30 s measurement, h1 concurrency 1/64/512, h2 eight connections with eight
parallel requests each, 15 s idle CPU, 1,000 TLS keep-alive connections, full
TLS handshakes c1/c64 for 30 s without resumption, and three reload trials.
Two proxy CPUs; backend and generator share the third. CPU 100% is one core.
A zero idle CPU sample means no observed CPU ticks in 15 seconds, not proof
of zero wakeups. RSS/HWM cover only the proxy process; no forced GC is used.
These are workload results with a generator/backend ceiling, not maximum
engine capacity. J20's other engines are historical reference rows, not
new same-session measurements.

The Caddy adapter generates native JSON, disables automatic HTTPS for the
common externally issued certificate workload, and binds admin to
127.0.0.1:12019 with persistence disabled. A 2 ms research watcher, started only for the reload phase and pinned to the
backend/generator CPU, translates
the common checker's atomic route-file replacement into POST /load.
Certificate PEMs are embedded in the submitted JSON so rotation changes its
content and does not depend on filename-cache behavior. Switch times include
certificate creation, watcher observation, API application and the common
client's polling interval. This adapter is not product implementation.
Traefik retains J20's file-provider defaults, including its update debounce.
These timings include the different update paths; they are not isolated
configuration-parser or API latency measurements.

stream_close_delay is explicitly 5 minutes for the configured Caddy runs.
That bounds retention of old streaming configuration; it is not a promise
that indefinitely open streams survive indefinitely. The default behavior
is checked separately. Both successful HTTP responses and surviving old
connections must be reported; a fast configuration switch is not proof of
zero drops.
The memory samples precede reload; they do not quantify retaining old
WebSocket configurations across frequent updates or long-lived streams.
The HWM column is J20's post-HTTP-load sample, not a whole-lifecycle peak.

Control surface and TLS

Caddy's native JSON admin API supports full POST /load, subtree changes
and inspection. /load blocks until application succeeds or fails; upstream
documents rollback to the previous working configuration on failure.
The default local endpoint is localhost:2019. For a supervised product,
prefer a permissioned Unix socket owned by the infraternal agent, serialize
desired revisions, and probe the intended route/certificate before marking
it applied. Loopback alone does not isolate untrusted local processes.
Set admin.disabled: true (Caddyfile admin off) to disable administration;
then ordinary API/CLI hot reload is unavailable and changes require another
lifecycle strategy. admin.config.persist: false prevents automatic config
snapshots, relevant when configuration contains DNS credentials.
API,
admin schema.

WebSocket upgrades are supported by the reverse proxy. Upstream explicitly
documents closing streams on config unload by default, and the
stream_close_delay mitigation. HTTP graceful shutdown and reconnect
behavior still need the common checker; the documentation's zero-downtime
wording is not evidence that every existing client connection remains open.
The versioned HTTP app stop path invokes Go Server.Shutdown during reload;
shutdown_delay is skipped for listeners reused by the new configuration.
The common checker deliberately does not hide failures with automatic request
retries. Closed idle connections/HTTP2 GOAWAY can be handled by ordinary client
reconnects; a checker failure is not proof that the listening port disappeared.
Versioned stop path.
Reverse proxy.

Automatic HTTPS includes issuance/renewal. On-demand TLS is different from
downloading a proxy on demand: it postpones certificate acquisition until
the first TLS handshake for a name, holding that handshake while issuance
completes. It requires an authorization/permission policy (for example an
internal ask endpoint) so arbitrary SNI cannot cause unrestricted issuance.
For known managed names, eager issuance avoids that first-user latency.
DNS-01 requires a provider module; the stock binary does not contain the
Cloudflare provider. Automatic HTTPS,
Cloudflare module.

The local DNS-01 check uses J20's Pebble 2.10.1/challtestsrv and disposable
CA, validation sleep disabled, two names, and no real credentials. A tiny
separate provider module calls the stub's TXT management API; a loopback
resolver adds SOA answers for Caddy's zone discovery and forwards other
queries to challtestsrv. Only DNS-01 is enabled; propagation polling is
disabled because this is local test DNS. This proves Caddy's ACME/DNS-01
path, not live Cloudflare authentication, public propagation, rate limits
or long-running renewal reliability. The measured Cloudflare module is
installed but unused in the ordinary static-certificate workloads.
A companion native Traefik ACME run uses its bundled DNS exec provider and
a local TXT adapter against the same CA/stub, with p-256 leaf keys for both
engines. Neither probe uses a real provider credential; concurrent issuance
scheduling and polling are engine-specific, so total elapsed time is not a
standalone ACME-library benchmark.

Distribution, verification and maintenance

xcaddy creates a small Go main importing Caddy and requested modules, resolves
their Go dependencies, and compiles one executable. Modules are compiled in,
not shared objects loaded at runtime. Pin Caddy, each module, Go, build flags
and the resolved dependency closure. The website custom-download service
builds combinations selected by OS/architecture/version/module list; Caddy's
upgrade/add-package commands fetch builds through that service. It removes
local build tooling from the installation path but adds service availability
and artifact-trust dependencies. For a managed fleet, build once in controlled
CI and distribute a reviewed, signed component manifest and exact hashes.
Build documentation,
xcaddy 0.4.7,
CLI.

Official GitHub release artifacts have Sigstore signature/certificate assets
and a documented transparency-log verification process. A valid signature
must be checked against the expected workflow identity and tag, not merely
against whichever certificate accompanies a download. This does not
establish equivalent verification for custom API builds. Upstream's open
website issue reports different build tags between the API and GitHub
releases and no published checksum/signature reference for API builds.
Treat that as an upstream report, not a reproduced measurement of 2.11.7.
The inspected 2.11.7 CLI download path reads the HTTPS response into the new
binary without an artifact-signature verification step.
Versioned download code.
Our own xcaddy outputs have hashes and reproducibility evidence, but no
upstream release signature over the custom executable.
This run checked the official archive against its published checksum; it
did not execute the Sigstore verification procedure. Checksum equality alone
is not an independent publisher-identity verification.
Signature verification,
upstream website issue 549.

Caddy and the pinned Cloudflare module use Apache-2.0, compatible with the
AGPL core; additional modules need their own dependency/license review.
Caddy has public history from 2015, Caddy 2 since 2020, and is maintained by
Matt Holt and community maintainers, with ZeroSSL/HID Global stewardship
and sponsorship. It is a mature product, not a pre-1.0 framework. Recent
releases in the snapshot include 2.11.2 (March 6), 2.11.3 (May 12), 2.11.4
(June 3), 2.11.6 (October 1) and 2.11.7 (October 3).
Project,
ten-year history,
releases,
license,
module license.

The snapshot contains 18 published Caddy GitHub advisories. Counts cannot
rank safety against Traefik's 89 in R2: features, usage and disclosure differ.
Relevant examples include local admin cross-origin configuration application,
remote admin authorization bypasses, mTLS fail-open on malformed/missing CA
files, and forward-auth connection reuse/identity risks. The forward-auth
upstream-confusion and hop-by-hop-header findings list fixes in 2.11.5;
our 2.11.7 pin is newer. Track Go and plugin advisories as well as Caddy's.
Do not assume that a minimal enabled feature set makes every core HTTP/auth
finding irrelevant. The Cloudflare module's pinned source includes the
supplied token in its invalid-token error; a future custody implementation
must prevent that error text reaching ordinary logs.
Advisories,
forward-auth finding,
pinned module source.

The separate runtime API probe passed: invalid JSON module configuration returned 400 while retaining the exact prior exported config and route; a valid load returned 200 and changed the response; a Unix socket beneath a 0700 directory accepted a load; disabling admin removed both administration endpoints while the HTTP route remained available. Its perf-run state diff was empty.

The final kit records the workload exit separately from the global cleanup
comparison. Other jobs joining/leaving the shared perf-lock queue can change
the global process snapshot while doing no measured work; those raw diffs
are retained and must be attributed explicitly. Empty socket, kit and
temporary-file diffs plus the owned-process sweep are separate cleanup
evidence, not a reason to erase the global diff.

Both engines support a product-owned UI/model and supervised sidecar shape.
Caddy offers a native writable configuration API and lower measured process
footprint; Traefik includes DNS providers and request-rate middleware in its
standard distribution and has a dynamic provider model. Caddy has built-in
auth/header/rewrite/reverse-proxy facilities, but stock request-rate limiting
and general TCP/UDP routing should not be assumed; those need additional
modules and measurements. Neither engine's schema should become the public
infraternal API.

Measurement results

The following tables come from the four final frozen-kit runs.
The final order is Cloudflare Caddy, Traefik B, the two native DNS-01 probes,
stock Caddy, Traefik A. No binary transfer, source edit or bulk checksum runs
overlap these four primary measurements. Earlier stock runs overlapped binary
staging and are retained as calibration only. Early attempts also changed
an active parent shell script and produced post-workload parse errors; they
are explicitly excluded from the final comparison.

R3 same-session order: Cloudflare Caddy → Traefik B → stock Caddy → Traefik A. DNS-01 probes ran between the pairs.

Engine	Idle MiB	Warm MiB	1,000 TLS connections MiB	Added KiB/connection	HTTP VmHWM MiB	Executable B
Caddy stock	47.54	50.20	96.45	47.37	147.12	51,953,826
Traefik A	93.41	95.50	150.01	55.82	275.19	182,120,608
Caddy + Cloudflare	47.93	48.78	94.25	46.56	139.78	52,019,362
Traefik B	93.97	95.86	150.21	55.65	275.09	182,120,608

J20 historical reference	Idle MiB	Warm MiB	1,000 TLS connections MiB	Added KiB/connection	HTTP VmHWM MiB
Pingora	6.66	8.09	64.01	57.26	63.88
Sozu	8.19	10.04	52.36	43.34	52.10
rpxy	6.14	6.68	44.25	38.47	59.39
Traefik	93.30	95.77	150.27	55.81	265.84

Engine	h1 c1 req/s / p99 ms	h1 c64	h1 c512	h2 c64
Caddy stock	5,027 / 0.568	20,776 / 9.480	12,577 / 73.882	15,738 / 10.863
Traefik A	5,609 / 0.508	30,340 / 8.024	23,241 / 42.226	23,604 / 7.917
Caddy + Cloudflare	4,950 / 0.587	22,009 / 8.776	13,198 / 69.576	17,417 / 9.335
Traefik B	5,104 / 0.617	26,888 / 8.895	20,643 / 53.022	23,036 / 8.030

J20 historical reference	h1 c1 req/s / p99 ms	h1 c64	h1 c512	h2 c64
Pingora	7,140 / 0.328	45,372 / 2.981	34,965 / 38.203	35,246 / 3.673
Sozu	11,489 / 0.140	50,052 / 3.161	38,217 / 45.940	29,444 / 4.482
rpxy	10,394 / 0.151	52,315 / 2.120	38,928 / 45.097	35,733 / 3.119
Traefik	6,418 / 0.224	32,956 / 7.296	25,711 / 39.075	25,289 / 7.393

Engine	Idle CPU %	h1 c1 CPU %	h1 c64 CPU %	h1 c512 CPU %	h2 c64 CPU %	TLS c64 CPU %
Caddy stock	0.00	89.68	192.46	191.06	190.59	158.80
Traefik A	0.00	83.31	191.84	179.72	194.07	158.68
Caddy + Cloudflare	0.00	89.54	193.23	189.54	192.01	159.01
Traefik B	0.00	83.61	191.33	182.12	194.43	158.21

Engine	Full TLS/s c1	p50 / p99 ms	Full TLS/s c64	p50 / p99 ms	TLS failures
Caddy stock	1,830	0.461 / 1.698	5,049	12.570 / 19.026	0
Traefik A	2,058	0.446 / 0.856	5,251	12.201 / 17.476	0
Caddy + Cloudflare	2,127	0.426 / 0.872	5,243	12.253 / 16.466	0
Traefik B	2,067	0.444 / 0.845	5,274	12.154 / 17.402	0

Engine	Failed requests 1 / 2 / 3	Route switch ms	Certificate switch ms	Old h1	WebSocket	Old h2
Caddy stock	3 / 3 / 2	17 / 13 / 13	17 / 12 / 12	closed / closed / closed	alive / alive / alive	closed / closed / closed
Traefik A	0 / 0 / 0	13 / 579 / 1035	12 / 579 / 1035	alive / alive / alive	alive / alive / alive	alive / alive / alive
Caddy + Cloudflare	3 / 2 / 2	16 / 14 / 16	16 / 13 / 15	closed / closed / closed	alive / alive / alive	closed / closed / closed
Traefik B	0 / 0 / 0	69 / 528 / 1083	69 / 527 / 1083	alive / alive / alive	alive / alive / alive	alive / alive / alive

The added-connection column is (RSS at 1,000 - warm RSS) / 1,000, in KiB.
Cloudflare adds 64 KiB of executable bytes. Its sampled RSS is not
consistently higher than stock; do not turn heap/run variability into a
causal module-cost claim. All four runs held 1,000/1,000 connections, passed
9/9 basic checks before and after reload (including WebSocket proxying),
and had zero steady HTTP and full-handshake errors. Slow in-flight requests
completed in every reload trial. The default-stream diagnostic, without
stream_close_delay, closed WebSockets in all three trials.

The proposed idle saving is about 46 MiB per active proxy process, not
76 MiB implied by 17 versus 93. For illustration, 10/100 active proxy
processes save about 460 MiB/4.5 GiB idle. These are arithmetic projections,
not measured scaling or a prediction that every node runs a proxy. Each
process's 1,000-connection footprint remains in the table; no linear
extrapolation to arbitrary connection counts is established.

Traefik delivered 1.22–1.46 times Caddy's h1 c64 rate and 1.56–1.85 times
its h1 c512 rate in these pairs. H2 also favored Traefik. TLS c64 results
were close (roughly 5.0–5.3k full handshakes/s), while stock Caddy's c1
sample was slower. There is one full run per Caddy artifact and two
Traefik repetitions, not a confidence interval. Traefik's own c64 range
(26,888–30,340 req/s) demonstrates variability. Hypervisor steal was
0.98% / 1.63% / 1.98% / 1.05% in chronological order; all runs passed the
wrapper's admission check. Direct h1 c64 was 56,451 req/s, versus J20's
historical 63,506. Do not compare the historical engines as if the host
conditions were unchanged or interpret this constrained generator as peak
server capacity.

Probe	Result	Process RSS / VmHWM	Workload / wrapper exit
Caddy native JSON API	invalid load rollback, valid load, private Unix socket, admin off passed	separate functional probe	0 / 0
Caddy native DNS-01	2 local Pebble certificates in 1,539 ms, both names served	52.77 / 52.77 MiB	0 / 0
Traefik native DNS-01	2 local Pebble certificates in 3,258 ms, both names served	101.18 / 101.18 MiB	0 / 1
Caddy stock primary	strict reload preservation failed; other checks passed	above	1 / 1
Caddy + Cloudflare primary	strict reload preservation failed; other checks passed	above	1 / 1
Traefik A primary	all workload and cleanup checks passed	above	0 / 0
Traefik B primary	all workload checks passed	above	0 / 1

The stock run and API/Caddy DNS probes had empty global state diffs.
Cloudflare and Traefik B contain only other jobs joining the perf-lock
queue in the process diff. Traefik DNS contains a new external SFTP process.
All final socket, kit and temporary-file diffs were empty. The final owned
process/socket sweep and removal of the R3 kit are recorded separately.
An unrelated global diff never turns a failed Caddy workload into a pass,
and a wrapper exit of 1 never gets silently discarded.

The DNS totals are native end-to-end issuance scenarios with local DNS and
validation delay disabled, not a public-provider comparison. Stock Caddy
cannot run the Cloudflare DNS provider; the measured Cloudflare artifact
has that module installed, while the separate stub artifact exercises local
DNS-01 without any live provider credential. J20's standalone ACME example
issued two certificates in 1,789 ms with 4,988 KiB peak RSS and a
2,990,800 B executable; that is a different program, not a sidecar RSS row.

Evidence and verification

Complete evidence bundle,
SHA-256 0d43623c1e396f09831e16a844129093bba0d10954aaf24911e5bb2f4c303f43.
Raw perf archive,
SHA-256 ed0cd3a4a46386a5e59a3f26243921504c7bc46d8a37f38af9cfe7bb4665700c.
Both are attached to the measurement comment.

The bundle includes all raw runs/receipts, failed attempts, generated build
closure and hashes, source snapshots, final source/syntax checks, the complete
unedited verify.log, and cleanup records. No executables, generated private
keys or compilation caches are attached. All 18 archived harness files match
the VM verification source; all nine measured kit files match that source.

scripts/verify fast completed through compute01's held build slot with exit
0. Product code/dependencies are unchanged. Both static artifacts match their
D243 caps exactly. Verbatim selected gate output (the complete output is in
the bundle):

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+

PT_INTERP: absent; size: 21431472 bytes; cap: 21431472 bytes; delta: +0 bytes
PT_INTERP: absent; size: 19605896 bytes; cap: 19605896 bytes; delta: +0 bytes


 39 pass
 0 fail
 50325 expect() calls
Ran 39 tests across 10 files. [4.31s]
R3 scripts/verify fast exit=0

The R3 perf kit, result copies and transfer archive were removed after the
copied archive's SHA-256 matched; the original J20 final kit is preserved.
The owned Rust target, Go compilation caches/toolchain, Caddy build artifacts
and six exact recorded xcaddy temporary directories were removed. The final
process sweep found no surviving owned target executable or private-fixture
process. Small source worktrees and evidence remain on compute01; no R3
background work remains.

Owner grill

Recommendation for the owner to consider: retain the proposed managed
Traefik component if preserving existing connections across route and
certificate changes is a hard requirement. Caddy is a credible lower-memory
alternative if graceful client reconnects and a controlled custom-module
build/update pipeline are acceptable. It is mature and has a useful native
JSON control surface; a smaller process alone does not resolve the lifecycle
contract or replace middleware/provider coverage measurements.

Does the saved per-node memory justify accepting Caddy's observed h1/h2
connection closure on reload, bounded WebSocket retention, and our own
Cloudflare-module artifact pipeline, or should D316 retain Traefik for its
observed connection preservation and bundled providers/middleware?

This is a measured trade-off for kayg, not an engine selection by R3. The
answer may be to relax the connection contract, but that is an explicit
owner decision rather than relabeling a failed zero-drop trial as a pass.

R3 complete, 15:24 IST. Measure only; no product integration, engine decision, merge or deployment. Branch **`research-reverse-proxy`**, final head **`24a3cf8ce98aff5fc28e56ef1b8140aac43bb4f3`**, one atomic R3 commit atop J20 **`7aa0c42036b312e88fea8768ae8540b38d740e02`**. Harness and report pushed. [Immutable report](https://git.kayg.org/kayg/infraternal/src/commit/24a3cf8ce98aff5fc28e56ef1b8140aac43bb4f3/research/reverse-proxy/CADDY.md). The VM worktree is on that head and clean. A fresh final-head check confirmed all 18 archived harness files match and the owned target/build directories are absent. No R3 background work remains. Issue remains open for kayg's decision. Both evidence archives are attached to [the full measurement tables](https://git.kayg.org/kayg/infraternal/issues/261#issuecomment-72620); checksums, full gate output, cleanup and sources follow. <!-- Role: R3 Caddy versus Traefik measurements and operational evidence for #261. --> <!-- SPDX-License-Identifier: AGPL-3.0-only --> # Caddy comparison, R3 Research only. D316's engine choice remains with kayg; no product dependency, configuration, deployment or decision is changed here. Baseline: J20 `7aa0c42036b312e88fea8768ae8540b38d740e02`. Prior findings: [J20](https://git.kayg.org/kayg/infraternal/issues/261#issuecomment-71101), [R2](https://git.kayg.org/kayg/infraternal/issues/261#issuecomment-72433), [decision reopened for this comparison](https://git.kayg.org/kayg/infraternal/issues/261#issuecomment-72438). **The owner's 17 MiB idle figure is not reproduced.** Stock Caddy is **47.54 MiB** idle and the Cloudflare build **47.93 MiB**, versus fresh Traefik **93.41 / 93.97 MiB**. Caddy roughly halves idle memory in this scenario, while Traefik preserves all checked connections through reload. The owner still chooses that trade-off; the complete tables are below. ## Pins and measurement contract Latest stable Caddy at the 2026-11-01 research snapshot: [2.11.7, published 2026-10-03](https://github.com/caddyserver/caddy/releases/tag/v2.11.7). `caddy/pins.env` pins that release, xcaddy 0.4.7, Cloudflare module 0.2.4, and Go 1.26.8, matching the official stock binary’s build info. Official archive checksums are **SHA-512**; Go archive and measured executable identities use SHA-256. Stock means the official GitHub Linux amd64 release, not the website's custom-download API. Build on compute01 through `build-slot bash caddy/build OUT`. Two different xcaddy working directories test byte identity with `cmp`. Saved generated `go.mod`/`go.sum`, build info, module inventory, toolchain and executable hashes identify the full closure. These are two builds on one machine with separate compilation caches and shared downloaded modules, not independent-builder attestation. The corrected two Cloudflare builds were byte-identical (SHA-256 `d46beb55fce5a7f7790674c6821ac96f0742e1f60990518483ed5713b9db8f2c`). Stock is 51,953,826 B and Cloudflare is 52,019,362 B: +65,536 B. The module inventory adds exactly `dns.providers.cloudflare v0.2.4` to 135 standard modules. Both builds use stripping and `nobadger,nomysql,nopgx` tags. A nonempty xcaddy build-flags override replaces its defaults; an initial unstripped 82,007,670 B artifact was caught before measurement and excluded. Its logs and metadata remain with the evidence. The module is installed but not provisioned with a live Cloudflare credential in the common static-certificate workload. A separate `caddy-dnsstub` executable adds the local Pebble adapter; it is excluded from the stock/Cloudflare footprint rows. Use J20's final kit on backup01 and verify its measured executable hashes. The saved original-manifest check marks `run-case.sh` failed because it was deliberately adapted for Caddy; every original executable matches. The final nine changed/added kit source files were compared byte-for-byte with the reviewable repository copies after measurement and matched. Copy the new Caddy executables and kit scripts **before** starting any run; freeze the kit throughout the measurements. Run all workloads via `bash run-all.sh CASE`, which calls `perf-run` and explicitly passes overrides inside its sanitized environment. Use a distinct `OUT` for each repetition. Preserve nonzero exits and before/after state diffs, including failed staging attempts. Never edit a shell script while a running shell is reading it. The scenario and client code are J20's unchanged binaries: 5 s warmup + 30 s measurement, h1 concurrency 1/64/512, h2 eight connections with eight parallel requests each, 15 s idle CPU, 1,000 TLS keep-alive connections, full TLS handshakes c1/c64 for 30 s without resumption, and three reload trials. Two proxy CPUs; backend and generator share the third. CPU 100% is one core. A zero idle CPU sample means no observed CPU ticks in 15 seconds, not proof of zero wakeups. RSS/HWM cover only the proxy process; no forced GC is used. These are workload results with a generator/backend ceiling, not maximum engine capacity. J20's other engines are historical reference rows, not new same-session measurements. The Caddy adapter generates native JSON, disables automatic HTTPS for the common externally issued certificate workload, and binds admin to 127.0.0.1:12019 with persistence disabled. A 2 ms research watcher, started only for the reload phase and pinned to the backend/generator CPU, translates the common checker's atomic route-file replacement into `POST /load`. Certificate PEMs are embedded in the submitted JSON so rotation changes its content and does not depend on filename-cache behavior. Switch times include certificate creation, watcher observation, API application and the common client's polling interval. This adapter is not product implementation. Traefik retains J20's file-provider defaults, including its update debounce. These timings include the different update paths; they are not isolated configuration-parser or API latency measurements. `stream_close_delay` is explicitly 5 minutes for the configured Caddy runs. That bounds retention of old streaming configuration; it is not a promise that indefinitely open streams survive indefinitely. The default behavior is checked separately. Both successful HTTP responses and surviving old connections must be reported; a fast configuration switch is not proof of zero drops. The memory samples precede reload; they do not quantify retaining old WebSocket configurations across frequent updates or long-lived streams. The HWM column is J20's post-HTTP-load sample, not a whole-lifecycle peak. ## Control surface and TLS Caddy's native JSON admin API supports full `POST /load`, subtree changes and inspection. `/load` blocks until application succeeds or fails; upstream documents rollback to the previous working configuration on failure. The default local endpoint is localhost:2019. For a supervised product, prefer a permissioned Unix socket owned by the infraternal agent, serialize desired revisions, and probe the intended route/certificate before marking it applied. Loopback alone does not isolate untrusted local processes. Set `admin.disabled: true` (Caddyfile `admin off`) to disable administration; then ordinary API/CLI hot reload is unavailable and changes require another lifecycle strategy. `admin.config.persist: false` prevents automatic config snapshots, relevant when configuration contains DNS credentials. [API](https://caddyserver.com/docs/api), [admin schema](https://caddyserver.com/docs/json/admin/). WebSocket upgrades are supported by the reverse proxy. Upstream explicitly documents closing streams on config unload by default, and the `stream_close_delay` mitigation. HTTP graceful shutdown and reconnect behavior still need the common checker; the documentation's zero-downtime wording is not evidence that every existing client connection remains open. The versioned HTTP app stop path invokes Go `Server.Shutdown` during reload; `shutdown_delay` is skipped for listeners reused by the new configuration. The common checker deliberately does not hide failures with automatic request retries. Closed idle connections/HTTP2 GOAWAY can be handled by ordinary client reconnects; a checker failure is not proof that the listening port disappeared. [Versioned stop path](https://github.com/caddyserver/caddy/blob/v2.11.7/modules/caddyhttp/app.go). [Reverse proxy](https://caddyserver.com/docs/caddyfile/directives/reverse_proxy). Automatic HTTPS includes issuance/renewal. **On-demand TLS is different from downloading a proxy on demand:** it postpones certificate acquisition until the first TLS handshake for a name, holding that handshake while issuance completes. It requires an authorization/permission policy (for example an internal `ask` endpoint) so arbitrary SNI cannot cause unrestricted issuance. For known managed names, eager issuance avoids that first-user latency. DNS-01 requires a provider module; the stock binary does not contain the Cloudflare provider. [Automatic HTTPS](https://caddyserver.com/docs/automatic-https), [Cloudflare module](https://github.com/caddy-dns/cloudflare/tree/v0.2.4). The local DNS-01 check uses J20's Pebble 2.10.1/challtestsrv and disposable CA, validation sleep disabled, two names, and no real credentials. A tiny separate provider module calls the stub's TXT management API; a loopback resolver adds SOA answers for Caddy's zone discovery and forwards other queries to challtestsrv. Only DNS-01 is enabled; propagation polling is disabled because this is local test DNS. This proves Caddy's ACME/DNS-01 path, not live Cloudflare authentication, public propagation, rate limits or long-running renewal reliability. The measured Cloudflare module is installed but unused in the ordinary static-certificate workloads. A companion native Traefik ACME run uses its bundled DNS exec provider and a local TXT adapter against the same CA/stub, with p-256 leaf keys for both engines. Neither probe uses a real provider credential; concurrent issuance scheduling and polling are engine-specific, so total elapsed time is not a standalone ACME-library benchmark. ## Distribution, verification and maintenance xcaddy creates a small Go main importing Caddy and requested modules, resolves their Go dependencies, and compiles one executable. Modules are compiled in, not shared objects loaded at runtime. Pin Caddy, each module, Go, build flags and the resolved dependency closure. The website custom-download service builds combinations selected by OS/architecture/version/module list; Caddy's `upgrade`/`add-package` commands fetch builds through that service. It removes local build tooling from the installation path but adds service availability and artifact-trust dependencies. For a managed fleet, build once in controlled CI and distribute a reviewed, signed component manifest and exact hashes. [Build documentation](https://caddyserver.com/docs/build), [xcaddy 0.4.7](https://github.com/caddyserver/xcaddy/tree/v0.4.7), [CLI](https://caddyserver.com/docs/command-line). Official GitHub release artifacts have Sigstore signature/certificate assets and a documented transparency-log verification process. A valid signature must be checked against the expected workflow identity and tag, not merely against whichever certificate accompanies a download. This does **not** establish equivalent verification for custom API builds. Upstream's open website issue reports different build tags between the API and GitHub releases and no published checksum/signature reference for API builds. Treat that as an upstream report, not a reproduced measurement of 2.11.7. The inspected 2.11.7 CLI download path reads the HTTPS response into the new binary without an artifact-signature verification step. [Versioned download code](https://github.com/caddyserver/caddy/blob/v2.11.7/cmd/packagesfuncs.go). Our own xcaddy outputs have hashes and reproducibility evidence, but no upstream release signature over the custom executable. This run checked the official archive against its published checksum; it did not execute the Sigstore verification procedure. Checksum equality alone is not an independent publisher-identity verification. [Signature verification](https://caddyserver.com/docs/signature-verification), [upstream website issue 549](https://github.com/caddyserver/website/issues/549). Caddy and the pinned Cloudflare module use Apache-2.0, compatible with the AGPL core; additional modules need their own dependency/license review. Caddy has public history from 2015, Caddy 2 since 2020, and is maintained by Matt Holt and community maintainers, with ZeroSSL/HID Global stewardship and sponsorship. It is a mature product, not a pre-1.0 framework. Recent releases in the snapshot include 2.11.2 (March 6), 2.11.3 (May 12), 2.11.4 (June 3), 2.11.6 (October 1) and 2.11.7 (October 3). [Project](https://github.com/caddyserver/caddy), [ten-year history](https://caddy.community/t/thanks-for-10-years-of-caddy/29615), [releases](https://github.com/caddyserver/caddy/releases), [license](https://github.com/caddyserver/caddy/blob/v2.11.7/LICENSE), [module license](https://github.com/caddy-dns/cloudflare/blob/v0.2.4/LICENSE). The snapshot contains 18 published Caddy GitHub advisories. Counts cannot rank safety against Traefik's 89 in R2: features, usage and disclosure differ. Relevant examples include local admin cross-origin configuration application, remote admin authorization bypasses, mTLS fail-open on malformed/missing CA files, and forward-auth connection reuse/identity risks. The forward-auth upstream-confusion and hop-by-hop-header findings list fixes in 2.11.5; our 2.11.7 pin is newer. Track Go and plugin advisories as well as Caddy's. Do not assume that a minimal enabled feature set makes every core HTTP/auth finding irrelevant. The Cloudflare module's pinned source includes the supplied token in its invalid-token error; a future custody implementation must prevent that error text reaching ordinary logs. [Advisories](https://github.com/caddyserver/caddy/security/advisories), [forward-auth finding](https://github.com/caddyserver/caddy/security/advisories/GHSA-6365-7ppr-5r92), [pinned module source](https://github.com/caddy-dns/cloudflare/blob/v0.2.4/cloudflare.go). The separate runtime API probe passed: invalid JSON module configuration returned 400 while retaining the exact prior exported config and route; a valid load returned 200 and changed the response; a Unix socket beneath a 0700 directory accepted a load; disabling admin removed both administration endpoints while the HTTP route remained available. Its perf-run state diff was empty. The final kit records the workload exit separately from the global cleanup comparison. Other jobs joining/leaving the shared perf-lock queue can change the global process snapshot while doing no measured work; those raw diffs are retained and must be attributed explicitly. Empty socket, kit and temporary-file diffs plus the owned-process sweep are separate cleanup evidence, not a reason to erase the global diff. Both engines support a product-owned UI/model and supervised sidecar shape. Caddy offers a native writable configuration API and lower measured process footprint; Traefik includes DNS providers and request-rate middleware in its standard distribution and has a dynamic provider model. Caddy has built-in auth/header/rewrite/reverse-proxy facilities, but stock request-rate limiting and general TCP/UDP routing should not be assumed; those need additional modules and measurements. Neither engine's schema should become the public infraternal API. ## Measurement results The following tables come from the four final frozen-kit runs. The final order is Cloudflare Caddy, Traefik B, the two native DNS-01 probes, stock Caddy, Traefik A. No binary transfer, source edit or bulk checksum runs overlap these four primary measurements. Earlier stock runs overlapped binary staging and are retained as calibration only. Early attempts also changed an active parent shell script and produced post-workload parse errors; they are explicitly excluded from the final comparison. R3 same-session order: Cloudflare Caddy → Traefik B → stock Caddy → Traefik A. DNS-01 probes ran between the pairs. | Engine | Idle MiB | Warm MiB | 1,000 TLS connections MiB | Added KiB/connection | HTTP VmHWM MiB | Executable B | |---|---|---|---|---|---|---| | Caddy stock | 47.54 | 50.20 | 96.45 | 47.37 | 147.12 | 51,953,826 | | Traefik A | 93.41 | 95.50 | 150.01 | 55.82 | 275.19 | 182,120,608 | | Caddy + Cloudflare | 47.93 | 48.78 | 94.25 | 46.56 | 139.78 | 52,019,362 | | Traefik B | 93.97 | 95.86 | 150.21 | 55.65 | 275.09 | 182,120,608 | | J20 historical reference | Idle MiB | Warm MiB | 1,000 TLS connections MiB | Added KiB/connection | HTTP VmHWM MiB | |---|---|---|---|---|---| | Pingora | 6.66 | 8.09 | 64.01 | 57.26 | 63.88 | | Sozu | 8.19 | 10.04 | 52.36 | 43.34 | 52.10 | | rpxy | 6.14 | 6.68 | 44.25 | 38.47 | 59.39 | | Traefik | 93.30 | 95.77 | 150.27 | 55.81 | 265.84 | | Engine | h1 c1 req/s / p99 ms | h1 c64 | h1 c512 | h2 c64 | |---|---|---|---|---| | Caddy stock | 5,027 / 0.568 | 20,776 / 9.480 | 12,577 / 73.882 | 15,738 / 10.863 | | Traefik A | 5,609 / 0.508 | 30,340 / 8.024 | 23,241 / 42.226 | 23,604 / 7.917 | | Caddy + Cloudflare | 4,950 / 0.587 | 22,009 / 8.776 | 13,198 / 69.576 | 17,417 / 9.335 | | Traefik B | 5,104 / 0.617 | 26,888 / 8.895 | 20,643 / 53.022 | 23,036 / 8.030 | | J20 historical reference | h1 c1 req/s / p99 ms | h1 c64 | h1 c512 | h2 c64 | |---|---|---|---|---| | Pingora | 7,140 / 0.328 | 45,372 / 2.981 | 34,965 / 38.203 | 35,246 / 3.673 | | Sozu | 11,489 / 0.140 | 50,052 / 3.161 | 38,217 / 45.940 | 29,444 / 4.482 | | rpxy | 10,394 / 0.151 | 52,315 / 2.120 | 38,928 / 45.097 | 35,733 / 3.119 | | Traefik | 6,418 / 0.224 | 32,956 / 7.296 | 25,711 / 39.075 | 25,289 / 7.393 | | Engine | Idle CPU % | h1 c1 CPU % | h1 c64 CPU % | h1 c512 CPU % | h2 c64 CPU % | TLS c64 CPU % | |---|---|---|---|---|---|---| | Caddy stock | 0.00 | 89.68 | 192.46 | 191.06 | 190.59 | 158.80 | | Traefik A | 0.00 | 83.31 | 191.84 | 179.72 | 194.07 | 158.68 | | Caddy + Cloudflare | 0.00 | 89.54 | 193.23 | 189.54 | 192.01 | 159.01 | | Traefik B | 0.00 | 83.61 | 191.33 | 182.12 | 194.43 | 158.21 | | Engine | Full TLS/s c1 | p50 / p99 ms | Full TLS/s c64 | p50 / p99 ms | TLS failures | |---|---|---|---|---|---| | Caddy stock | 1,830 | 0.461 / 1.698 | 5,049 | 12.570 / 19.026 | 0 | | Traefik A | 2,058 | 0.446 / 0.856 | 5,251 | 12.201 / 17.476 | 0 | | Caddy + Cloudflare | 2,127 | 0.426 / 0.872 | 5,243 | 12.253 / 16.466 | 0 | | Traefik B | 2,067 | 0.444 / 0.845 | 5,274 | 12.154 / 17.402 | 0 | | Engine | Failed requests 1 / 2 / 3 | Route switch ms | Certificate switch ms | Old h1 | WebSocket | Old h2 | |---|---|---|---|---|---|---| | Caddy stock | 3 / 3 / 2 | 17 / 13 / 13 | 17 / 12 / 12 | closed / closed / closed | alive / alive / alive | closed / closed / closed | | Traefik A | 0 / 0 / 0 | 13 / 579 / 1035 | 12 / 579 / 1035 | alive / alive / alive | alive / alive / alive | alive / alive / alive | | Caddy + Cloudflare | 3 / 2 / 2 | 16 / 14 / 16 | 16 / 13 / 15 | closed / closed / closed | alive / alive / alive | closed / closed / closed | | Traefik B | 0 / 0 / 0 | 69 / 528 / 1083 | 69 / 527 / 1083 | alive / alive / alive | alive / alive / alive | alive / alive / alive | The added-connection column is `(RSS at 1,000 - warm RSS) / 1,000`, in KiB. Cloudflare adds **64 KiB of executable bytes**. Its sampled RSS is not consistently higher than stock; do not turn heap/run variability into a causal module-cost claim. All four runs held 1,000/1,000 connections, passed 9/9 basic checks before and after reload (including WebSocket proxying), and had zero steady HTTP and full-handshake errors. Slow in-flight requests completed in every reload trial. The default-stream diagnostic, without `stream_close_delay`, closed WebSockets in all three trials. The proposed idle saving is about **46 MiB per active proxy process**, not 76 MiB implied by 17 versus 93. For illustration, 10/100 active proxy processes save about 460 MiB/4.5 GiB idle. These are arithmetic projections, not measured scaling or a prediction that every node runs a proxy. Each process's 1,000-connection footprint remains in the table; no linear extrapolation to arbitrary connection counts is established. Traefik delivered 1.22–1.46 times Caddy's h1 c64 rate and 1.56–1.85 times its h1 c512 rate in these pairs. H2 also favored Traefik. TLS c64 results were close (roughly 5.0–5.3k full handshakes/s), while stock Caddy's c1 sample was slower. There is one full run per Caddy artifact and two Traefik repetitions, not a confidence interval. Traefik's own c64 range (26,888–30,340 req/s) demonstrates variability. Hypervisor steal was 0.98% / 1.63% / 1.98% / 1.05% in chronological order; all runs passed the wrapper's admission check. Direct h1 c64 was 56,451 req/s, versus J20's historical 63,506. Do not compare the historical engines as if the host conditions were unchanged or interpret this constrained generator as peak server capacity. | Probe | Result | Process RSS / VmHWM | Workload / wrapper exit | |---|---|---|---| | Caddy native JSON API | invalid load rollback, valid load, private Unix socket, admin off passed | separate functional probe | 0 / 0 | | Caddy native DNS-01 | 2 local Pebble certificates in 1,539 ms, both names served | 52.77 / 52.77 MiB | 0 / 0 | | Traefik native DNS-01 | 2 local Pebble certificates in 3,258 ms, both names served | 101.18 / 101.18 MiB | 0 / 1 | | Caddy stock primary | strict reload preservation failed; other checks passed | above | 1 / 1 | | Caddy + Cloudflare primary | strict reload preservation failed; other checks passed | above | 1 / 1 | | Traefik A primary | all workload and cleanup checks passed | above | 0 / 0 | | Traefik B primary | all workload checks passed | above | 0 / 1 | The stock run and API/Caddy DNS probes had empty global state diffs. Cloudflare and Traefik B contain only other jobs joining the perf-lock queue in the process diff. Traefik DNS contains a new external SFTP process. All final socket, kit and temporary-file diffs were empty. The final owned process/socket sweep and removal of the R3 kit are recorded separately. An unrelated global diff never turns a failed Caddy workload into a pass, and a wrapper exit of 1 never gets silently discarded. The DNS totals are native end-to-end issuance scenarios with local DNS and validation delay disabled, not a public-provider comparison. Stock Caddy cannot run the Cloudflare DNS provider; the measured Cloudflare artifact has that module installed, while the separate stub artifact exercises local DNS-01 without any live provider credential. J20's standalone ACME example issued two certificates in 1,789 ms with 4,988 KiB peak RSS and a 2,990,800 B executable; that is a different program, not a sidecar RSS row. ### Evidence and verification [Complete evidence bundle](https://git.kayg.org/attachments/4510f483-cbe2-4e8f-8909-fe4d002d2848), SHA-256 `0d43623c1e396f09831e16a844129093bba0d10954aaf24911e5bb2f4c303f43`. [Raw perf archive](https://git.kayg.org/attachments/0608c9d3-a178-42f0-a004-0d36a23946fc), SHA-256 `ed0cd3a4a46386a5e59a3f26243921504c7bc46d8a37f38af9cfe7bb4665700c`. Both are attached to the [measurement comment](https://git.kayg.org/kayg/infraternal/issues/261#issuecomment-72620). The bundle includes all raw runs/receipts, failed attempts, generated build closure and hashes, source snapshots, final source/syntax checks, the complete unedited `verify.log`, and cleanup records. No executables, generated private keys or compilation caches are attached. All 18 archived harness files match the VM verification source; all nine measured kit files match that source. `scripts/verify fast` completed through compute01's held build slot with exit 0. Product code/dependencies are unchanged. Both static artifacts match their D243 caps exactly. Verbatim selected gate output (the complete output is in the bundle): ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | static builds | PASS | +---------------------------+--------------+ PT_INTERP: absent; size: 21431472 bytes; cap: 21431472 bytes; delta: +0 bytes PT_INTERP: absent; size: 19605896 bytes; cap: 19605896 bytes; delta: +0 bytes 39 pass 0 fail 50325 expect() calls Ran 39 tests across 10 files. [4.31s] R3 scripts/verify fast exit=0 ``` The R3 perf kit, result copies and transfer archive were removed after the copied archive's SHA-256 matched; the original J20 final kit is preserved. The owned Rust target, Go compilation caches/toolchain, Caddy build artifacts and six exact recorded xcaddy temporary directories were removed. The final process sweep found no surviving owned target executable or private-fixture process. Small source worktrees and evidence remain on compute01; no R3 background work remains. ## Owner grill Recommendation for the owner to consider: retain the proposed managed Traefik component if preserving existing connections across route and certificate changes is a hard requirement. Caddy is a credible lower-memory alternative if graceful client reconnects and a controlled custom-module build/update pipeline are acceptable. It is mature and has a useful native JSON control surface; a smaller process alone does not resolve the lifecycle contract or replace middleware/provider coverage measurements. **Does the saved per-node memory justify accepting Caddy's observed h1/h2 connection closure on reload, bounded WebSocket retention, and our own Cloudflare-module artifact pipeline, or should D316 retain Traefik for its observed connection preservation and bundled providers/middleware?** This is a measured trade-off for kayg, not an engine selection by R3. The answer may be to relax the connection contract, but that is an explicit owner decision rather than relabeling a failed zero-drop trial as a pass.

kayg commented 2026-11-01 09:58:10 +00:00

Author

Owner

Copy link

Owner decision (kayg, 2026-11-01), final: reverse-proxy engine = Traefik, downloaded on demand (R3 comparison 72700 on #261). Caddy idle RSS measured 48 MiB (not 17), half of Traefik's 94 MiB, but it drops existing connections on every config reload (Traefik kept all, 6/6), is slower under load (c512 13k vs 21–23k req/s; p99 70–74 vs 42–53 ms) and needs a custom build for Cloudflare DNS-01. Connection-preserving reloads are required because the UI changes proxy config while terminals/canvas run over long-lived WebSockets.

**Owner decision (kayg, 2026-11-01), final: reverse-proxy engine = Traefik, downloaded on demand** (R3 comparison 72700 on #261). Caddy idle RSS measured 48 MiB (not 17), half of Traefik's 94 MiB, but it drops existing connections on every config reload (Traefik kept all, 6/6), is slower under load (c512 13k vs 21–23k req/s; p99 70–74 vs 42–53 ms) and needs a custom build for Cloudflare DNS-01. Connection-preserving reloads are required because the UI changes proxy config while terminals/canvas run over long-lived WebSockets.

kayg referenced this issue 2026-11-01 09:58:11 +00:00

Reverse proxy: Traefik on demand — implementation plan #281

kayg referenced this issue 2026-11-01 22:46:26 +00:00

Reverse proxy: Traefik on demand — implementation plan #281

kayg referenced this issue 2026-11-02 01:17:08 +00:00

Secrets: an intuitive interface over OpenBao #287

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

j470b-server-mesh-rejoin

mesh-compat-382

j444b-d381-key-wins-flood

j461b-macos-utun-reader-thread

j475b-remove-host-motion

fix-reconnect-new-link

j481-d380

dev

j479-mesh-soak

fix/j476-host-facts

j475-motion-pill-split

j470-mac-node-join

root-upgrade-d376

j474-review-merge

latency-targets-120hz

nightly-suite-shards

gotatun-pin-j469

macos-utun-reader-thread

fix-flood-echo-stall

wip/j416

motion-fast-ease-out

mesh-addhost-j259

takeover-close-terminals-d368

kbd-instant-followups

fix-trust-key-double-click

nightly-suite-partial-report

appwin-icons-j207

rusqlite-slice1

swipe-content-first-d374

mesh-mtu-381

addhost-instant-open

glass-contrast-d365-mock

glass-contrast-d365

signout-followups-442

fix-startup-toast-red

research-gotatun-matrix-j317

themes-sync-j304

fix-load-flakes

fix-nix-zig-patch

fix-compute01-browser-baseline

ratchet-size-baseline-419

predict-auto-d130

revoke-coverage-d367

nightly-suite-queue-timeout

nightly-browser-suite

fix-rustsec-2026-0459

fonts-ioskeley-obel

log-retention-d375

fix-kbd-no-anim-427

fix-dev-red-specs-2

takeover-proof-305-j205

wip/j410-server-pump

ui-consistency-d372

fix-osc52-clipboard

libghostty-vt-1.0

forget-retired-host-d369

j404-merge-426

fix-setup-lock-race

fix-touch-select-426

revoke-access-report-ui

fix-predictive-echo-specs

fix-selinux-agent-exec

wip/j395-checkpoint2

wip/j395

j391-review-scratch

ui-consistency-j195-deferred-rebased

fix-rustc-paths-421

decisions-wording-341

fix-install-suite-red

fix-dev-rtt-specs

fix-pending-removal-align

fix-privileged-skip

fix-install-suite-404

review-j368

mesh-dns-j254

revoke-access-report-d356b

revoke-privilege-round5

fix-removal-confirm

review-j360

gotatun-rotation-367

deps-refresh-340

review-j354

review-J351

review-j350

addhost-takeover-d359

revoke-report-items-347

revoke-end-sessions-d362

review-j348

size-mesh-ssh-346

size-trim-337

revoke-access-diff-d356

j259-pre-rebase-b06c5a6

j305-ended-at-start-spec

addhost-sudo-consent

fix-layout-real-specs-411

fix-shell-open-411

fix-console-hosts-373

fix-shell-exit

fuzz-host-name-353

fix-host-details-410

macos-filter-j197

research-gotatun-tricks

revoke-p3-j216l

stale-state-pill

revoke-followups-j216k

revoke-progress-j216e

backup/macos01-20261106/lume/j197

backup/macos01-20261106/lume/appwin-icons-j207-stash0

backup/macos01-20261106/claude/stash-0

backup/macos01-20261106/claude/stash-1

backup/macos01-20261106/claude/J80-macos-s1

backup/macos01-20261106/claude/J80

backup/macos01-20261106/claude/J80-slice1-20261102

backup/macos01-20261106/claude/J16b-appwin-on-dev

backup/macos01-20261106/claude/appwin-launch-idle-146

backup/macos01-20261106/claude/J80-s1-run

wip/j259

trust-285-p3c

runtime-lock-j216f

mesh-ssh-p3-j175

tun-proof-p3b

trust-285-p3

terminal-polish-j271

tun-proof-p3

ssh-install-flakes-j216i

fuzz-fixes-j264

trust-285-a

trust-285-b

restart-oversized-fixture-j198

backup/macos01-20261106/lume/appwin-icons-j207

restart-bound-j216h

up-preflight-p3-j198

backup/macos01-20261106/lume/j251

up-preflight-j198

wip/up-preflight-j198-j245-p2

wip/up-preflight-j198-j245-p1

agent-fsync-offloop-j196

wip/up-preflight-j198-j245

settings-offscreen-j231

wip/up-preflight-j198-j226-inspection

wip/j103f

themes-j206

wip/up-preflight-j198-j226

flakes-j216c

backup/macos01-20261106/claude/j197

flakes-j216b

backup/macos01-20261106/claude/j197-red

wip/up-preflight-j198-review

flakes-j216

backup/macos01-20261106/claude/appwin-icons-j207

engineering-j208b

appwin-mock-j209

toast-stack-j165

engineering-j208

decisions-d351

decisions-d350

decisions-d349

appearance-presets-mock-j129

decisions-d348

docs-audit-spec

doc-coverage-baseline-raise

decisions-d347

decisions-d346b

decisions-d346

j88h-profile

ui-consistency-j195-deferred

ui-consistency-j195

engineering-practices-j194

preferred-login-ui-j166

signin-pools-263-j174

macos-node-265-s1

decisions-d345

decisions-d344

proxy-cgroup-j164

verify-quick-fuzz-329

proxy-start-io-j168

timing-flakes-j163

up-263-s1

mcp-events-j147

verify-quick-fuzz-j158

proxy-p3-j159

agent-resource-j157-detect

wip/j103d-final

remote-cancel-hang-j155

proxy-traefik-281-j62d

j149-verify-quick

proxy-traefik-281-j62b

j150-drop-unnamed

wip/j79d-red-r4

wip/j103d-d332

rootless-toast-j142

wip/j103d-head

wip/j103d

j125c-mcp

macos-build-j136

backup/macos01-20261106/claude/j136

wip/j79d-red-cli

wip/j79d-red-server

j121-link-liveness

j118c-tracer-failures

wip/j79c-head

release-desktop-j131

j124c-signed-macos

wip/j79c-red-p3

wip/j79c-red-p23

wip/j79c-red-p22

merge-j128-check

perf-deferred-j128

sidebar-toggle-260

j127-shortcut-changed-dot

j125-mcp

j118-tracer-failures

j126-account-summary

settings-sections-mock-j92

j120-pinch-lock

sidebar-292

backup/macos01-20261106/claude/j120

taste-2026-11-03

row-actions-242-215

backup/macos01-20261106/claude/stash-0-untracked

backup/macos01-20261106/claude/J80-265-s1-run

backup/macos01-20261106/claude/J80-macos-s1-20261102

j112-canvas-commands

ui-no-overshoot-scroll

fix-recovery-restart

backup/macos01-20261106/claude/stash-1-untracked

taste-2026-11-02

size-baseline-ratchet

machine-cards-structure-j108

j109-boot-copy-mockups

fix-dev-red

wip/up-263-s1

j95b-settings-rows

proxy-traefik-281

j95-option-rows

docs-human-j91

docs-keys-palette

fix-flood-1013-270-j38d

mesh-ssh-s1

mesh-ssh-s0

addhost-d291-tunnel

test-stability-load

mesh-relay-220

repro-296

shim-thp-off-24

mesh-pins-258-v2

mesh-dns

signin-d311-merged

docs-pages

ui-size-283

fix-ui-batch-specs

wip/j12b-edge-update

gate-on-arm

fix-rename-sync-293

trust-285-j60b

fix-focus-ring-clip

integ-ui-1101

edge-update-v2

signin-d311

taste-2026-11-01

layout-per-tab-250

ui-approved-splitopt-224

ui-approved-splitopt-224-on-dev

icon-text-centre-282

ui-approved-189-212-216

research-reverse-proxy

transfer-budget-320

review-mesh-pins-258-r2

sidebar-pill-259

review-mesh-relay-220

wip/j12-edge-update-v2

nix-package

status-crash-loop-237

fix-dev-test-failures

fix-fixture-socket-path

wip/mesh-relay-220-pre-dev-advance-rebase

fix-canvas-side-clip

research-head-to-head

frosted-real-256

review-mesh-pins-258

wip/mesh-relay-220-pre-final-rebase

review-layout-per-tab-250-r2

broad-suite-triage

nix-devshell

review-status-crash-loop-237-r3

wip/nix-devshell-pre-p3

taste-frosted-real

wip/code-serve-demo-2026-10-30

wip/code-serve-glass-2026-10-30

wip/code-serve-clusters-2026-10-30

wip/code-merge-release94-dev-2026-10-30

wip/rproxy-delta-scratch-2026-10-30

wip/stale-note-257-staged-2026-10-30

wip/d241-size-staged-2026-10-30

review-broad-suite-207

wip/mesh-pins-258-pre-split

wip/mesh-relay-220-pre-rebase

mesh-measure-220

plan-211-update

review-edge-autoupdate-r3

perf24-reconnect

review-stale-note-r2

release-prod-live

d241-shell-marker

review-mesh-pins-server-258

review-revocation-r3

review-setup-token-r2

review-sync-journal-r3

wip/review-mesh-pins-258-prune-review-1995fee46

frosted-scroll-256

sync-journal-r3

revocation-per-account-wip

review-tokio-asyncfd-0386

review-sync-p3s

wip/stash-8by64-stash0-2026-10-30

mesh-drag-r2

setup-token-data-wip

release-9-4

auth-busy-1013

mesh-view-overlap

sync-p3s

review-auth-busy-wake

review-readd-r8-wip

mesh-drag-bounds

audit-auth-93

app-windows-stream-start

review-authority-cap-r3

app-windows-r7-measure

readd-minimal-r6

readd-minimal-r5

app-windows-r7-soak

readd-minimal-r4

app-windows-r7

app-windows-r6

readd-minimal-r3

readd-minimal-r2

readd-minimal

merge-release8-dev

review-busy-page-liveness-r2

review-busy-page-liveness

mesh-view-trim

review-authority-read-retry-r2

review-remove-uninstall-r2

release-retry-reattach

release-retry

review-remove-uninstall

review-authority-read-retry

app-windows-ship

mesh-autoswitch-p2

mesh-ssh

mesh-relay

mesh-macos-node

screen-and-control

session-labels

review-real-logins-r5

review-real-logins-r4

review-real-logins-r3

shim-memory-r2

review-real-logins-r2

shim-memory

shadow-addhost-instant

shadow-real-logins

shadow-same-session-flood

shadow-prod-perf-fix

hosts-from-server

app-windows-real

app-windows-transport

release-prod-6w

mesh-policy-verify

quiet-close-toast

dev-merge-r4

review-mesh-agent-transport

review-app-windows-transport

settings-roundtrip

review-release-pipeline-r4

mesh-agent-transport

ended-strip-fix

review-audit-ca-update-d244

settings-rows

phone-width

signout-clear

fix-restart-tests

dev-merge-rp

signer-hardening

review-audit-ca-update

terminal-fidelity

review-audit-ca-update-wip

mesh-integrate

review-mesh-agent-transport-515d3712

review-mesh-dataplane-feec1988

mesh-dataplane

signout

research-canvas-node

wip/review-renderer-cost

wip/review-terminal-fidelity

padlock-glyph

review-release-pipeline-2e95b689

phone-polish

sidebar-polish

inbox-generic

review-mesh-dataplane-378d4055

prod-batch

ui-inbox

ui-ca-unlock

ui-machine-sections

review-mesh-dataplane-a5f744fb

tls-session-prod

session-bounds-prod

fix-dead-session

session-bounds

tls-session

palette-polish

shortcuts-d222

research-appmeta

phone-gutter

ui-centre-button

wip/ui-centre-button

wip/tls-session

review-addhost-breakit

ui-platform

renderer-cost

remote-agents

mesh-api

ssh-bootstrap

prod-hotfix2

prod-fixes

review-remote-agents

review-mesh-dataplane

session-life

research-browser

review-mesh-api

research-capture

review-mesh-api-round3

review-mesh-api-round1

gate-stability

review-gate-stability

review-relay-direct

review-mesh-api-findings

proto-sessions

font-picker

proto-addhost

proto-windows

mesh-core

latency-gate

research-bsd

research-discovery

keytray-mods

mock-core

ui-tweaks

fix-motion-sel

fix-motion

integrate2

fix-robust

t-web

t-server

vt-ghostty

fix-snap-echo

t-web-auth

review-vt2

t-agent

lane-protocol

lane-shim

lane-auth

s1-base

s1-contracts

s1-workspace

No results found.

Labels

Clear labels   

launch-qa

Findings from launch QA.

No labels

launch-qa

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/infraternal#261

No description provided.