Mesh: measure WireGuard userspace engines and Linux kernel control (D163, D167) #81

Closed
opened 2026-10-07 11:12:18 +00:00 by kayg · 20 comments
Owner

Measure WireGuard engine candidates for the mesh (D163, D167). The job measures and reports; the owner decides (foundational-choice rule).

  1. Userspace engines for macOS and Windows nodes (AGPL-compatible licences only): throughput, added latency, CPU per Gbit, memory per peer × 100/1,000 peers, binary size added, maintenance and licence.
  2. Linux kernel control: how to configure kernel WireGuard from Rust (netlink crates vs our own minimal netlink code): correctness, binary size, dependencies, licence, speed of applying 1,000 peers.

Rootless harness: user + network namespaces (unshare -rn), where creating a wireguard link works on infraternal-dev (verified). Results in research/mesh-engines/REPORT.md with the raw numbers.

Measure WireGuard engine candidates for the mesh (D163, D167). The job measures and reports; the owner decides (foundational-choice rule). 1. Userspace engines for macOS and Windows nodes (AGPL-compatible licences only): throughput, added latency, CPU per Gbit, memory per peer × 100/1,000 peers, binary size added, maintenance and licence. 2. Linux kernel control: how to configure kernel WireGuard from Rust (netlink crates vs our own minimal netlink code): correctness, binary size, dependencies, licence, speed of applying 1,000 peers. Rootless harness: user + network namespaces (`unshare -rn`), where creating a wireguard link works on infraternal-dev (verified). Results in research/mesh-engines/REPORT.md with the raw numbers.
Author
Owner

Started on branch mesh-engines from base 159f437. Reading maintained userspace and Rust netlink candidates; next I will run reproducible host benchmarks and record platform limits.

Started on branch mesh-engines from base 159f437. Reading maintained userspace and Rust netlink candidates; next I will run reproducible host benchmarks and record platform limits.
Author
Owner

Host check: rootless successfully created/deleted a kernel wg link and a TUN device; is available. succeeds. Machine: 8-core i9-9900K, Debian 13, kernel 6.12.111; load average at check 8.03/5.54/7.10.

Host check: rootless successfully created/deleted a kernel wg link and a TUN device; is available. succeeds. Machine: 8-core i9-9900K, Debian 13, kernel 6.12.111; load average at check 8.03/5.54/7.10.
Author
Owner

Correction: the previous comment was shell-expanded. Actual probes: unshare -rn created/deleted a kernel WireGuard link successfully; /dev/net/tun exists and ip tuntap add/delete succeeded; systemd-run --user --scope -p CPUWeight=10000 true exited successfully. Host: 8-core i9-9900K, Debian 13, kernel 6.12.111; load average 8.03/5.54/7.10 at the probe.

Correction: the previous comment was shell-expanded. Actual probes: unshare -rn created/deleted a kernel WireGuard link successfully; /dev/net/tun exists and ip tuntap add/delete succeeded; systemd-run --user --scope -p CPUWeight=10000 true exited successfully. Host: 8-core i9-9900K, Debian 13, kernel 6.12.111; load average 8.03/5.54/7.10 at the probe.
Author
Owner

Candidate build step complete: BoringTun 0.7.1 release binary 1.6 MiB and GotaTun 0.9.2 default AWS-LC release build succeeded; Go wireguard-go stripped static binary is 3.2 MiB. GotaTun audit file audits/2026-02-17-Assured.md reports 12 findings (2 low, 9 notes, 1 good), with no major vulnerabilities; it flags unsafe code and maintenance concerns in the tun dependency. Next: five-run paired tunnel and netlink control measurements.

Candidate build step complete: BoringTun 0.7.1 release binary 1.6 MiB and GotaTun 0.9.2 default AWS-LC release build succeeded; Go wireguard-go stripped static binary is 3.2 MiB. GotaTun audit file audits/2026-02-17-Assured.md reports 12 findings (2 low, 9 notes, 1 good), with no major vulnerabilities; it flags unsafe code and maintenance concerns in the tun dependency. Next: five-run paired tunnel and netlink control measurements.
Author
Owner

Finding: nlink 0.29.0 full-set apply with 1,000 peers returned EINVAL: bytes leftover after parsing attributes on kernel 6.12.111. Its set path emits one peer-list request and does not chunk. I will load the 1,000-peer baseline through the chunking client to measure nlink single-peer update and removal separately.

Finding: nlink 0.29.0 full-set apply with 1,000 peers returned EINVAL: bytes leftover after parsing attributes on kernel 6.12.111. Its set path emits one peer-list request and does not chunk. I will load the 1,000-peer baseline through the chunking client to measure nlink single-peer update and removal separately.
Author
Owner

Candidate survey addendum: NepTUN is a maintained BSD-3-Clause Rust engine. GitHub Release v1.0.8 is dated 2025-07-21; upstream main was updated 2026-10-06 (source commit ce185155e1639403cc40e01260d0830065a49555). It has a Linux/macOS CLI and Windows library target, but the library leaves tunnel I/O to the integrator. Source: https://github.com/NordSecurity/NepTUN and https://github.com/NordSecurity/NepTUN/releases. I am building and adding it to the Linux dataplane comparison before writing the final recommendation.

Candidate survey addendum: NepTUN is a maintained BSD-3-Clause Rust engine. GitHub Release v1.0.8 is dated 2025-07-21; upstream main was updated 2026-10-06 (source commit ce185155e1639403cc40e01260d0830065a49555). It has a Linux/macOS CLI and Windows library target, but the library leaves tunnel I/O to the integrator. Source: https://github.com/NordSecurity/NepTUN and https://github.com/NordSecurity/NepTUN/releases. I am building and adding it to the Linux dataplane comparison before writing the final recommendation.
Author
Owner

Packaging finding: a stripped Go c-archive host with the measured engine path linked into a stripped Rust executable is 3,870,864 B versus a 332,688 B Rust baseline (+3,538,176 B); the archive itself is 4,332,236 B. The reproducible bridge and Rust host are being added under research/mesh-engines/go-embed/; standalone Go CLI was 3,313,824 B.

Packaging finding: a stripped Go c-archive host with the measured engine path linked into a stripped Rust executable is 3,870,864 B versus a 332,688 B Rust baseline (+3,538,176 B); the archive itself is 4,332,236 B. The reproducible bridge and Rust host are being added under research/mesh-engines/go-embed/; standalone Go CLI was 3,313,824 B.
Author
Owner

License finding: GotaTun CLI enables its optional tun feature and therefore includes tun 0.8.6, whose Cargo.toml declares WTFPL (source https://github.com/meh/rust-tun/blob/master/Cargo.toml). WTFPL is outside the approved license set. The GotaTun core can be built without that feature; its own MPL-2.0 license is allowed, but Windows/macOS TUN integration then needs a different adapter. Its Windows adapter loads wintun.dll at runtime. Recording this as a hard integration constraint in REPORT.md.

License finding: GotaTun CLI enables its optional tun feature and therefore includes tun 0.8.6, whose Cargo.toml declares WTFPL (source https://github.com/meh/rust-tun/blob/master/Cargo.toml). WTFPL is outside the approved license set. The GotaTun core can be built without that feature; its own MPL-2.0 license is allowed, but Windows/macOS TUN integration then needs a different adapter. Its Windows adapter loads wintun.dll at runtime. Recording this as a hard integration constraint in REPORT.md.
Author
Owner

Candidate sweep: tailscale/wireguard-go main resolves to 469159ecf7d108dc1d44f8789dedbb463b7bd3a6 (last commit 2023-07-04; local checkout under /home/kayg/.cache/codex-scratch/mesh-engines/src/tailscale-wireguard-go), so it is not a maintained standalone candidate. Its security page reports no published advisories, but that does not change the stale-source finding. I am recording recent exploratory Go/Rust projects separately where target OS support or license evidence is incomplete.

Candidate sweep: tailscale/wireguard-go main resolves to 469159ecf7d108dc1d44f8789dedbb463b7bd3a6 (last commit 2023-07-04; local checkout under /home/kayg/.cache/codex-scratch/mesh-engines/src/tailscale-wireguard-go), so it is not a maintained standalone candidate. Its security page reports no published advisories, but that does not change the stale-source finding. I am recording recent exploratory Go/Rust projects separately where target OS support or license evidence is incomplete.
Author
Owner

Windows packaging finding: Wintun source labels the driver GPL-2.0-only, and the prebuilt wintun.dll has a separate custom license allowing redistribution only alongside software that uses its API (WireGuard/wintun prebuilt-binaries-license.txt). Neither is on the AGPL-compatible allowlist. Go/GotaTun can call Wintun, but this driver/package route is not license-cleared by the current rule; no Windows host was available to measure it. Evidence: https://github.com/WireGuard/wintun/blob/master/wintun.proj and https://github.com/WireGuard/wintun/blob/master/prebuilt-binaries-license.txt

Windows packaging finding: Wintun source labels the driver GPL-2.0-only, and the prebuilt wintun.dll has a separate custom license allowing redistribution only alongside software that uses its API (WireGuard/wintun prebuilt-binaries-license.txt). Neither is on the AGPL-compatible allowlist. Go/GotaTun can call Wintun, but this driver/package route is not license-cleared by the current rule; no Windows host was available to measure it. Evidence: https://github.com/WireGuard/wintun/blob/master/wintun.proj and https://github.com/WireGuard/wintun/blob/master/prebuilt-binaries-license.txt
Author
Owner

Security review finding: current NepTUN source still contains in at commit ce185155e1639403cc40e01260d0830065a49555. This leaves the same inner-packet-length traffic-analysis exposure documented in upstream BoringTun #494; GotaTun's 2026 audit says its padding fix shipped in 0.3.0, and current source has padding logic. NepTUN has no audit located; its open macOS clock issue is #13. https://github.com/cloudflare/boringtun/issues/494 https://github.com/NordSecurity/NepTUN/issues/13

Security review finding: current NepTUN source still contains in at commit ce185155e1639403cc40e01260d0830065a49555. This leaves the same inner-packet-length traffic-analysis exposure documented in upstream BoringTun #494; GotaTun's 2026 audit says its padding fix shipped in 0.3.0, and current source has padding logic. NepTUN has no audit located; its open macOS clock issue is #13. https://github.com/cloudflare/boringtun/issues/494 https://github.com/NordSecurity/NepTUN/issues/13
Author
Owner

Correction to prior comment: the shell expanded my quoted source text. The complete finding is that current NepTUN source at neptun/src/noise/session.rs (commit ce185155e1639403cc40e01260d0830065a49555) retains a TODO for 16-byte data padding, leaving the inner packet-length traffic-analysis exposure noted in BoringTun issue 494. GotaTun fixed padding in its 0.3.0 line; the current source contains padding logic. NepTUN has no audit located, and issue 13 is open about macOS clock behavior. https://github.com/cloudflare/boringtun/issues/494 https://github.com/NordSecurity/NepTUN/issues/13

Correction to prior comment: the shell expanded my quoted source text. The complete finding is that current NepTUN source at neptun/src/noise/session.rs (commit ce185155e1639403cc40e01260d0830065a49555) retains a TODO for 16-byte data padding, leaving the inner packet-length traffic-analysis exposure noted in BoringTun issue 494. GotaTun fixed padding in its 0.3.0 line; the current source contains padding logic. NepTUN has no audit located, and issue 13 is open about macOS clock behavior. https://github.com/cloudflare/boringtun/issues/494 https://github.com/NordSecurity/NepTUN/issues/13
Author
Owner

Harness commit 71d0b76882 adds the rootless data-plane, netlink, artifact-size, and Go c-archive runners. Shell syntax passed; both locked netlink release builds completed offline, including the nlink feature build.

Harness commit 71d0b76882797ec42b2ccb99f4530e3f4c99f2c9 adds the rootless data-plane, netlink, artifact-size, and Go c-archive runners. Shell syntax passed; both locked netlink release builds completed offline, including the nlink feature build.
Author
Owner

Raw-results commit f25c68d5dd retains five samples per data-plane and control candidate, environment capture, artifact bytes, and generated medians. Data-plane four-stream medians: kernel 2,658 Mbit/s; NepTUN 989; wireguard-go 1,541; BoringTun 586; GotaTun 514. Netlink applied 1,000 peers in 61,747 us (nl-wireguard) and 66,333 us (typed client); nlink full apply returned EINVAL. Files are under research/mesh-engines/results/.

Raw-results commit f25c68d5dd0fa2284ffd99958b1a1133645fc32b retains five samples per data-plane and control candidate, environment capture, artifact bytes, and generated medians. Data-plane four-stream medians: kernel 2,658 Mbit/s; NepTUN 989; wireguard-go 1,541; BoringTun 586; GotaTun 514. Netlink applied 1,000 peers in 61,747 us (nl-wireguard) and 66,333 us (typed client); nlink full apply returned EINVAL. Files are under research/mesh-engines/results/.
Author
Owner

Report commit e6bb136160 records source revisions, platform TUN coverage, audits/security reports, dependency licenses, build commands, and measured trade-offs. Recommendation recorded for owner review: kernel WireGuard on Linux; typed netlink bindings plus bounded chunks or nl-wireguard for control; no userspace candidate currently meets all platform, license, and single-Rust-binary constraints.

Report commit e6bb136160e52400aa452fe1482909cafe09d5b5 records source revisions, platform TUN coverage, audits/security reports, dependency licenses, build commands, and measured trade-offs. Recommendation recorded for owner review: kernel WireGuard on Linux; typed netlink bindings plus bounded chunks or nl-wireguard for control; no userspace candidate currently meets all platform, license, and single-Rust-binary constraints.
Author
Owner

Fast gate blocked: scripts/verify is absent from this worktree; scripts/ contains only perf-run. The required command returned: zsh:1: no such file or directory: scripts/verify (exit 127). Shell syntax, generated summary comparison, and both locked offline Rust release builds passed. Research scope does not include adding product gate code; the gate entrypoint must be restored to the worktree to run the prescribed gate.

Fast gate blocked: scripts/verify is absent from this worktree; scripts/ contains only perf-run. The required command returned: zsh:1: no such file or directory: scripts/verify (exit 127). Shell syntax, generated summary comparison, and both locked offline Rust release builds passed. Research scope does not include adding product gate code; the gate entrypoint must be restored to the worktree to run the prescribed gate.
Author
Owner

Finished on branch mesh-engines. Head: e6bb136160.

Path Throughput 1/4 Mbps Added ping p50/p99 ms CPU seconds/Gbit 1/4 RSS growth per peer at 100/1,000 Stripped CLI bytes
Kernel 2,280/2,658 0.121/0.244 2.442/1.709* Not isolatable --
BoringTun 595/586 0.115/0.486 1.543/1.954 19,743/18,760 1,306,712
GotaTun 598/514 0.181/0.353 1.895/2.338 20,398/20,017 2,261,256
NepTUN 956/989 0.236/4.154 2.063/2.368 21,012/20,652 1,483,408
wireguard-go 1,345/1,541 0.259/0.540 0.784/0.884 48,742/37,614 3,312,248; embedded host delta +3,537,824
Linux control client Apply 1,000 peers One-peer update Remove one peer Result
nl-wireguard 61,747 us 60 us 217 us Full correctness checks passed
Typed minimal 66,333 us 10 us 198 us Full checks passed; 400-peer chunks
nlink EINVAL at 2,535 us 58 us 5,651 us Full apply fails; update/remove tested after nl-wireguard baseline load

Recommendation: kernel WireGuard for Linux speed; typed minimal netlink or nl-wireguard for control. No measured userspace option clears the macOS, Windows, old Linux, AGPL-compatible dependency, and single Rust binary constraints. Wintun source is GPL-2.0-only and its prebuilt DLL has a separate custom license. Native macOS/Windows measurements remain unavailable. Kernel CPU counts wg-crypt workers only; userspace CPU counts both endpoints. Shared-host p99 is scheduler-sensitive.

Report: research/mesh-engines/REPORT.md. Raw samples and host evidence: research/mesh-engines/results/.

Required fast gate attempt: scripts/verify fast
Output (verbatim): zsh:1: no such file or directory: scripts/verify
Exit code: 127. The worktree contains scripts/perf-run but no scripts/verify. Bash syntax, summary regeneration, both locked offline netlink builds, and git diff --check passed.

Finished on branch mesh-engines. Head: e6bb136160e52400aa452fe1482909cafe09d5b5. | Path | Throughput 1/4 Mbps | Added ping p50/p99 ms | CPU seconds/Gbit 1/4 | RSS growth per peer at 100/1,000 | Stripped CLI bytes | | --- | ---: | ---: | ---: | ---: | ---: | | Kernel | 2,280/2,658 | 0.121/0.244 | 2.442/1.709* | Not isolatable | -- | | BoringTun | 595/586 | 0.115/0.486 | 1.543/1.954 | 19,743/18,760 | 1,306,712 | | GotaTun | 598/514 | 0.181/0.353 | 1.895/2.338 | 20,398/20,017 | 2,261,256 | | NepTUN | 956/989 | 0.236/4.154 | 2.063/2.368 | 21,012/20,652 | 1,483,408 | | wireguard-go | 1,345/1,541 | 0.259/0.540 | 0.784/0.884 | 48,742/37,614 | 3,312,248; embedded host delta +3,537,824 | | Linux control client | Apply 1,000 peers | One-peer update | Remove one peer | Result | | --- | ---: | ---: | ---: | --- | | nl-wireguard | 61,747 us | 60 us | 217 us | Full correctness checks passed | | Typed minimal | 66,333 us | 10 us | 198 us | Full checks passed; 400-peer chunks | | nlink | EINVAL at 2,535 us | 58 us | 5,651 us | Full apply fails; update/remove tested after nl-wireguard baseline load | Recommendation: kernel WireGuard for Linux speed; typed minimal netlink or nl-wireguard for control. No measured userspace option clears the macOS, Windows, old Linux, AGPL-compatible dependency, and single Rust binary constraints. Wintun source is GPL-2.0-only and its prebuilt DLL has a separate custom license. Native macOS/Windows measurements remain unavailable. Kernel CPU counts wg-crypt workers only; userspace CPU counts both endpoints. Shared-host p99 is scheduler-sensitive. Report: research/mesh-engines/REPORT.md. Raw samples and host evidence: research/mesh-engines/results/. Required fast gate attempt: scripts/verify fast Output (verbatim): zsh:1: no such file or directory: scripts/verify Exit code: 127. The worktree contains scripts/perf-run but no scripts/verify. Bash syntax, summary regeneration, both locked offline netlink builds, and git diff --check passed.
Author
Owner

Owner answers (2026-10-07):

  • M1: Linux kernel control uses our own small typed client (netlink-bindings + netlink-socket2, 400-peer chunks).
  • M3: Windows driver: deferred until Windows nodes.
  • M2: owner asked what Tailscale and NetBird use; follow-up pending.
Owner answers (2026-10-07): - M1: Linux kernel control uses **our own small typed client** (netlink-bindings + netlink-socket2, 400-peer chunks). - M3: Windows driver: **deferred** until Windows nodes. - M2: owner asked what Tailscale and NetBird use; follow-up pending.
Author
Owner

Owner decisions recorded as D169 (main): our own typed netlink client for Linux; GotaTun for macOS nodes with our own utun adapter; Windows driver deferred.

Owner decisions recorded as **D169** (main): our own typed netlink client for Linux; GotaTun for macOS nodes with our own utun adapter; Windows driver deferred.
Author
Owner

Closed by the orchestrator during the 2026-10-28 handover (#254), on the owner's request to close finished issues. Evidence: Engine measurement report research/mesh-engines/REPORT.md delivered and owner decisions recorded as D169 (a24ee0bd6 in dev). Reopen if anything is still missing.

Closed by the orchestrator during the 2026-10-28 handover (#254), on the owner's request to close finished issues. Evidence: Engine measurement report research/mesh-engines/REPORT.md delivered and owner decisions recorded as D169 (a24ee0bd6 in dev). Reopen if anything is still missing.
kayg closed this issue 2026-10-28 17:52:32 +00:00
Sign in to join this conversation.
No labels
launch-qa
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/infraternal#81
No description provided.