Research: how mesh SSH takes port 22 (userspace vs kernel redirect) and identifies users

kayg commented

2026-10-11 06:59:12 +00:00

Owner

Owner request (2026-10-11): research how existing mesh SSH products take over port 22, before we build SSH over our mesh (D197 M1, D227 step 3). Owner's recollection: Tailscale SSH does not go through the kernel (it intercepts in its own userspace network stack), while NetBird does and redirects 22 to 22022, which causes problems on his hosts.

Also in scope: how each identifies which person (not only which machine) is connecting, and what happens on a machine with several local users (e.g. a Mac with kayg and agents accounts), including SSH to another local user on the same machine.

Owner request (2026-10-11): research how existing mesh SSH products take over port 22, before we build SSH over our mesh (D197 M1, D227 step 3). Owner's recollection: Tailscale SSH does not go through the kernel (it intercepts in its own userspace network stack), while NetBird does and redirects 22 to 22022, which causes problems on his hosts. Also in scope: how each identifies *which person* (not only which machine) is connecting, and what happens on a machine with several local users (e.g. a Mac with `kayg` and `agents` accounts), including SSH to another local user on the same machine.

kayg commented

2026-10-11 07:04:16 +00:00

Author

Owner

Started research-mesh-ssh on branch research-mesh-ssh at base/head 4fc2460d92 (mesh slice 1), clean worktree, only writer. Read #162 (0 comments), its mesh parent #129 and parent #79 history, repository contract, requested decisions from origin/dev, and contracts/mesh-dataplane.md. Research and throwaway prototypes only; no product code or deployment.

Started research-mesh-ssh on branch research-mesh-ssh at base/head 4fc2460d92bd9d625eb6c34adb2b1bab2c1ffdaa (mesh slice 1), clean worktree, only writer. Read #162 (0 comments), its mesh parent #129 and parent #79 history, repository contract, requested decisions from origin/dev, and contracts/mesh-dataplane.md. Research and throwaway prototypes only; no product code or deployment.

kayg commented

2026-10-11 08:54:19 +00:00

Author

Owner

Finding: Tailscale SSH is a per-flow userspace interception. In pinned source 3bb8a906c27738c7904ef301c9539da763181f85, ShouldInterceptTCPPort only enables interception when netstack is built (ipn/ipnlocal/local.go:8379-8387); inbound packets from WireGuard to a local Tailscale IP are selected by wgengine/netstack/netstack.go:1253-1280 and consumed with DropSilently instead of continuing toward the host stack (:1440-1487). tailssh.HandleSSHConn is the in-process SSH entry (ssh/tailssh/tailssh.go:162-178). The docs say port 22 is claimed only for tailnet traffic and ordinary host SSH remains available for non-tailnet traffic: https://tailscale.com/docs/features/tailscale-ssh.

Finding: Tailscale SSH is a per-flow userspace interception. In pinned source 3bb8a906c27738c7904ef301c9539da763181f85, `ShouldInterceptTCPPort` only enables interception when netstack is built (`ipn/ipnlocal/local.go:8379-8387`); inbound packets from WireGuard to a local Tailscale IP are selected by `wgengine/netstack/netstack.go:1253-1280` and consumed with `DropSilently` instead of continuing toward the host stack (`:1440-1487`). `tailssh.HandleSSHConn` is the in-process SSH entry (`ssh/tailssh/tailssh.go:162-178`). The docs say port 22 is claimed only for tailnet traffic and ordinary host SSH remains available for non-tailnet traffic: https://tailscale.com/docs/features/tailscale-ssh.

kayg commented

2026-10-11 08:54:39 +00:00

Author

Owner

Finding: NetBird’s “port 22 to 22022” description is a kernel DNAT rule on Linux (iptables/nftables), with a packet-rewrite equivalent in its userspace firewall. In pinned source 67c2041fb5534c19d7b2320617cfc3aaab8e7d3d, client/internal/engine_ssh.go:31-52,274-310 installs 22 -> 22022 for the WireGuard address and starts the embedded listener at 22022; client/firewall/iptables/dnat_linux.go:16-45 and client/firewall/nftables/dnat_linux.go:17-93 match the WireGuard interface; client/firewall/uspfilter/filter.go:1218-1249 rewrites before its ACL path. Current docs date the native port model to v0.60.0+ and require v0.61.0+ for all SSH peers: https://docs.netbird.io/manage/peers/ssh. Reports show a v0.54.1 Docker PREROUTING/Input-ACL bypass (#4351) and a Kubernetes/CNI ordering report (#4470); they are user reports, not proof that every current setup is affected. IPv6 redirection/listener setup exists in the current code but is best-effort (warnings on failure); PR #5687 added it, merged 2026-04-07: https://github.com/netbirdio/netbird/pull/5687.

Finding: NetBird’s “port 22 to 22022” description is a kernel DNAT rule on Linux (iptables/nftables), with a packet-rewrite equivalent in its userspace firewall. In pinned source 67c2041fb5534c19d7b2320617cfc3aaab8e7d3d, `client/internal/engine_ssh.go:31-52,274-310` installs `22 -> 22022` for the WireGuard address and starts the embedded listener at 22022; `client/firewall/iptables/dnat_linux.go:16-45` and `client/firewall/nftables/dnat_linux.go:17-93` match the WireGuard interface; `client/firewall/uspfilter/filter.go:1218-1249` rewrites before its ACL path. Current docs date the native port model to v0.60.0+ and require v0.61.0+ for all SSH peers: https://docs.netbird.io/manage/peers/ssh. Reports show a v0.54.1 Docker PREROUTING/Input-ACL bypass (#4351) and a Kubernetes/CNI ordering report (#4470); they are user reports, not proof that every current setup is affected. IPv6 redirection/listener setup exists in the current code but is best-effort (warnings on failure); PR #5687 added it, merged 2026-04-07: https://github.com/netbirdio/netbird/pull/5687.

kayg commented

2026-10-11 09:00:17 +00:00

Author

Owner

Finding: Tailscale’s source SSH identity is the node’s tailnet user (or tag identity), not the local Unix account running the SSH client. Its docs explicitly warn that any local OS user on a client can use Tailscale SSH, so a multi-user machine can exercise the node owner’s policy; check/always adds SSO reauthentication but does not bind to the caller UID. Tagged nodes are service identities, and tag-based SSH is restricted to tagged destinations; shared-user cases have autogroup:shared policy semantics. Sources: https://tailscale.com/docs/features/tailscale-ssh and https://tailscale.com/docs/features/tags. Tailscale’s own issue #12810 is still a request to support SSH into the source node itself; the inbound WireGuard hook is the path handled in wgengine/netstack/netstack.go above: https://github.com/tailscale/tailscale/issues/12810.

NetBird’s current code separates mesh peer ACL identity from SSH OIDC user identity and target OS username. A valid JWT user ID plus requested local username are checked by client/ssh/server/server.go:678-725 and client/ssh/auth/auth.go:102-149; no mapping fails closed. In addition, merged PR #7378 binds cached SSH JWTs to the local Unix UID/Windows SID that obtained them (merge commit 7b22d55bf685427e7d8a313dc79358326af10497); the current cache and SO_PEERCRED implementation are in client/server/jwt_cache.go:42-103 and client/internal/ipcauth/peercred_linux.go:12-38: https://github.com/netbirdio/netbird/pull/7378. With default cache TTL 0, each new session does OIDC; enabled cache is caller-scoped. Thus a different local account cannot reuse a cached JWT through the daemon, though it can authenticate as its own IdP user. This differs from a design that authorizes solely by the shared mesh peer.

Finding: Tailscale’s source SSH identity is the node’s tailnet user (or tag identity), not the local Unix account running the SSH client. Its docs explicitly warn that any local OS user on a client can use Tailscale SSH, so a multi-user machine can exercise the node owner’s policy; `check`/`always` adds SSO reauthentication but does not bind to the caller UID. Tagged nodes are service identities, and tag-based SSH is restricted to tagged destinations; shared-user cases have `autogroup:shared` policy semantics. Sources: https://tailscale.com/docs/features/tailscale-ssh and https://tailscale.com/docs/features/tags. Tailscale’s own issue #12810 is still a request to support SSH into the source node itself; the inbound WireGuard hook is the path handled in `wgengine/netstack/netstack.go` above: https://github.com/tailscale/tailscale/issues/12810. NetBird’s current code separates mesh peer ACL identity from SSH OIDC user identity and target OS username. A valid JWT user ID plus requested local username are checked by `client/ssh/server/server.go:678-725` and `client/ssh/auth/auth.go:102-149`; no mapping fails closed. In addition, merged PR #7378 binds cached SSH JWTs to the local Unix UID/Windows SID that obtained them (merge commit 7b22d55bf685427e7d8a313dc79358326af10497); the current cache and `SO_PEERCRED` implementation are in `client/server/jwt_cache.go:42-103` and `client/internal/ipcauth/peercred_linux.go:12-38`: https://github.com/netbirdio/netbird/pull/7378. With default cache TTL 0, each new session does OIDC; enabled cache is caller-scoped. Thus a different local account cannot reuse a cached JWT through the daemon, though it can authenticate as its own IdP user. This differs from a design that authorizes solely by the shared mesh peer.

kayg commented

2026-10-11 09:03:35 +00:00

Author

Owner

Correction on Tailscale SSH-to-self: #12810 was closed as a duplicate of #4475, not as a fix. #4475 remains open/needs-fix and documents that connecting to the node’s own Tailscale IP bypasses the TUN/WireGuard hook and reaches host OpenSSH instead: https://github.com/tailscale/tailscale/issues/4475. NetBird has a corresponding explicit own-peer rejection in client/ssh/server/server.go:835-850 at source commit 67c2041fb5534c19d7b2320617cfc3aaab8e7d3d. I’ll distinguish both self cases from remote mesh SSH in the report.

Correction on Tailscale SSH-to-self: #12810 was closed as a duplicate of #4475, not as a fix. #4475 remains open/needs-fix and documents that connecting to the node’s own Tailscale IP bypasses the TUN/WireGuard hook and reaches host OpenSSH instead: https://github.com/tailscale/tailscale/issues/4475. NetBird has a corresponding explicit own-peer rejection in `client/ssh/server/server.go:835-850` at source commit 67c2041fb5534c19d7b2320617cfc3aaab8e7d3d. I’ll distinguish both self cases from remote mesh SSH in the report.

kayg referenced this issue from a commit

2026-10-11 09:13:10 +00:00

Add rootless mesh SSH path benchmark

kayg commented

2026-10-11 09:13:36 +00:00

Author

Owner

Completed benchmark step; harness committed and pushed as 1a7ffd85. On infraternal-dev (8 vCPU, Linux 6.12.111, rootless user/net namespaces, CPUWeight=10000), two full runs used 100 setup samples and five 32 MiB transfers per path. Run 1 p50/p95 setup RTT and median goodput (MiB/s): direct 0.591/2.522, 1574.97; DNAT 0.514/1.443, 2041.93; TPROXY 0.409/0.923, 1673.01; Unix proxy 1.149/4.194, 1059.57. Run 2: direct 0.378/2.792, 986.31; DNAT 0.528/6.016, 727.26; TPROXY 0.371/7.087, 950.24; Unix proxy 2.683/11.688, 535.83. Load average was 15.18→15.18 in run 1 and 15.76→17.06 in run 2, so treat these as noisy TCP transport comparisons, not a ranking. A 0.0.0.0:22 listener made an additional mesh-IP:22 bind fail with EADDRINUSE. The UDS probe saw UID 0 in the user namespace, mapped to host UID 1001. No SSH protocol, WireGuard encryption, or gVisor was measured. Raw results are under $HOME/.cache/codex-scratch/research-mesh-ssh/.

Completed benchmark step; harness committed and pushed as 1a7ffd85. On infraternal-dev (8 vCPU, Linux 6.12.111, rootless user/net namespaces, CPUWeight=10000), two full runs used 100 setup samples and five 32 MiB transfers per path. Run 1 p50/p95 setup RTT and median goodput (MiB/s): direct 0.591/2.522, 1574.97; DNAT 0.514/1.443, 2041.93; TPROXY 0.409/0.923, 1673.01; Unix proxy 1.149/4.194, 1059.57. Run 2: direct 0.378/2.792, 986.31; DNAT 0.528/6.016, 727.26; TPROXY 0.371/7.087, 950.24; Unix proxy 2.683/11.688, 535.83. Load average was 15.18→15.18 in run 1 and 15.76→17.06 in run 2, so treat these as noisy TCP transport comparisons, not a ranking. A 0.0.0.0:22 listener made an additional mesh-IP:22 bind fail with EADDRINUSE. The UDS probe saw UID 0 in the user namespace, mapped to host UID 1001. No SSH protocol, WireGuard encryption, or gVisor was measured. Raw results are under $HOME/.cache/codex-scratch/research-mesh-ssh/.

kayg referenced this issue from a commit

2026-10-11 09:26:45 +00:00

Calibrate Unix proxy throughput timing

kayg commented

2026-10-11 09:27:49 +00:00

Author

Owner

Correction after reviewing the harness: the first Unix-proxy goodput timer began before the proxy's remote TCP connect finished. Fixed the timing in pushed commit da5dd08c and reran two full series; all earlier throughput values in my previous comment are superseded by these calibrated results. Each run used 100 setup samples and five 32 MiB transfers per path. p50/p95 setup RTT ms and median MiB/s: run 1 — direct 0.244/0.581, 1974.39; DNAT 0.343/0.532, 1420.92; TPROXY 0.391/1.276, 1457.14; Unix proxy 1.335/4.577, 1313.71. Run 2 — direct 0.398/1.304, 1799.57; DNAT 0.460/3.650, 982.06; TPROXY 0.515/3.701, 893.49; Unix proxy 2.323/10.949, 615.55. Load averages were 19.80→19.80 and 19.65→21.76 on 8 vCPUs. Run-to-run variance remains material; this is TCP transport only, without SSH, WireGuard, or gVisor. Calibrated summary and current raw run are in $HOME/.cache/codex-scratch/research-mesh-ssh/.

Correction after reviewing the harness: the first Unix-proxy goodput timer began before the proxy's remote TCP connect finished. Fixed the timing in pushed commit da5dd08c and reran two full series; all earlier throughput values in my previous comment are superseded by these calibrated results. Each run used 100 setup samples and five 32 MiB transfers per path. p50/p95 setup RTT ms and median MiB/s: run 1 — direct 0.244/0.581, 1974.39; DNAT 0.343/0.532, 1420.92; TPROXY 0.391/1.276, 1457.14; Unix proxy 1.335/4.577, 1313.71. Run 2 — direct 0.398/1.304, 1799.57; DNAT 0.460/3.650, 982.06; TPROXY 0.515/3.701, 893.49; Unix proxy 2.323/10.949, 615.55. Load averages were 19.80→19.80 and 19.65→21.76 on 8 vCPUs. Run-to-run variance remains material; this is TCP transport only, without SSH, WireGuard, or gVisor. Calibrated summary and current raw run are in $HOME/.cache/codex-scratch/research-mesh-ssh/.

kayg commented

2026-10-11 10:37:41 +00:00

Author

Owner

Fast-gate finding (2026-10-11): the full rerun with the scratch Cargo target and a temporary worktree target symlink failed the Rust test and static-build rows. All six agent_review probes returned “terminal open refused” inside the high-priority user scope. The x86_64 static binary was 20,614,624 bytes against the 16,404,816-byte maximum. A direct run stopped at the latency self-test (p50 318.44 ms for configured 300 ms). This research job made no runtime-code changes; I’m recording the gate output and continuing with the requested research artifact.

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| clippy                    | PASS         |
| test                      | FAIL (101)   |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Fast-gate finding (2026-10-11): the full rerun with the scratch Cargo target and a temporary worktree `target` symlink failed the Rust test and static-build rows. All six `agent_review` probes returned “terminal open refused” inside the high-priority user scope. The x86_64 static binary was 20,614,624 bytes against the 16,404,816-byte maximum. A direct run stopped at the latency self-test (p50 318.44 ms for configured 300 ms). This research job made no runtime-code changes; I’m recording the gate output and continuing with the requested research artifact. ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | clippy | PASS | | test | FAIL (101) | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ```

kayg referenced this issue from a commit

2026-10-11 10:39:26 +00:00

Document mesh SSH port and identity research

kayg commented

2026-10-11 10:42:24 +00:00

Author

Owner

Finished on branch research-mesh-ssh; pushed head SHA: e5854958ab47b397c86a8ca47ddcbff8377026c0. Research note: research/mesh-ssh/README.md.

Short answer: Tailscale serves tailnet-IP TCP 22 inside tailscaled, diverting selected traffic into its gVisor netstack; NetBird redirects matching mesh-interface TCP 22 to its embedded listener on 22022 with nftables/iptables DNAT. Tailscale policy identifies the source node owner or tags, so local accounts share node rights. NetBird adds an OIDC user JWT and scopes cached tokens to the calling local UID/SID, while the target username selects the local account. The note covers the exceptions, versioned issue evidence, and owner decisions without choosing an infraternal transport or identity model.

The corrected transport measurements remain load-sensitive and are transport-only. The high-priority full gate run failed the Rust test and static-build rows; six agent-review probes returned “terminal open refused”, and the x86_64 binary measured 20,614,624 bytes against a 16,404,816-byte maximum. The direct run stopped at the latency self-test (p50 318.44 ms for configured 300 ms).

scripts/verify fast Rust gate output, verbatim:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| clippy                    | PASS         |
| test                      | FAIL (101)   |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Finished on branch `research-mesh-ssh`; pushed head SHA: `e5854958ab47b397c86a8ca47ddcbff8377026c0`. Research note: [research/mesh-ssh/README.md](https://git.kayg.org/kayg/infraternal/src/branch/research-mesh-ssh/research/mesh-ssh/README.md). Short answer: Tailscale serves tailnet-IP TCP 22 inside `tailscaled`, diverting selected traffic into its gVisor netstack; NetBird redirects matching mesh-interface TCP 22 to its embedded listener on 22022 with nftables/iptables DNAT. Tailscale policy identifies the source node owner or tags, so local accounts share node rights. NetBird adds an OIDC user JWT and scopes cached tokens to the calling local UID/SID, while the target username selects the local account. The note covers the exceptions, versioned issue evidence, and owner decisions without choosing an infraternal transport or identity model. The corrected transport measurements remain load-sensitive and are transport-only. The high-priority full gate run failed the Rust test and static-build rows; six agent-review probes returned “terminal open refused”, and the x86_64 binary measured 20,614,624 bytes against a 16,404,816-byte maximum. The direct run stopped at the latency self-test (p50 318.44 ms for configured 300 ms). `scripts/verify fast` Rust gate output, verbatim: ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | clippy | PASS | | test | FAIL (101) | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ```

kayg referenced this issue from a commit

2026-10-11 14:03:02 +00:00

decisions: D235 SSH over the mesh (identity, linking, access, audit); D236 session row labels

kayg commented

2026-10-11 14:03:07 +00:00

Author

Owner

Orchestrator: owner grill concluded 2026-10-11; recorded as D235 (dev 3fb20239). Build is queued after D215 and the TCP/443 fallback (D227 order).

Orchestrator: owner grill concluded 2026-10-11; recorded as D235 (dev 3fb20239). Build is queued after D215 and the TCP/443 fallback (D227 order).

kayg referenced this issue

2026-10-12 15:02:30 +00:00

Research: typing feels local anywhere #192

kayg referenced this issue

2026-10-12 15:02:45 +00:00

Research: typing feels local anywhere #192

kayg referenced this issue

2026-10-15 08:17:21 +00:00

Mesh: SSH over the mesh (D197 M1, D235) #221

kayg referenced this issue

2026-10-15 08:19:02 +00:00

Mesh: SSH over the mesh (D197 M1, D235) #221

kayg referenced this issue

2026-10-15 08:37:13 +00:00

Mesh: SSH over the mesh (D197 M1, D235) #221

kayg referenced this issue

2026-10-16 01:48:31 +00:00

Mesh: SSH over the mesh (D197 M1, D235) #221

kayg referenced this issue

2026-10-16 15:13:01 +00:00

Mesh: SSH over the mesh (D197 M1, D235) #221

kayg referenced this issue

2026-10-16 20:08:29 +00:00

Mesh: SSH over the mesh (D197 M1, D235) #221

kayg referenced this issue

2026-10-16 21:39:29 +00:00

Mesh: SSH over the mesh (D197 M1, D235) #221

kayg commented

2026-10-28 17:52:55 +00:00

Author

Owner

Closed by the orchestrator during the 2026-10-28 handover (#254), on the owner's request to close finished issues. Evidence: Research note delivered (research/mesh-ssh) and owner grill outcome recorded as D235. Reopen if anything is still missing.

kayg closed this issue

2026-10-28 17:52:55 +00:00

~~kayg referenced this issue 2026-10-28 17:53:55 +00:00~~

Orchestrator handover 2026-10-28: moving to a new machine #254

kayg referenced this issue

2026-10-28 18:00:45 +00:00

Orchestrator handover 2026-10-28: moving to a new machine #254

Rows
Columns

Research: how mesh SSH takes port 22 (userspace vs kernel redirect) and identifies users #162