Step 19 (S2): remote infraternal agents, enrollment and tunnel #40
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
PLAN step 19: a single-use join token enrolls an infraternal agent with its own certificate identity. The agent reconnects over mutual TLS on 443 through one multiplexed WebSocket, honors signed grants, and supports revocation.
--no-local-agentdisables the embedded infraternal agent. The same job owns both the server tunnel hub andcrates/agent(the paired writer the plan requires). This is a security surface, so it runs on Sol.Starting step 19 on branch
agent-tunnelfrom basedbd66850f8a5d6190e65b4bd364e075950033425. Read repository contracts and inspected the existing embedded infraternal agent grant/framing path. I will append and commit the remote enrollment/tunnel contract before product code, then implement enroll → connect → list → exec → revoke withscripts/verify fastbetween slices. No push.Contract committed at
750cebc31d726259843f7fd8b272edb26bb49ca2indocs/contracts.md§6. It fixes token TTL and atomic single use, CSR/Ed25519 identity and CA storage, 30-day certificate lifetime with rotation, direct mTLS listener and Traefik TCP passthrough follow-up, durable nonce replay protection, 15-second revocation bound, and reuse of existing signed OPEN/credit framing. Existing scope registry already hasagents:enrollandagents:revoke; no new scopes are needed. Version references: https://docs.rs/crate/rustls/0.23.45, https://docs.rs/crate/tokio-rustls/0.26.5, https://docs.rs/crate/rcgen/0.14.10.Enrollment storage finding:
0017_remote_agents.sqlapplies to all three legacy migration fixtures, so I updated the migration assertions to expect version 17;cargo test -p infraternal-server --test migrations -qpassed (5 passed). The firstscripts/verify fastfailure was those outdated assertions. The next fast run reachedcrates/engine::logs::tests::steady_state_framing_does_not_allocate_per_chunkand failed under load withStats { allocations: 117, deallocations: 125, reallocations: 31, bytes_allocated: 120970, bytes_deallocated: 14584, bytes_reallocated: 8161 }. One direct rerun,cargo test -p infraternal-engine --lib logs::tests::steady_state_framing_does_not_allocate_per_chunk -- --exact, passed:test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 15 filtered out; finished in 0.00s. This is a load-sensitive preexisting test; no engine files changed.Enrollment vertical slice committed at
36b822493fa7abecb372f20e4ef8d7cc9e668a5b. The real-router test creates a scoped join token, enrolls a client-generated Ed25519 CSR once, rejects reuse, and revokes the infraternal agent identity while the human token remains valid. The CA test verifies a CSR requesting CA privileges receives only a clientAuth leaf and the CA persists across reopen. OpenAPI and TypeScript client were regenerated. Gate output verbatim:Remote mTLS connect, multiplexed routing, process CLI, certificate rotation, and revocation of live tunnels remain in progress.
Connection slice committed at
86ee057725. Direct mTLS listener validates the CA, URI SAN, serial, and revocation before upgrade; HELLO records a durable 24-hour nonce, a new connection fences the old one, idle revocation checks run each second. The remote infraternal agent now enrolls with a locally generated key, stores 0600 identity files, connects over WSS, and reconnects with a fresh nonce. A real TLS test covers wrong CA, replayed nonce, fencing, and revocation. A compiled server plus infraternal agent process test covers enrollment and restart with a fresh nonce. Channel routing/list/exec remain the next slice.scripts/verify fast output (verbatim):
WARN load5=12.26 exceeds online CPU count=8; timing-sensitive checks may be delayed
PASS fast: migrations are append-only
PASS fast: workspace package dependencies
PASS fast: JavaScript dependencies
PASS fast: build web app
PASS fast: cargo fmt --check
PASS fast: cargo clippy
PASS fast: cargo test
PASS fast: svelte-check
PASS fast: UI package svelte-check
PASS fast: stream package typecheck
PASS fast: Vitest
PASS fast: UI package Vitest
PASS fast: stream package Vitest
PASS fast: OpenAPI and TypeScript client drift
PASS fast: terminal TypeScript
PASS fast: terminal Vitest
Contract follow-up committed at
ebfb554f22. §6.4 defines stable host id mapping from the enrolled infraternal agent id, bounded inventory item/end frames on channel zero, atomic snapshot publication and invalidation, and unique even channel ids on each multiplexed connection. This precedes implementation of remote list/exec routing.Inventory/list/exec slice committed at
fdedea2e3e. Enrollment now creates a stable remote host id; an infraternal agent sends bounded inventory generations over channel zero; the server publishes them atomically and routes signed stream OPENs through even, bounded multiplexed channels.--no-local-agentskips local engine discovery and local stream dispatch. Invalid join-token consumption is audited, and the CA public file is set to 0644. The compiled process test ran with--no-local-agent, used a real Docker busybox container selected by a unique run name, listed it through the remote host API, and executed a shell command through the tunnel. Targeted output:running 1 test
test real_infraternal_agent_process_enrolls_and_reconnects_with_new_nonce ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.90s
scripts/verify fast output (verbatim):
WARN load5=10.71 exceeds online CPU count=8; timing-sensitive checks may be delayed
PASS fast: migrations are append-only
PASS fast: workspace package dependencies
PASS fast: JavaScript dependencies
PASS fast: build web app
PASS fast: cargo fmt --check
PASS fast: cargo clippy
PASS fast: cargo test
PASS fast: svelte-check
PASS fast: UI package svelte-check
PASS fast: stream package typecheck
PASS fast: Vitest
PASS fast: UI package Vitest
PASS fast: stream package Vitest
PASS fast: OpenAPI and TypeScript client drift
PASS fast: terminal TypeScript
PASS fast: terminal Vitest
Certificate rotation contract refined and committed at
a31e628fa2before implementation. The direct mTLS listener accepts{csr_pem}, returns{certificate_pem, expires_at_ms}, and invalidates the old serial immediately after the transaction commits. This matches migration 0017's single active serial and requires an immediate reconnect with the new key.Rotation/revocation slice committed at
94a47a00df. The mTLS listener now accepts a fresh CSR, atomically replaces the active certificate serial and audits rotation. The infraternal agent persists a new owner-only key before requesting a certificate, atomically switches its identity document after receipt, and renews in the last seven days. The tunnel hub immediately fences a revoked or superseded connection through an in-memory flag and close signal; the durable certificate/revocation check runs each second rather than querying SQLite for every data frame. The compiled process test rotated between two connections, rejected use of the old certificate, and confirmed user-session API access survived agent revocation. Its optional Docker path also listed and executed through the renewed tunnel. Targeted output:running 1 test
test real_infraternal_agent_process_enrolls_and_reconnects_with_new_nonce ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.70s
scripts/verify fast output (verbatim):
PASS fast: migrations are append-only
PASS fast: workspace package dependencies
PASS fast: JavaScript dependencies
PASS fast: build web app
PASS fast: cargo fmt --check
PASS fast: cargo clippy
PASS fast: cargo test
PASS fast: svelte-check
PASS fast: UI package svelte-check
PASS fast: stream package typecheck
PASS fast: Vitest
PASS fast: UI package Vitest
PASS fast: stream package Vitest
PASS fast: OpenAPI and TypeScript client drift
PASS fast: terminal TypeScript
PASS fast: terminal Vitest
Connection lifecycle slice committed at
3e3bd3cc49. A silent TCP client no longer holds the direct mTLS accept loop: up to 128 handshakes run concurrently with ten-second deadlines, proven by a real TLS test keeping one TCP connection silent while another upgrades. Valid PONG is accepted, and an optional 17-second compiled-process run observed only one durable nonce. Connection drop now aborts its writer, inventory loop and bounded channel relay tasks. Identity loads check private file mode and reject symlinks before reading.Targeted liveness/Docker output:
Finished
testprofile [unoptimized] target(s) in 21.27sRunning tests/agent_tunnel.rs (target/debug/deps/agent_tunnel-0ececc8331ed5a80)
running 1 test
test real_infraternal_agent_process_enrolls_and_reconnects_with_new_nonce ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 23.45s
scripts/verify fast output (verbatim):
WARN load5=8.61 exceeds online CPU count=8; timing-sensitive checks may be delayed
PASS fast: migrations are append-only
PASS fast: workspace package dependencies
PASS fast: JavaScript dependencies
PASS fast: build web app
PASS fast: cargo fmt --check
PASS fast: cargo clippy
PASS fast: cargo test
PASS fast: svelte-check
PASS fast: UI package svelte-check
PASS fast: stream package typecheck
PASS fast: Vitest
PASS fast: UI package Vitest
PASS fast: stream package Vitest
PASS fast: OpenAPI and TypeScript client drift
PASS fast: terminal TypeScript
PASS fast: terminal Vitest
Integration gate finding: the existing infraternal-engine test
pty_session_drop_and_close_terminate_exec_processes_on_all_real_enginesfailed once withexec_termination_unconfirmedafter its Podman marker scan could not capture a host PID namespace. This did not exercise the new remote mTLS tunnel. The required one direct rerun passed. Full original gate and direct rerun output are quoted verbatim below.Original scripts/verify integration output:
PASS integration: JavaScript dependencies
PASS integration: build web app
PASS integration: build release binary
PASS integration: real HTTP behavior
PASS integration: Docker socket _ping
PASS integration: rootless Podman socket _ping
FAIL integration: engine adapters against Docker and Podman
Compiling infraternal-proto v0.0.1 (/home/kayg/wt/agent-tunnel/crates/proto)
Compiling infraternal-engine v0.0.1 (/home/kayg/wt/agent-tunnel/crates/engine)
Finished
testprofile [unoptimized] target(s) in 8.63sRunning tests/integration.rs (target/debug/deps/integration-f92617287b4ea6a8)
running 11 tests
test docker_and_podman_share_the_real_engine_contract ... infraternal-engine-foreign-docker-3935934-1790320051169-0
infraternal-engine-foreign-docker-3935934-1790320051169-1
infraternal-engine-foreign-docker-3935934-1790320051169-2
time="2026-09-25T09:08:19+02:00" level=warning msg="StopSignal SIGTERM failed to stop container infraternal-engine-foreign-podman-3935934-1790320051169-0 in 10 seconds, resorting to SIGKILL"
infraternal-engine-foreign-podman-3935934-1790320051169-0
time="2026-09-25T09:08:31+02:00" level=warning msg="StopSignal SIGTERM failed to stop container infraternal-engine-foreign-podman-3935934-1790320051169-1 in 10 seconds, resorting to SIGKILL"
infraternal-engine-foreign-podman-3935934-1790320051169-1
time="2026-09-25T09:08:42+02:00" level=warning msg="StopSignal SIGTERM failed to stop container infraternal-engine-foreign-podman-3935934-1790320051169-2 in 10 seconds, resorting to SIGKILL"
infraternal-engine-foreign-podman-3935934-1790320051169-2
ok
test list_preserves_health_and_pod_identity_on_all_real_engines ... ok
test missing_container_scan_tools_report_unconfirmed_when_host_proc_is_restricted ... ok
test podman_completed_containers_are_safe_for_inventory_and_mutations ... ok
test podman_model_preserves_health_and_pod_identity ... ok
test podman_mutation_response_loss_is_unknown_outcome ... ok
test pty_close_reaps_setsid_and_double_fork_descendants ... ok
test pty_close_reaps_term_ignoring_group_child_after_leader_exits ... docker same-group child reaped in 590.613781ms
podman same-group child reaped in 1.390640095s
ok
test pty_session_drop_and_close_terminate_exec_processes_on_all_real_engines ...
thread 'pty_session_drop_and_close_terminate_exec_processes_on_all_real_engines' (3951358) panicked at crates/engine/tests/integration.rs:743:27:
pty close: EngineError { code: Conflict, message: "marker scan failed and no host PID namespace was captured", kind: "exec", engine_code: Some("exec_termination_unconfirmed"), retryable: false, outcome_unknown: true }
note: run with
RUST_BACKTRACE=1environment variable to display a backtraceFAILED
test rootful_podman_covers_inventory_and_mutations ... ok
test stalled_pty_stdin_cannot_hold_exec_termination ... docker stalled stdin + ignored SIGTERM reaped in 467.837974ms
podman stalled stdin + ignored SIGTERM reaped in 1.267999288s
ok
failures:
failures:
pty_session_drop_and_close_terminate_exec_processes_on_all_real_engines
test result: FAILED. 10 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 221.36s
error: test failed, to rerun pass
-p infraternal-engine --test integrationDirect single-test rerun output:
Finished
testprofile [unoptimized] target(s) in 0.21sRunning tests/integration.rs (target/debug/deps/integration-f92617287b4ea6a8)
running 1 test
test pty_session_drop_and_close_terminate_exec_processes_on_all_real_engines ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 10 filtered out; finished in 34.12s
Second integration run finding: the existing stream regression
authorization_loss_terminates_stalled_stdin_on_both_enginesfailed once because Podman reported termination completion asunconfirmed/engine_errorinstead of the expected completed audit. The direct single-test rerun passed. The full gate reached this point after the engine, bounded logs and TypeScript stream suites passed. Both outputs are quoted verbatim below.Original scripts/verify integration output:
WARN load5=8.08 exceeds online CPU count=8; timing-sensitive checks may be delayed
PASS integration: JavaScript dependencies
PASS integration: build web app
PASS integration: build release binary
PASS integration: real HTTP behavior
PASS integration: Docker socket _ping
PASS integration: rootless Podman socket _ping
PASS integration: engine adapters against Docker and Podman
PASS integration: bounded follow:false log snapshots against Docker and rootless Podman
PASS integration: TypeScript stream client against Docker and Podman
FAIL integration: stream authorization and CLOSE regressions against Docker
Compiling infraternal-server v0.0.1 (/home/kayg/wt/agent-tunnel/crates/server)
Finished
testprofile [unoptimized] target(s) in 4.77sRunning tests/stream_regressions.rs (target/debug/deps/stream_regressions-3b9402f0572a6a1c)
running 12 tests
test authorization_loss_terminates_detached_persistent_exec_process ... ok
test authorization_loss_terminates_persistent_exec_process ... ok
test authorization_loss_terminates_stalled_stdin_on_both_engines ... docker stalled stdin CLOSE forbidden in 47.199093ms
podman stalled stdin CLOSE forbidden in 182.04µs
thread 'authorization_loss_terminates_stalled_stdin_on_both_engines' (4011127) panicked at crates/server/tests/stream_regressions.rs:836:9:
podman termination completion was not audited: [("started", Some("forbidden")), ("unconfirmed", Some("engine_error"))]
note: run with
RUST_BACKTRACE=1environment variable to display a backtraceFAILED
test client_data_written_after_grant_commit_never_reaches_exec ... ok
test events_channel_replies_to_client_close ... ok
test every_grant_shrink_closes_logs_and_persistent_exec_on_both_engines ... docker role_change both CLOSE forbidden in 18.343868ms
docker role_change owner process reaped in 553.389127ms
docker role_grant_removed both CLOSE forbidden in 39.104232ms
docker role_grant_removed owner process reaped in 538.277252ms
docker role_grant_removed_then_restored both CLOSE forbidden in 48.369314ms
docker transient shrink had an active log stream before mutation and delivered 2 DATA frames after commit before CLOSE
docker role_grant_removed_then_restored owner process reaped in 429.39828ms
docker token_grant_removed both CLOSE forbidden in 87.638792ms
docker token_grant_removed owner process reaped in 411.706529ms
docker tag_change both CLOSE forbidden in 9.068686ms
docker tag_change owner process reaped in 546.481133ms
docker user_disabled both CLOSE forbidden in 12.731671ms
docker user_disabled owner process reaped in 687.547152ms
docker token_revoked both CLOSE forbidden in 4.67689ms
docker token_revoked owner process reaped in 467.306738ms
docker token_expired both CLOSE forbidden in 9.324045ms
docker token_expired owner process reaped in 421.694632ms
podman role_change both CLOSE forbidden in 83.512789ms
podman role_change owner process reaped in 1.200820477s
podman role_grant_removed both CLOSE forbidden in 83.539218ms
podman role_grant_removed owner process reaped in 434.21709ms
podman role_grant_removed_then_restored both CLOSE forbidden in 45.253738ms
podman transient shrink had an active log stream before mutation and delivered 3 DATA frames after commit before CLOSE
podman role_grant_removed_then_restored owner process reaped in 1.091246185s
podman token_grant_removed both CLOSE forbidden in 71.058384ms
podman token_grant_removed owner process reaped in 1.132664902s
podman tag_change both CLOSE forbidden in 128.833162ms
podman tag_change owner process reaped in 2.429496473s
podman user_disabled both CLOSE forbidden in 65.392092ms
podman user_disabled owner process reaped in 660.208547ms
podman token_revoked both CLOSE forbidden in 29.391213ms
podman token_revoked owner process reaped in 360.991279ms
podman token_expired both CLOSE forbidden in 44.484846ms
podman token_expired owner process reaped in 1.159917596s
ok
test failed_dispatch_commit_never_leaves_a_persistent_exec ... ok
test failed_final_open_audit_terminates_new_persistent_exec ... ok
test logs_channel_replies_to_client_close ... ok
test logs_channel_replies_to_close_while_waiting_for_credit ... ok
test removing_a_host_tag_closes_an_existing_channel_with_forbidden ... ok
test session_http_lists_only_the_owner_and_closes_real_docker_and_podman_ptys ... ok
failures:
failures:
authorization_loss_terminates_stalled_stdin_on_both_engines
test result: FAILED. 11 passed; 1 failed; 0 ignored; 0 measured; 7 filtered out; finished in 134.92s
error: test failed, to rerun pass
-p infraternal-server --test stream_regressionsDirect single-test rerun output:
Finished
testprofile [unoptimized] target(s) in 0.55sRunning tests/stream_regressions.rs (target/debug/deps/stream_regressions-3b9402f0572a6a1c)
running 1 test
test authorization_loss_terminates_stalled_stdin_on_both_engines ... docker stalled stdin CLOSE forbidden in 25.550595ms
podman stalled stdin CLOSE forbidden in 41.205337ms
ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 18 filtered out; finished in 15.90s
Final implementation slices committed:
1433cb0adds an opt-in real Docker/compiled-process remote tunnel benchmark (CLI exec, log throughput, concurrent paced echo/log/RSS);07dcf75avoids local SSH identity creation under--no-local-agent, retries transient renewal HTTP failures, and prefilters invalid join tokens before CSR signing while retaining atomic consume. Benchmark execution is deferred to the dedicated perf-test VM required by PLAN.md; this dev VM only compiled the harness.Targeted compiled-process check:
scripts/verify fastoutput after these changes:Third full
scripts/verify integrationattempt started with load5=7.30. It passed release build, HTTP, Docker/rootless Podman probes, real engine adapters, bounded log snapshots and TypeScript stream client. It stopped in a pre-existing stream regression under later VM load (load5=13.04 at failure), before the new remote tunnel gate step. The targeted remote compiled-process test has passed separately. Exact gate result:Failure excerpt from the gate, verbatim:
I will rerun that single test directly once when VM load falls; no source changes are planned for this unrelated timing failure.
Required one direct rerun of the failing existing regression was started at load5=9.46. It failed again in a different Podman subcase (
tag_change, rather than the full gate'stoken_expired):The full integration gate remains red; no further timing reruns will be used as a substitute for a green gate. This test is outside step 19 ownership and does not exercise the remote mTLS path. The step 19 compiled-process test passed separately (enroll, fresh nonce reconnect, rotation, Docker list/exec, revoke, 17s liveness).
Step 19 branch handoff. Head
07dcf7547a12d6b8033eabfcd77334cb81773799; basedbd66850f8a5d6190e65b4bd364e075950033425; worktree clean; no push or issue closure.Delivered: contract-first enrollment with single-use hashed expiring join tokens and audit; private CA and issued/rotated certificate identity; direct mTLS WebSocket listener with nonce replay protection, active-session fencing, bounded multiplexed signed-grant list/exec and revocation; real remote infraternal agent CLI/process;
--no-local-agent; generated OpenAPI/client; compiled-process Docker integration test; remote stream benchmark harness. Browser gate was not run because no web source was changed. TCP passthrough deployment configuration remains follow-up because DECISIONS.md does not select a Traefik mode. Perf numbers were not collected: PLAN.md requires the dedicated perf-test VM viaperf-test-vm, unavailable in this job environment. The benchmark harness compiled and passed Clippy but was not executed on this dev VM.Final
scripts/verify fastoutput:Targeted step 19 compiled-process check:
Full integration is red, documented in preceding comments with exact failing output and direct reruns. The third run reached stream regressions after passing release/HTTP/Docker/Podman/engine/TypeScript checks, then failed existing Podman grant-shrink timing; its single direct rerun failed in another Podman subcase. The new remote gate step was not reached by a complete integration run. This branch should not be merged until that gate is resolved and perf numbers are obtained from the perf VM.
Review-19 started. Branch agent-tunnel at
07dcf75; base devdbd6685. Scope: independent defensive audit of step 19. Read-only source; findings and test logs under /mnt/hdd/infraternal/jobs/review-19/.Review-19 finding (high):
crates/server/src/tunnel.rs:273-280,579-586retains each channel'sto_streamsender until the remote side sendsCLOSE;OPEN_ERRdoes not remove it.crates/agent/src/lib.rs:1034-1043can sendOPEN_ERR(for example stats unavailable) and return withoutCLOSE. Repeating such opens leaves entries in the per-tunnel map; after 256,reserve()refuses all further channels until reconnect. Fix: retire onOPEN_ERR, and remove/drop channel state when the paired server relay ends; test repeated failed opens followed by a successful open without reconnect.Review-19 finding (high): the direct tunnel has no 8 MiB aggregate receive-window budget.
crates/server/src/tunnel.rs:265-280reserves until 256 live channels with no sum-of-windows accounting;crates/server/src/stream.rs:1484-1510accounts only the client WebSocket budget, so opens from multiple clients can each reserve 256 KiB or more on the same tunnel.crates/agent/src/remote.rs:553-615likewise bounds channel count but not aggregate tunnel credit. At 256 channels and the default 256 KiB window, the advertised receive budget can reach 64 MiB per direction, eight times §3.5's 8 MiB connection budget. Fix: account credit/windows once per physical tunnel in both directions and reject opens or credit beyond that bound, with concurrent multi-client tests.Review-19 enrollment contract gaps (medium):
crates/server/src/storage/agents.rs:30-43,100-126compares SHA-256 digests through SQLiteWHERE secret_hash = ?; there is no constant-time comparison, despite the explicit job requirement. Token entropy remains 256 bits, so this is contract compliance rather than a demonstrated practical token recovery.crates/server/src/agents.rs:166-177returns malformed CSR without anagents.enrollaudit record; the outer failure middleware atcrates/server/src/lib.rs:576-583records the genericPOST /api/v1/agents/enrollaction instead. Fix with a constant-time digest check on a bounded candidate set (or amend the contract if DB equality is intended), and use the named enrollment audit action for CSR failures.Review-19 functional gap (medium):
crates/server/src/stream.rs:1162-1173treats every remoteOpenPayload::Sshas nonexistent, so a remote infraternal agent can never receive an SSH OPEN. In addition,crates/server/src/stream.rs:1289-1328configures SSH settings only on the embedded infraternal agent. A user trying to SSH through a remote host receivesnot_founddespite an active tunnel. Fix the host-to-infraternal-agent ownership route for SSH targets and deliver authorized SSH settings to the chosen remote infraternal agent, with a real sshd tunnel test. This concerns the §6.2 same-channel-path contract; step 21 may cover persistence separately.Review-19 certificate lifecycle gap (medium):
crates/server/src/enrollment_ca.rs:96-101hardcodes the enrollment CA validity to 2026-01-01 through 2036-01-01. A server first installed after 2026-01-01 does not receive a ten-year CA as §6.1 states; a new install after 2036-01-01 generates an already-expired CA, making enrollment and mTLS unusable. Fix: persist a creation-time CA certificate with a ten-year validity from issuance, and verify key/cert match on reload. Preserve existing certificate bytes to avoid breaking pins.Review-19 security finding (high): user authorization loss closes attached remote channels, but does not terminate remote persistent processes.
crates/server/src/stream.rs:636-713,2591-2670closes the attachment on token/role revocation;crates/server/src/stream.rs:204-220sweeps onlyembedded_agent.active_ssh_bindings()(and the same local runtime for persistent exec). The remote runtime atcrates/agent/src/remote.rs:397-402uses the same persistent exec implementation, yet there is no tunnel command to terminate those sessions or remote sweep. A user can start a persistent remote exec, lose access, and its process remains on the remote host after the stream is cut. Fix: propagate durable authorization loss to the owning remote infraternal agent and terminate its persistent sessions; test revocation during a remote persistent exec against a real engine.Review-19 finished at reviewed head
07dcf75(basedbd6685). Verdict: request changes. Full independent audit:/mnt/hdd/infraternal/jobs/review-19/findings.md.Findings: high —
OPEN_ERRleaks per-tunnel channel slots (256 failed opens can block later opens); high — no 8 MiB aggregate tunnel credit budget across clients; high — user authorization loss closes a remote persistent exec attachment but leaves its process alive. Medium — remote SSH cannot route; token digest comparison is not constant-time; bad enrollment/rotation CSRs lack the contract's named audit actions; CA certificate expiry is fixed at 2036 rather than ten years after creation. Each finding has a scenario, location and fix in the report and earlier issue comments.Evidence: two simultaneous enrollment requests with one token produced exactly one success. The original enrollment test passed; disabling the consumed-token predicate made it fail (second response 200 instead of 401); disabling revocation made it fail (identity remained active); restoring both made it pass. The direct Docker + liveness tunnel integration step passed. All scratch edits were restored; the throwaway worktree was removed. This review branch remains unchanged.
Direct integration output:
scripts/verify fastoutput (exit 0):fix-19 started on branch
agent-tunnel, base head07dcf7547a12d6b8033eabfcd77334cb81773799. Fetched and merged currentdev: already up to date. Scope: review findings 1, 2, 3, 5, 6, 7 and TLS 1.3; remote SSH routing remains out of scope.Review follow-up evidence:
git fetch origin devand merge reportedAlready up to dateat07dcf75. The server tunnel kept channel-map senders until CLOSE, while ordinary failed OPENs end at OPEN_ERR; reservation accounting was per client socket. New implementation retires OPEN_ERR and relay teardown, reserves bounded windows on the shared tunnel, and adds channel-zero remote persistent-exec termination with an acknowledgment andunconfirmedaudit outcome. Existing CA PEM bytes are retained and a new CA receives validity at creation; both tunnel TLS configs now select TLS 1.3. Targeted compilation of server, infraternal agent and binary passed before the latest tests/docs edits.Regression proof for high findings 1 and 2: baseline
tunnel::tests::failed_opens_release_slots_and_tunnel_budget_across_clientspassed (1 passed; 0 failed). Temporarily replacing channel retirement with a no-op failed atfailed OPEN must free a slot(FAILED. 0 passed; 1 failed); original source restored. Temporarily raising the tunnel reservation budget to 64 MiB failed atanother client cannot exceed physical tunnel credit(FAILED. 0 passed; 1 failed); original source restored. The test opens 257 failed slots, then reserves 32 concurrent 256 KiB channels on one tunnel and checks that another client is refused until a relay tears down.Verification scheduling: at 10:54 VM local time
uptimereportedload average: 14.34, 13.79, 11.99on an 8-core VM. The requested integration gate requires load5 below 8, and the job rule defers heavy builds while load is above 10. Changes are committed through564f1bc; targeted unit and API audit tests passed before the final real-Docker test additions. Waiting for the shared VM load to fall before running compiled tunnel regression and full gates.High finding 3 proof against real Docker: the compiled tunnel test passed with 257 failed OPENs, a later successful exec, and a remote persistent
shprocess visibly running before token revocation. It then observed process exit within the three-second bound and an acceptedstream.session.terminateaudit. Temporarily disabling the remote authorization recheck made the same test fail after three seconds with the remotesh -c while :; do sleep 60; donestill visible indocker exec ps -o args(FAILED. 0 passed; 1 failed); source restored. The firstscripts/verify fastrun stopped at clippy'smatch_same_armsdiagnostic for the new protocol types; merged those match arms and rerunning.Fast gate progress: Clippy now passes. The full workspace cargo test reached
infraternal-protoand found the newunsupportedcode missing from its expected status table (left: 27,right: 28). Added(Unsupported, 501, false); rerun pending load reduction. The real compiled Docker tunnel regression passed before this gate, including 257 failed OPENs and process revocation; its remote-recheck mutation failed as expected and was restored.Integration gate reached the compiled CLI/MCP step after passing release build, real HTTP, Docker/rootless Podman sockets, engine adapters, bounded log snapshots, TypeScript stream client, stream authorization/CLOSE regressions, blocked PTY termination, and resource HTTP API. It failed because my remote SSH
unsupportedcheck classified an ordinary SSH host as remote, returningSSH routing to a remote infraternal agent is not implemented. Fixed ownership detection: only an enrolled remote identity uses that error; ordinary SSH hosts remain on the embedded SSH path. Direct CLI/MCP rerun in progress. This was a real regression, not the known PTY resize flake.Corrected head
43a304b: ordinary SSH hosts route through the embedded infraternal agent; enrolled remote hosts returnOPEN_ERR unsupported. The direct compiled CLI/MCP test passed (1 passed; 0 failed, 128.62s), and the compiled Docker tunnel test passed with 257 failed OPENs, later exec, remote SSH unsupported, revocation cleanup, and accepted termination audit (1 passed; 0 failed, 11.61s).scripts/verify fastpassed on the corrected source, verbatim:The full integration rerun awaits load5 below 8. The first integration attempt passed all steps through resource HTTP API, then failed CLI/MCP because of the SSH ownership regression described above; the direct CLI/MCP rerun passed after the fix.
fix-19 finished on branch
agent-tunnel, head43a304bb6c41755c419e67b3783e5c18722cda07(base07dcf7547a12d6b8033eabfcd77334cb81773799; currentdevmerged and already up to date). No push. Worktree clean.Fixed review findings 1, 2, 3, 5, 6, 7 and enforced TLS 1.3 on both tunnel endpoints. Remote SSH routing remains out of scope; enrolled remote hosts return
unsupported, while ordinary SSH hosts still work through the embedded infraternal agent. The compiled Docker tunnel test proves 257 failed OPENs then successful OPEN on one tunnel, remote persistent exec termination after user token revocation, and the remote SSH error. Disabling channel retirement, lifting aggregate tunnel credit to 64 MiB, and disabling the remote authorization recheck each made their corresponding high regression test fail; source restored after each proof. The compiled CLI/MCP SSH test and real Docker tunnel test passed on the corrected head. No known-flake rerun was needed.scripts/verify fastoutput verbatim:scripts/verify integrationstarted when load5 was 7.30 and output verbatim:Starting verify-19 on branch agent-tunnel, base head
43a304bb6c. I am checking the requested step 19 findings and regression tests, then will merge current dev and run the required gates.Negative proof at
43a304b(isolated throwaway worktree):cargo test -p infraternal-server --lib failed_opens_release_slots_and_tunnel_budget_across_clientspassed 1/1 on the baseline. With channel retirement disabled, it failed withfailed OPEN must free a slot(exit 101).cargo test -p infraternal-agent --lib physical_tunnel_credit_is_shared_across_clients_and_releasedpassed 1/1 on the baseline. With the infraternal-agent aggregate CREDIT guard removed, it failed at its post-OPEN over-budget CREDIT assertion (exit 101).TunnelService::terminate_sessiondisabled,real_infraternal_agent_process_enrolls_and_reconnects_with_new_nonceran against real Docker and failed after revocation because the remote persistent exec was still running (exit 101).The remote process test checks the confirmed termination and accepted audit.
spawn_remote_persistent_exec_recheckalso records anunconfirmedcompletion audit for a missing or negative acknowledgment; the current suite does not directly force that branch.Verdict: verified.
Branch and merge head:
agent-tunnel,43a304bb6c41755c419e67b3783e5c18722cda07.git fetch origin devfetcheddbd66850f8a5d6190e65b4bd364e075950033425; mergingFETCH_HEADreturnedAlready up to date., so no merge commit was needed.Findings
Closed.
crates/server/src/tunnel.rs:714-715retires a remote channel afterOPEN_ERR; relay teardown also retires it at line 371. The unit test at line 828 checks repeated failed opens and reservation reuse. The real tunnel test atcrates/infraternal/tests/agent_tunnel.rs:114sends 257 rejected opens and then executes a command at line 240. In the throwaway worktree, disabling retirement made the unit test fail withfailed OPEN must free a slot.Closed. Server reservation and frame accounting are in
crates/server/src/tunnel.rs:328-371,764-806; infraternal-agent reservation and accounting are incrates/agent/src/remote.rs:51-119. The server test attunnel.rs:828and infraternal-agent test atremote.rs:741both send an over-budgetCREDITafterOPEN. Removing the server guard failed the server test at that assertion; removing the infraternal-agent guard failed its test at the same assertion.Closed.
crates/server/src/stream.rs:410-553rechecks remote persistent exec authorization, audits termination start, requests termination from the owning infraternal agent, and audits completion asacceptedorunconfirmed(line 536). The protocol requirement is also indocs/contracts.md:1480. The real-Docker tunnel test atcrates/infraternal/tests/agent_tunnel.rs:315,339revokes the token, checks that the persistent process exits, and checks the accepted completion audit. DisablingTunnelService::terminate_sessionmade this test fail because the remote process stayed alive. A separate throwaway failure-injection run terminated the real PTY but returnedconfirmed: false; the test passed after requiring anunconfirmed/engine_erroraudit row.Closed.
crates/server/src/storage/agents.rs:32-48,97-123,304-308uses a bounded lookup and constant-time digest comparison.join_token_is_hashed_single_use_and_auditedat line 335 covers enrollment token hashing and single use.Closed. Named enrollment and certificate-rotation rejection audits are written in
crates/server/src/agents.rs:182andcrates/server/src/tunnel.rs:463,495.bad_csr_and_missing_rotation_identity_have_named_failure_auditsatcrates/server/tests/agent_enrollment.rs:143checks the audit names and forces audit storage failures, which must return 500.Closed.
crates/server/src/enrollment_ca.rs:95-128sets validity relative to creation and loads an existing certificate.ca_persists_and_enrollment_certificate_ignores_requested_ca_extensionsat line 240 checks validity and byte preservation after reload.TLS and SSH checks
crates/server/src/enrollment_ca.rs:179,crates/agent/src/remote.rs:269).tunnel_listener_refuses_tls12atenrollment_ca.rs:278proves a TLS 1.2 client is refused.unsupportedincrates/server/src/stream.rs:1401-1402; the real tunnel test asserts it atcrates/infraternal/tests/agent_tunnel.rs:145-178. The local SSH product test atcrates/server/tests/ssh_product.rs:143connects through an isolated realsshdand opens the SSH stream.Negative proof
In the throwaway worktree created at
43a304b, the baseline server tunnel and infraternal-agent credit tests passed. Mutants disabling channel retirement and each endpoint's aggregate credit guard made their tests fail. Disabling remote session termination made the real-Docker revocation test fail. Both throwaway worktrees were removed; the main worktree remains clean.Gate output
Fast (exit 0):
Integration attempt 1 (started at load5 7.83; exit 1 on the documented #42 Podman cleanup flake):
Integration retry (started at load5 4.24; exit 0):
Starting merge-19 on branch agent-tunnel at base head
43a304bb6c. I will rename the remote agents migration to 0015, inspect references, and run the requested fast and step 19 integration checks.Finding: crates/server/tests/migrations.rs current_versions() expects migration version 17 via (1..=14).chain([17]); after the rename it must expect versions 1 through 15. The repository text search found no old filename references or other literal 0017 references.
Targeted verification finding: both enrollment tests in crates/server/tests/agent_enrollment.rs passed, and the mTLS listener test in crates/server/tests/agent_tunnel_tls.rs passed. The real Docker tunnel test in crates/infraternal/tests/agent_tunnel.rs failed twice at line 345: Docker showed the persistent process had exited, but the test did not observe an accepted stream.session.terminate audit row with error_code forbidden within 2 seconds. The failure reproduces with the integration gate environment flags INFRATERNAL_TUNNEL_DOCKER_TEST=1 and INFRATERNAL_TUNNEL_LIVENESS_TEST=1. The migration change does not touch tunnel behavior.
Finished merge-19 on branch agent-tunnel.
Head SHA:
e9a434f65dscripts/verify fast output (verbatim):
WARN load5=8.68 exceeds online CPU count=8; timing-sensitive checks may be delayed
PASS fast: migrations are append-only
PASS fast: workspace package dependencies
PASS fast: JavaScript dependencies
PASS fast: build web app
PASS fast: cargo fmt --check
PASS fast: cargo clippy
PASS fast: cargo test
PASS fast: svelte-check
PASS fast: UI package svelte-check
PASS fast: stream package typecheck
PASS fast: Vitest
PASS fast: UI package Vitest
PASS fast: stream package Vitest
PASS fast: OpenAPI and TypeScript client drift
PASS fast: terminal TypeScript
PASS fast: terminal Vitest
Targeted step 19 tests: enrollment suite passed 2 tests; mTLS listener test passed 1 test. The real Docker tunnel test failed twice at crates/infraternal/tests/agent_tunnel.rs:345 with remote termination completion was not audited. The test observed the persistent process exit but did not observe the accepted stream.session.terminate audit row with error_code forbidden within 2 seconds. This finding is recorded above.
The migration blob hash before and after rename is
80634cbdb1. The only source reference to version 17 was updated in crates/server/tests/migrations.rs. No SQLx offline data exists in the repository.fix-19b starting on branch
agent-tunnel, base heade9a434f65d000f93436f02251be977edaf64965f. I’m tracing the test and product audit ordering to determine whether the Docker failure is a test read race or a missing/incorrect completion audit. I’ll run the test 10× at the current load, runscripts/verify fast, and commit the scoped fix.Finding (a): this is a test read race.
exercise_docker_execobserves the process gone with Docker, then previously allowed only 2 s for the accepted audit row. In the product path,drive_persistent_execreports termination only afterPtySession.close()returns; engineclose()waits until the exec is no longer running. The infraternal agent waits up to 5 s for that cleanup (crates/agent/src/remote.rs), and the server waits up to 6 s for the result, persists session exit state, then writes the completion audit (crates/server/src/stream.rs). The test now polls for up to 10 s for a completion row, still requiresoutcome = acceptedanderror_code = forbidden, and includes actual audit rows in failure output. No product change. The 10× loop and fast gate remain to run after VM load drops below 10.Finding (b): the 10-run loop with the real Docker exercise enabled was 0/10. Every run observed the container process disappear, then received a completion audit row with
outcome = unconfirmed,error_code = engine_error; these are completion rows, not a 10 s read timeout. Sample row: requestreq_01M3C6AST6JY26G53SE6YY5YXY, targets["s_01M3C6AS951SJ08TX4SAEJENPG","h_01M3C6AFQHGZF35B0QGRWDSEXY/docker/e09293dca3f991377ab8e0b2898c76cc94816d746328da3fd5aab749fb7cf042"], outcomeunconfirmed, errorengine_error. Five-minute load before the runs ranged 7.16–7.55.The code has a nested deadline mismatch:
crates/engine/src/lib.rsreports an unknown cleanup result after 3 s while its ownedterminate_execcleanup continues; the infraternal agent waits 5 s and the server waits 6 s. The cleanup sequence itself includes Docker inspect/signal/verification work plus a final 2 s process-group check, so the 3 s engine acknowledgment can expire while the process is subsequently observed gone. I’m addressing the too-short engine acknowledgment window while preserving the accepted-only test assertion, then I’ll rerun the Docker loop and fast gate.Post-fix evidence: after changing the engine cleanup acknowledgment deadline from 3 s to 4 s, the real Docker tunnel test passed 10/10. Each run had
INFRATERNAL_TUNNEL_DOCKER_TEST=1; the test still requires the completion audit to beacceptedwithforbidden. The pre-run load averages were 9.80 / 8.09 / 8.50; five-minute load samples before individual runs ranged 8.17–8.91. The initial 10-run loop before the product fix was 0/10 and recordedunconfirmed/engine_errorin every run.Finished on branch
agent-tunnel, final head78a20601d6fc9e2eff97113b89264b814eae542e.Diagnosis was (b), not a read race. The first Docker-enabled 10× loop was 0/10 at five-minute load 7.16–7.55; each run observed the process gone but recorded
unconfirmed/engine_error. A sample actual row was requestreq_01M3C6AST6JY26G53SE6YY5YXY, targets["s_01M3C6AS951SJ08TX4SAEJENPG","h_01M3C6AFQHGZF35B0QGRWDSEXY/docker/e09293dca3f991377ab8e0b2898c76cc94816d746328da3fd5aab749fb7cf042"], outcomeunconfirmed, errorengine_error.The engine cleanup supervisor’s 3 s acknowledgment deadline could report unknown while its cleanup future continued. The infraternal agent and server have 5 s and 6 s outer waits. I raised the bounded engine acknowledgment deadline to 4 s. The test now polls up to 10 s for the completion row, reports actual rows on failure, and still requires
acceptedwithforbidden; it does not acceptunconfirmed.After the product fix, the real Docker tunnel test passed 10/10. Five-minute load before the loop was 8.09 and ranged 8.17–8.91 during it.
scripts/verify fastpassed. Verbatim output:Commits:
10c7728 fix(agent-tunnel): allow remote terminate cleanup confirmation;78a2060 test(agent-tunnel): keep audit poll clippy-clean. Worktree is clean. Issue remains open.Starting doc-19 on branch
agent-tunnel, base head78a2060. I’m checking cleanup acknowledgement wording and timing references indocs/contracts.md,docs/DESIGN.md, and tests before updating the contract and runningscripts/verify fast.Finding:
docs/contracts.mdhad one stale “after three seconds” reference;docs/DESIGN.mdhas no cleanup deadline reference. The test scan foundcrates/agent/src/lib.rswaiting three seconds forPtySession::close()completion, which can include the engine acknowledgement. I raised that bound to five seconds; the separate PTY replay and actor-exit timeouts remain three seconds.Finished doc-19 on branch
agent-tunnel.Commit:
b63a853186e046a774ff367ee74f9f4d175e58a7(docs(contracts): cleanup acknowledgement deadline is four seconds).scripts/verify fastoutput:Starting diagnosis on detached
devatd9197837c5adb20ad0da2a6b8872ed2826a92aee(expected origin/dev head). Working tree is clean. Current load average is 12.09 / 9.63 / 9.11, so I am waiting for load5 below 8 before the browser gate, up to 30 minutes. I will record the gate result and trace the 401 if it reproduces.Diff finding while the gate runs:
dbd6685..b63a853changescrates/server/src/resources.rsthroughfdedea2for remote inventory. Inlist_containers(current lines 1572–1640), the new remote-host branch returns 403 for a scope denial and 404/host_offline for lookup or inventory conditions;--no-local-agentreturns host_offline. The ordinary localContainersReadauthorization remains below that branch at lines 1642–1650. The range has no changes tocrates/server/src/auth.rs,authorization.rs, or browser auth handling. The 401 response is produced earlier byauthenticate()incrates/server/src/auth.rs:1345–1372when the bearer token is missing or invalid. I’m waiting for the dev-head browser result before concluding whether this is the cause.Diagnosis on
devheadd9197837c5adb20ad0da2a6b8872ed2826a92aee:scripts/verify browserexited 0. The reported 401-induced browser failure did not reproduce as a gate failure, so I did not createfix-dev-browser.The step 19 resource-listing change is not a 401 path.
fdedea2added the remote-host branches incrates/server/src/resources.rs:1572–1640; they return 403 for a scope denial and 404/host-offline for lookup or inventory failures.--no-local-agentalso returns host-offline. The ordinary local scope check remains at lines 1642–1650. The 401 path is bearer validation in unchangedcrates/server/src/auth.rs:1345–1372; thedbd6685..b63a853range did not change the auth or web refresh files.The likely branch-only trigger is
sidebar-sync's newcheckSidebarPersistenceinscripts/browser-check.mjs: commita6949b8adds a second passkey-authenticated context at lines 569–595 (the independent sign-in is line 594), and invokes the helper before the container flow at line 1855. That fits the failing request appearing later on/api/v1/containers. Commit63216e4adds the branch-only 401 path diagnostics at lines 1815–1817.One qualification: the code does not establish refresh-family revocation.
browser.newContext()has isolated cookies and storage (Playwright BrowserContext API); server passkey login callsupdate_passkey_and_issue_sessionwith no existing family id, which inserts a new random family rather than revoking the first (crates/server/src/storage/mod.rs:1136–1169, 2872). So the second-context check is the branch-only suspect, but the exact refresh-family mechanism is not proven by the current source.Verbatim
scripts/verify browseroutput (exit code 0):Starting o2-tunnel on branch
o2-tunnel, base6f6fd896214f05d9ef19a47e43fb8be2561ab626. Scope: deploy bundle opt-in TLS 1.3 mTLS listener with Traefik TCP passthrough, owner runbook, local real enrollment/certificate/TLS tests and fast/merge gates. Prepare only: no deployment or contact with o2.Dedicated SNI prerequisite found at base
6f6fd89:crates/server/src/lib.rs:370–373callsca.tunnel_tls_config(config.public_origin.host_str());crates/infraternal/src/main.rsoffers--tunnel-listenbut no independent hostname;crates/agent/src/remote.rshas no distinct TLS ALPN. Same-host HostSNI routing would capture browser/enrollment HTTPS because TCP routers run before HTTP (Traefik router docs). Preparing dedicatedagents.infraternal.kayg.orgrouting with an activation guard until a server-owned--tunnel-hostnameprerequisite lands. Asked owner for scope choice. Local passthrough can use localhost with a matching certificate and test real mTLS without claiming dedicated production hostname readiness. Official Traefik latest stable checked: v3.7.13, Docker manifest digestsha256:24841fe2de7304c149343d877d2923b4c8800a38ba015dea9174c23b20e344a0. Its CLI exposes healthcheck/version but no lint subcommand. Load5 exceeded 15 while other verification jobs were active; heavy builds deferred.Preparation commits saved:
c24fc75(opt-in TCP/unit templates, manifest migration, activation guard, runbook) and26bb392(isolated real install/rollback fixture and fail-closed configuration restore handling). No deployment or contact with o2.Dedicated SNI activation remains explicitly blocked on the server-owned independent hostname option. Default-off rendering was compared against base
6f6fd89; real output:bash -n,shellcheck -xandgit diff --checkpassed. Fast gate started after load1 fell to 9.60 withCARGO_BUILD_JOBS=2; merge will wait for load5 < 8 and no other active verification run. Both new local tests select resources by a per-invocation random id; no shared fixed systemd test unit is used.Pinned official Traefik accepted the rendered production SNI/TCP template in a local container with
--network none; only the existing HTTP route's ACME resolver was replaced with the default local TLS certificate (no ACME/network calls). Queried the actual file-provider API from inside the exact uniquely named container, then deleted it. Real output:This proves production routing configuration decoding and activation, not the still-blocked dedicated certificate hostname. The committed process fixture is being extended to include this production-template check before switching to matching localhost certificates for lifecycle testing.
Both real local fixtures passed against the native binary built by this checkout. No o2 connection or deployment.
Passthrough output:
Isolated systemd install/upgrade/rollback output:
Evidence/decisions: initial same-SNI local fixture received Traefik default certificate; keeping HTTP/tunnel SNI distinct as production resolved it. Direct backend control identified the base rcgen leaf missing Authority Key Identifier: Python 3.13 create_default_context strict verification rejects it (verify_code 85). The probe now uses the documented TLS client constructor with CERT_REQUIRED, hostname checks and only the enrollment CA trusted. The actual rustls infraternal agent independently verifies and completes HELLO. No certificate or hostname verification is disabled; backend strict-X.509 compatibility remains a server-owning follow-up alongside the independent hostname option.
First fast gate failed at cargo test, but the gate runner printed the first 200 lines of compiler output and omitted the cause. A rerun using the documented Cargo CARGO_TERM_QUIET=true output setting (Cargo config) is running; tests are not filtered or skipped.
Fast gate failed twice at the unchanged server private-storage test; the quiet rerun exposed the cause. At
crates/server/tests/private_storage.rs:49, creating a symlink forserver.sqlite3-walfails withAlreadyExistsafterreopened.close().await. Source identity is unchanged from base: bothgit rev-parse 6f6fd89:crates/server/tests/private_storage.rsandgit hash-objectreturned5cb4a53ad123357bf0d4d44f594812f70cd0cf82.Actual gate excerpt:
An isolated rerun of the already compiled exact test passed:
This indicates intermittent existing behavior rather than a deterministic failure; no backend/test source was edited. Retrying the complete fast gate after load5 drops below 8 and active verification runs finish, then attempting the requested merge gate under the same required conditions. Both deployment-specific real fixtures pass.
Prepared owner runbook at current committed head
08e5ae5fb1e8d25786cc205e811e3d5413557e53; no push, deployment or contact with o2. Activation remains guarded on the independent server hostname prerequisite. Fast retries encountered the unchanged private-storage WAL cleanup race; an isolated exact rerun passed. A serial complete fast retry is pending the permitted build load, then merge will wait for load5<8 and no active verification run.o2 infraternal agent tunnel: owner runbook
Prepared under issue #40. No deployment or connection to o2 was performed.
Fresh installs and HTTP-only upgrades default to off.
--with-agent-tunnelenables the listener and routing; upgrades retain an already enabled tunnel
from the ownership manifest. Enrollment remains at the ordinary HTTPS origin.
Activation prerequisite
At base
6f6fd89,Server::openincrates/server/src/lib.rsissues the tunnelcertificate for the
--public-originhostname. There is no separate tunnelhostname option. The prepared opt-in unit requires
--tunnel-hostname agents.infraternal.kayg.org; install and upgrade refuseactivation with a binary lacking that option. The server-owning change must
implement the separate certificate SAN while preserving the existing default.
This branch does not implement that prerequisite. Do not change the public
origin to the tunnel hostname: it also controls passkeys and OAuth.
The local test uses a matching localhost certificate. It proves TCP
passthrough, identity and TLS enforcement, not production hostname readiness.
Routing and DNS
Add this DNS-only A record in Cloudflare (proxy disabled, automatic TTL):
agents.infraternal.kayg.org10.69.69.52Keep
infraternal.kayg.organd its HTTP route. Enrollment and user APIs usehttps://infraternal.kayg.org; tunnels usewss://agents.infraternal.kayg.org/api/v1/tunnelon 443.The TCP router matches only the dedicated SNI on
websecure, applies TCPipAllowListwith10.69.69.0/24, and forwards encrypted bytes to the inspectedDocker bridge gateway on 7443 (currently
172.18.0.1:7443). TLS 1.3, clientcertificate authentication, identity/serial checks and revocation remain in
the server. The TCP route has no certificate forwarding header, ACME resolver,
backend TLS re-encryption or PROXY protocol.
Dedicated SNI is necessary because TCP routers precede HTTP
routers.
A same-host rule would also capture browser and enrollment HTTPS; the proxy
cannot inspect encrypted paths. See the TCP TLS passthrough
reference
and TCP IP allowlist
reference.
Remote hosts must already be mesh peers. Another proxy or source NAT would
change the allowlist's peer address and requires a separate review.
The owner manages tunnel DNS separately; the installer manages only the
original application DNS record. HTTP and optional TCP sections share the one
owned dynamic file. Uninstall removes both sections; remove tunnel DNS manually.
Port 7443 is an internal bridge listener, not another public inbound port.
Owner activation (after the prerequisite lands)
On the dev VM, use the reviewed commit containing both deployment and server
hostname changes:
Inspect the checksummed bundle. On o2, confirm
websecureis TCP :443 and thefile provider watches
/dynamic. Add the DNS-only record above. The owner canthen deploy from the dev VM:
For a manually transferred and extracted bundle, choose the relevant path on o2:
The upgrade validates the binary's options, checks owned unit/proxy hashes,
renders candidates and runs
systemd-analyze verifybefore replacing files.It checks the installed Traefik's CLI for lint support. The official pinned
v3.7.13 release has
only
versionandhealthcheck, with no config lint CLI. That absence isreported; healthcheck is not presented as lint. The local test validates real
file-provider router/middleware state. A future lint CLI needs a reviewed
adapter; deployment refuses rather than silently skipping it.
Original unit, proxy and manifest bytes survive through candidate health
verification. Failure restores them before the existing binary/database
rollback. Hand-edited owned files abort. Legacy proxy files without a hash
are adopted only if they exactly match the HTTP-only template. Existing
snapshot and capacity checks still apply; see DEPLOY.md.
The unit remains disabled at boot unless the owner explicitly enables it.
Check on o2:
HTTP health must still succeed from the mesh, and the TCP router must have no
file-provider errors. HTTP health alone does not prove the tunnel works.
Enroll a remote infraternal agent
Run as the remote OS user that will run the infraternal agent, on a mesh host.
Use Bash for the following snippets (
bashif the login shell is zsh).Use a human access token with
agents:enroll,agents:revoke,hosts:readandcontainers:read. These commands keep credentials in private files and do notplace them in curl arguments or print the join capability:
The final command stays in the foreground. Reuse that private identity
directory on restart. A join token is one-use; mint a new one if consumption
succeeded but enrollment did not finish. Production enrollment verifies its
ordinary HTTPS origin; the tunnel checks its dedicated hostname and pins the
enrollment CA returned during enrollment.
Verify identity and revocation through passthrough
In another shell, set
identity_dirandcredential_dirto the same paths.Read only the public certificate and CA for the probe; retain the private key
in its identity file:
A successful remote inventory query proves identity-bound HELLO. Compare the
client certificate SAN
urn:infraternal:agent:<infraternal_id>locally with theidentity id if needed. A TLS handshake alone does not prove current admission:
the server checks serial/revocation before passing the connection to HTTP.
Revoke a disposable test identity using the same
new_ulidfunction:Within 15 seconds the running infraternal agent must disconnect. Reconnecting
with that identity must never complete HELLO or publish inventory. A foreign
certificate must also be refused before the WebSocket upgrade. Inspect
agents.enrollandagents.revokein the audit UI, and confirm an authenticatedhost-list request still succeeds: revocation must preserve human tokens.
Delete the private temporary credential/probe directory when finished:
Local reproduction
After
scripts/verify fastbuilds this checkout's native binary:The test owns a uniquely named container, temporary database/identity and two
real processes. Every proxy entrypoint binds loopback. It deletes only those
resources. It uses the same TCP template with localhost and a loopback
allowlist, plus a loopback HTTP enrollment fixture router (the product's
development exception). Production enrollment remains HTTPS. It checks real
file-provider state, unit rendering/verification, CLI enrollment, CA/hostname
validation, missing/foreign certificate rejection, TLS 1.2's explicit protocol
alert, HELLO/inventory, revocation and TCP allowlist enforcement. No runtime hot
path is introduced, and no performance claims come from this shared VM.
The local HTTP and tunnel SNI names stay distinct, like production. An earlier
same-SNI fixture received Traefik's default certificate; the distinct-SNI
fixture reaches the server certificate and the real infraternal agent completes
HELLO. The positive probe checks the direct backend first as a control.
The base server's generated leaf omits Authority Key Identifier. Python 3.13's
create_default_contextadds RFC 5280 strict checks and rejects that extensionomission. The probe uses the documented
SSLContext(PROTOCOL_TLS_CLIENT)constructor with
CERT_REQUIRED, hostname checks and only the enrollment CAtrusted; it never disables chain or hostname verification. The real rustls
infraternal agent independently verifies the passthrough connection. See the
Python TLS client context reference.
The server-owning change should address strict X.509 compatibility alongside
the independent hostname option; this deployment job does not alter certificates.
The second test uses unique systemd/binary/state paths, a private bundle and
a uniquely named Docker network/proxy. It exercises default-off install and
upgrade, refuses the missing hostname prerequisite before mutation, forces a
candidate health failure after changing valid unit/proxy templates, checks
that configuration/binary bytes are restored, and refuses a hand-edited proxy.
It stops and deletes only that invocation's own resources.
Actual local test output
Third fast attempt (RUST_TEST_THREADS=1, CARGO_BUILD_JOBS=2, CARGO_TERM_QUIET=true; no filtered/skipped tests) also failed at unchanged
private_storage.rs:49WAL symlink AlreadyExists. Direct local deployment/TLS fixtures remain passing. Queuing the required merge attempt once load5<8 and no other scripts/verify process is active; merge includes fast, so this baseline race may stop it before release/integration/browser checks.Actual serial fast output excerpt:
Current committed head
08e5ae5fb1e8d25786cc205e811e3d5413557e53; no unowned source edits, deployment, o2 contact or push.Orchestrator, gate policy from 00:10 IST: with about 12 jobs running, waiting for 'no other scripts/verify run active' deadlocks. Jobs: run
scripts/verify fastplus your targeted tests, then finish and report 'merge gate pending'. Do not wait forscripts/verify merge. The orchestrator runs merge gates centrally, one at a time, on ready branches.Preparation handoff: branch
o2-tunnel, base6f6fd896214f05d9ef19a47e43fb8be2561ab626, head1adef70e1e0dab0c8b38a38d27a961815e8d7248. Working tree clean. No push, deployment, or contact with o2.Owner runbook is
docs/DEPLOY-TUNNEL.md; the complete runbook and earlier evidence are in this issue. Opt-in unit/TCP passthrough config, default-off install/upgrade handling, candidate validation, configuration ownership and rollback are committed. The pinned official Traefik image, separate tunnel SNI, TCP mesh allowlist and owner commands are documented there.Activation remains blocked: the base server derives the tunnel certificate hostname from its browser public origin and does not provide
--tunnel-hostname. The prepared opt-in guard rejects the current binary before persistent changes. Backend ownership is outside this job; no answer arrived to the scope question. Independent security review remains with the orchestrator; this job did not delegate.Local tests (verbatim):
Shell syntax, shellcheck and git diff --check passed. Disabled rendered configurations are byte-identical to the base. These are functional checks on the dev VM; no performance numbers are claimed.
scripts/verify fastfailed on all three attempts, including serial Rust tests. Final failure output (verbatim):The failing
private_storage.rssource blob is unchanged from base (5cb4a53ad123357bf0d4d44f594812f70cd0cf82). Running that exact compiled test alone passed; the gate remains failed and no tests were skipped or changed.scripts/verify mergewas NOT RUN. Its waiting shell was queued while enforcing load5<8, no other active scripts/verify and no concurrent full release build. Repeated verification jobs held the slot; a new remote end-to-end run began while the browser gate was still active. Final observed process check (verbatim):Final load snapshot:
2.67 2.94 4.60 1/1496 1665857. Stopped only this job's uniquely identified waiting shell (PID1523258), before any merge invocation; no other process was signaled. There is no merge.log because the merge gate never started. This is a blocked gate handoff, not a green merge claim. The orchestrator must runscripts/verify mergein an available slot after resolving the baseline fast failure, and complete independent security review.progress.mdis excluded and preserves the continuation state. Issue left open.Resuming
rs-tunnel-hostnamein branchtunnel-hostname. Base head:450c297354f4d2f07c84ace930949a6bacdbb900(origin/devmerge-base). The worktree starts clean at5c6359ba. I am reviewing the existing implementation and will complete any remaining items for--tunnel-hostname, enrollment URL handoff, deployment activation, tests and docs. No NetBird or network configuration changes are in scope. I will finish with the requestedscripts/verify fastand report its output verbatim.Verification finding:
scripts/verify fastfailed atcargo test --workspacebefore running the tests. Cargo reportedfailed to build archive at .../libssh_encoding-ced0d006fa0abe46.rlib: failed to open object file: No such file or directoryand a second missing object path under the configuredrs-tunnel-hostname/debug/deps. Inspection showed/home/kayg/.cache/infraternal-targets/rs-tunnel-hostnameis a symlink to this worktree'starget/on/mnt/hdd; the root filesystem has 21 GiB free. I will rerun the required gate withCARGO_TARGET_DIRon the root filesystem so compiler outputs stay off/mnt/hdd. No test assertion failed.Finished the
rs-tunnel-hostnameresume in branchtunnel-hostname.HEAD:
5c6359ba715aec3e8122872c0bfa8f6c7fb12347. The existing commit already contains the complete hostname/SAN/SNI behavior, unchanged public HTTPS origin, enrollment URL handoff and identity persistence, deployment flag/manifest handling, tests, and owner-placeholder runbook. No further tracked edits were needed; the worktree is clean. No NetBird commands or network configuration were run or changed.The first
fastattempt stopped when Cargo could not write objects through the preset target path, which was a symlink into/mnt/hdd. The successful rerun used/home/kayg/.cache/infraternal-targets/rs-tunnel-hostname-rootfor Cargo output. Full gate output from that rerun:INFRATERAL_VERIFY_RUN_ID=rs-tunnel-hostname scripts/verify fastexited 0. Bun's one low advisory is explicitly report-only; all Rust checks and the full gate passed.