deploy-monitor01: rootless server and direct NetBird mTLS tunnel #50
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Job deploy-monitor01. Branch deploy-monitor01; base/head
1dd62b854f. Build a Debian-compatible x86_64 OCI image, deploy the server under kayg rootless Podman on monitor01, preserve fresh production setup, verify HTTPS and browser screenshots, restart/daemon-reload and subuid, and prove remote enrollment/tunnel on an isolated throwaway instance. Only infraternal-owned files and containers are changed; no DNS or infrastructure changes. Owner authorization and scope supplied in the job prompt.Finding: monitor01 has no existing infraternal Quadlet, data or route. Grafana publishes 172.18.0.1:13030 and uses :U, NoNewPrivileges, Restart=always and default.target; its route matches the supplied NetBird allowlist and HTTPS/HSTS headers. Server::open already issues the tunnel leaf for public_origin.host_str(), and the CLI accepts an explicit WSS URL. The minimum change is deployment only: direct 10.69.69.50:7443 with hostname infraternal.kayg.org, no parked hostname code or DNS changes.
Commit
b214592reuses the checksummed o2 build pipeline for x86_64 and Rust 1.99.0. Research: https://docs.podman.io/en/v5.4.2/markdown/podman-systemd.unit.5.html ; https://doc.traefik.io/traefik/v3.4/user-guides/websocket/ ; https://blog.rust-lang.org/2026/10/01/Rust-1.99.0/ . Build and fast gate are running. Production setup has not been attempted.Fast gate finding (unchanged runtime code): formatting and Clippy passed, then the existing stream::tests::contended_native_close_fence_preserves_original_watcher_deadline test failed while the disposable release build was compiling concurrently. Exact failure:
Full gate output: /mnt/hdd/infraternal/artifacts/deploy-monitor01/verify-fast.log; full cargo log retained by scripts/verify under artifacts/verify/deploy-monitor01. No test/threshold changes. Will rerun scripts/verify fast after the release build finishes to remove this job's compilation contention.
Release build evidence: native x86_64 ELF, stripped, highest glibc symbol requirement 2.39. Bundle checksums passed. Packaging then failed because a root-owned o2 Traefik template could not be replaced by the rootless driver. Fixed in
cc789f8by returning the output files to the bind output directory owner inside the disposable build container. Native Cargo caches retained; retry is running. No image or configuration has reached monitor01 yet.Live deployment findings:
e68458a.Artifacts: /mnt/hdd/infraternal/artifacts/deploy-monitor01/. Fast retry cargo tests passed; remaining gate stages are running.
Completed deploy-monitor01. Branch deploy-monitor01; base
1dd62b854f; heada0a2e39ac3. No push, merge or issue closure.Production runs image
localhost/infraternal:monitor01-3ab412ac3c826b9ddbcfcea6849c3a54b049cbeb(OCI image ID a2089ef09812cf1577d2975e290ec0b6aecb585136a3e8d10c0b169e699ec4bd), built in a disposable Ubuntu 24.04 container with Rust 1.99.0 and Bun 1.4.2. Native x86_64 ELF, stripped; maximum required glibc symbol 2.39. All final bundle checksums and actual Debian runtime execution passed. Subsequent commits change only deployment bootstrap/docs and test behavior;git diff 3ab412a HEAD -- crates packages Cargo.lock Cargo.toml bun.lock rust-toolchain.toml scripts/build-web deploy/o2/build.shis empty. No server, protocol, authentication or tunnel-runtime changes were required.Exact infraternal files on monitor01 (only these application paths were added):
The
/home/kayg/services/infraternalparent was created as kayg-owned 0700 under the existing root-owned services directory. The data directory is owned by the mapped subordinate UID. Podman manages its own image layers/container storage; systemd generates/run/user/1001/systemd/generator/infraternal.serviceand/run/user/1001/systemd/generator/default.target.wants/infraternal.service. No other containers, Quadlets or routes were altered; no DNS, firewall, network or NetBird changes; no Traefik restart; o2 untouched.Live verification output:
The idempotent deploy rerun took the listed SQLite backup and retained the exact CA certificate bytes. Production users, passkeys and remote infraternal agents remain zero. No setup token, join token, key or human credential was printed. Screenshots were also visually inspected; no overlap/clipping at either requested width.
Screenshots:
The remote fixture used the same image in a second rootless server container on loopback HTTP 17011 and NetBird mTLS 17443, with separate temporary data and CA, and a second rootless infraternal agent container. Only the throwaway database received a scoped test human credential; real API join/enrollment, certificate issuance, CLI and identity-bound HELLO/inventory were exercised. Cleanup removed all fixture containers, database, CA and credentials; fixture ports have no listening socket. Missing client certificates and TLS 1.2 were refused, healthy liveness held, restart used a new nonce, and revocation closed the client within 15s and refused reconnection while retaining its human credential.
The production direct endpoint is
wss://infraternal.kayg.org:7443/api/v1/tunnel, verifying the public-origin hostname and pinning the enrollment CA. Enrollment after owner setup useshttps://infraternal.kayg.org. Traefik handles ordinary browser/API HTTP and WebSocket routing; the mTLS tunnel bypasses it. A production WebSocket request without a ticket returned HTTP 401ticket_invalid, proving the upgrade request reaches the product handler. Valid-ticket production streaming was not exercised because production remains unsetup. The daemon-reload/default-target check is a reboot/generator simulation; no machine reboot was performed.Gate:
INFRATERNAL_VERIFY_RUN_ID=deploy-monitor01 scripts/verify fast, exit 0. Real output quoted verbatim:The initial fast run failed the unchanged existing 650ms native-close contention test while release compilation was running. Its evidence remains in verify-fast-initial-failure.log and the earlier issue comment. The unchanged gate passed on retry after compilation. Bun's existing low cookie advisory is report-only. ShellCheck (including the sourced helper; remote-expression info excluded), shell/JS/Python syntax checks and the repo skill validator passed. No thresholds weakened. This job did not run merge/nightly/perf-test-VM A/B or certify a release; no hot path changed and no performance result is claimed.
Artifact logs: build-verified.log, deploy-final.log, deploy-rerun.log, service.log, browser.log, tunnel-final.log, final-state.log and verify-fast.log under /mnt/hdd/infraternal/artifacts/deploy-monitor01/. The skill entry is .claude/skills/deploy-monitor01/SKILL.md and the operator runbook is docs/DEPLOY-MONITOR01.md.
Owner next step: open https://infraternal.kayg.org/ and complete first-run passkey setup. Read the one-time setup token privately from
podman logs kayg-infraternal-rootlessas kayg on monitor01 (access only with NetBird SSH). If expired, restart only infraternal.service and read the current container log. Production remains unsetup for the owner.Starting deploy-agent-binaries: only writer in /mnt/hdd/infraternal/wt-lp/deploy-agent-binaries, branch deploy-agent-binaries, base
eb8ddb354f. Scope: monitor01 deployment script/image/Quadlet and operator docs; build both same-commit x86_64 and aarch64 infraternal binaries, configure installation, deploy, verify health and preserved CA/data/setup. Required gate: INFRATERNAL_VERIFY_RUN_ID=deploy-agent-binaries scripts/verify fast; no merge gate. First-run setup will remain untouched. No other jobs launched.Confirmed server CLI accepts --tunnel-public-url wss://[:port]/api/v1/tunnel; its host supplies the tunnel TLS certificate name. monitor01 resolves infraternal.kayg.org to 10.69.69.50. Existing deployed image is monitor01-67dffa58513296c936c3909fd1999b85c37122a1 and service is active. GET /api/v1/hosts/install/public-key requires bearer authentication and both hosts:create / agents:enroll (crates/server/src/install.rs); authenticated check will be skipped until owner setup. SETUP_LIFETIME_MS is 3600000, and startup does not replace an unexpired unused token.
Implementation commits:
1562e4epackaging/Quadlet and9ee1c99997docs. Both same-commit release builds and scripts/verify fast are running. Deployment leaves the already-provisioned Traefik route untouched. Podman command inspection is logged by ExecStartPost so the service journal records actual nonsecret public configuration without adding server code outside this job ownership.Primary references checked: https://docs.podman.io/en/v5.4.2/markdown/podman-systemd.unit.5.html (Quadlet regeneration and service management); https://docs.podman.io/en/latest/markdown/podman-container-inspect.1.html (.Args formatting); https://doc.rust-lang.org/rustc/platform-support/aarch64-unknown-linux-gnu.html (GNU/Linux cross-target).
Pre-deployment production HTTP evidence: GET /api/health returns {"status":"ok","version":"0.0.1"}; unauthenticated GET /api/v1/hosts/install/public-key currently returns HTTP 404 with code not_found. This matches the older deployed
67dffa58image; the selected eb8ddb3-derived build includes the new route, whose expected unauthenticated response after deployment is 401. No setup request was made.Gate passed at head
9ee1c99997: INFRATERNAL_VERIFY_RUN_ID=deploy-agent-binaries scripts/verify fast, exit 0. No merge gate run. Full output below, verbatim. The supply-chain gate regenerated THIRD_PARTY_NOTICES.md (removed an installed-package-only jiti notice); I restored only that observed generated change because the file was clean at job start and is outside this job ownership. Worktree is clean. Native release passed ELF/glibc checks (maximum GLIBC_2.39); aarch64 same-commit build is still running.Completed deploy-agent-binaries (#50, #49).
Branch: deploy-agent-binaries
Base:
eb8ddb354fHead and deployed commit:
9ee1c99997Commits:
1562e4e(deployment packaging/configuration),9ee1c99(operator documentation).Worktree clean. No push or issue closure. No merge gate run.
Both binaries were built from the same archived commit by deploy/o2/build.sh,
using Rust 1.99.0 in disposable Ubuntu 24.04 containers. ELF architecture checks
passed for x86_64-unknown-linux-gnu and aarch64-unknown-linux-gnu; both require
at most GLIBC_2.39. The actual Debian OCI image passed native --version and
server option parsing, SHA-256 equality against the bundle, UID 10001 and
read/execute/read-only artifact checks before transfer. Production artifact
hashes and permissions match. The server uses the x86_64 artifact via a symlink.
Binary SHA-256:
Final installed Quadlet Exec line:
Deployment command: scripts/deploy-monitor01 --ref
9ee1c99997Deployment exited 0; it retained the provisioned Traefik route, took
/var/lib/infraternal/pre-upgrade-1791127979530415371.sqlite3, and restarted
systemctl --user restart infraternal.service. It did not use podman restart.
First-run setup was untouched. The private baseline already contained one
owner account, one passkey and one consumed setup token; all remained unchanged.
Correction to the earlier commentary: zero remaining unused-token lifetime
did not imply startup would issue a fresh token, because the owner already
exists. No setup request or credential issuance was made by this job. Existing
database inode and all existing identity-file bytes, including enrollment CA,
are retained; SQLite quick_check is ok.
GET /api/v1/hosts/install/public-key changed from 404 on the previous image to
401 unauthenticated on the deployed image. Its authenticated success check was
skipped: it requires bearer authentication plus hosts:create and agents:enroll,
and no operator credential was supplied. No first-run setup was performed to
obtain one.
Reachability limitation: the hostname resolves to 10.69.69.50 on the dev VM
and monitor01. Port 7443 is published on 10.69.69.50 and responds locally on
monitor01 with DNS:infraternal.kayg.org in its TLS certificate. A bounded TCP
connection from the dev VM to infraternal.kayg.org:7443 timed out. Installed
hosts require this network reachability; this result does not establish it
from other hosts. NetBird, DNS, firewall rules, Traefik configuration and other
monitor01 containers were left untouched as instructed.
Required gate: INFRATERNAL_VERIFY_RUN_ID=deploy-agent-binaries scripts/verify fast
Result: PASS, exit 0, 19 stages, total 868.538s. The Bun low-severity cookie
advisory is report-only in the existing gate. The gate's generated notices
change was restored only after confirming the exact generated delta and the
clean starting state; no out-of-scope file was committed.
Deployment output (verbatim):
Production checks and selected nonsecret service log lines (verbatim):
HTTP checks (verbatim):
Retained state checks (verbatim):
Tunnel certificate from monitor01 (verbatim):
Dev VM reachability check (verbatim):
Fast gate output (verbatim):
Primary references checked for Quadlet management/inspection and the cross target: https://docs.podman.io/en/v5.4.2/markdown/podman-systemd.unit.5.html ; https://docs.podman.io/en/latest/markdown/podman-container-inspect.1.html ; https://doc.rust-lang.org/rustc/platform-support/aarch64-unknown-linux-gnu.html
Started fix-deploy-node on branch fix-deploy-node at base head
851d502909. Scope: make scripts/build-web work with the repository's existing JS runtime, inspect deploy/build paths, and run the requested build-only and fast gates. No production deploy will be run.Finding:
deploy/o2/build.shpins Bun 1.4.2 and puts it on PATH, then callsscripts/build-web; that script's final precompression step invokednode, which the disposable Ubuntu build image never installs. I switched that invocation to Bun and documented the shared pinned runtime indocs/DEPLOY-MONITOR01.md. Bun's currentnode:zlibreference marks the module fully supported and exposes bothgzipSyncandbrotliCompressSync: https://bun.sh/reference/node/zlibPath check:
scripts/deploy-o2still exists and uses the samedeploy/o2/build.sh;scripts/verify fastprepares the bundle throughscripts/build-web;scripts/previewalso shares it. No tracked CI/workflow files were present.Local output evidence:
scripts/build-webgenerated 109.gz, 109.br, and 348 ETag metadata files, including_metadata/index.html.identity.etag. A byte check found identity assets and Brotli variants identical to Node output. Bun's gzip bytes differ from Node for 47 variants (and their 47 gzip ETags); the generated ETags match the actual Bun bytes. A decode/hash check verified all 109 gzip and 109 Brotli variants reproduce their source bytes and have matching ETags.Finished fix-deploy-node. Branch:
fix-deploy-node; base:851d50290998f1328b7e5557f9084a5d7d0dff07; head:5af6c2797a7a955baf06eb5884ba4c8dbac0edb8. Commit:fix(deploy): use Bun for web precompression.The web build now invokes
scripts/precompress-web.mjswith the pinned Bun runtime already installed bydeploy/o2/build.sh.docs/DEPLOY-MONITOR01.mdrecords that shared build choice.scripts/deploy-o2is still present and uses this same builder;scripts/verify fastandscripts/previewcall the sharedscripts/build-web. No tracked CI/workflow files were present.rtk scripts/build-webpassed locally.scripts/deploy-monitor01 --build-only --ref HEADpassed for x86_64 and aarch64; both builds emitted 109 gzip and 109 Brotli variants and completed bundle checksums and local OCI image validation. The packaged image HTTP smoke checked identity, Brotli, and gzip bodies against each other and verified each response ETag against its embedded bytes:Fast gate exited 0. Its stage output:
Build-only output:
The fast gate's Bun audit step reported one low
cookie@0.6.0advisory as report-only; it did not fail the gate. No production deployment was run.