Deploy infraternal to o2 at infraternal.kayg.org (NetBird only) #24
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Owner request (2026-09-24): run infraternal on o2 as a systemd service unit, served by the existing Traefik at
infraternal.kayg.org, allowed only from the NetBird mesh10.69.69.0/24. This overrides the PLAN rule that o1/o2 are dogfood targets only after S1; the owner decided it.Facts gathered read-only from o2:
/is 94% full (13 GB free), so nothing is built on o2./dynamic(on the host,/opt/infra/traefik, owned byagents), Let's Encrypt through a Cloudflare DNS challenge, and the networkkayg-edge.kayg(30 services). The rootful Docker socket isroot:docker.infraternal.kayg.orgcurrently resolves to o1 through the*.kayg.orgwildcard.The root steps (the unit, the Traefik file, and the DNS record through the Cloudflare token) need the owner's
sudo.Starting deploy-o2 on branch deploy-o2, base
7bda001e5b. Scope: VM-built deploy bundle only; no connection to o2. I own deploy/o2/ and docs/DEPLOY.md.Starting ops-deploy on branch
deploy-o2, based atd0428e0(current worktree head). The pre-existing installer correction changes the Traefik container name tokayg-traefik; I will commit that first, then fetch and mergeorigin/devbefore the scoped deployment, dependency-helper, hardening, job-tooling, and docs work.Hardening evidence: a temporary system unit rendered from
deploy/o2/infraternal.servicepassed Docker_ping, rootless Podman_ping, and TTY allocation through both engine sockets withPrivateDevices=noand again withPrivateDevices=yes, while retainingProtectSystem=strict,ProtectHome=read-only,PrivateTmp, and the other unit directives. The service template now usesPrivateDevices=yes; its comment records that engine PTYs are allocated through the sockets and host device access is unnecessary. The test uses this VM'skayguid (1001) and cleans up its transient units.The forced health-failure test exposed a rollback defect. In the first run, the SQLite
.backupdump checksum matched the pre-upgrade database, but after replacing only the main database file and restarting, the failed candidate's probe table appeared again. Removing the stopped service's old-wal,-shmand-journalsidecars resolved that replay; the final run verified the restored snapshot against the backup before restart and confirmed the probe table was absent after restart. Rollback health checks now retry while systemd brings the previous server up. The full local Traefik/systemd upgrade and forced-failure rollback test passes and cleans up its test unit, paths and labeled container.Completed ops-deploy on branch
deploy-o2.Head:
de048598789920814e16c8109af2d52db5c7ecdf(feat(deploy): build and roll back o2 upgrades). The branch has five commits ahead oforigin/dev; no push was made.The required gate passed with Cargo output directed to the root-disk target cache because this worktree's existing ignored
target/directory is root-owned. Exact command and output:Other VM evidence:
shellcheck -xandbash -npassed for all 14 scripts. The copiedscripts/jobs/launch-job.shwas checked with SC2155 suppressed to preserve its required byte-for-byte match with/mnt/hdd/infraternal/bin/launch-job.sh.bash scripts/tests/js-dependencies.sh→PASS shared JavaScript dependency freshness.scripts/deploy-o2 --dry-runresolvedorigin/devto7bda001e5bd8ff42c3768fb1e6cbba317d22789eand printed:rtk sudo -n bash scripts/tests/service-hardening.sh→PASS system unit reached Docker and rootless Podman sockets and allocated exec PTYs with PrivateDevices=yes.rtk sudo -n bash scripts/tests/deploy-o2-local.sh /home/kayg/wt/deploy-o2-target/release/infraternalpassed against a labeled local Traefik container and temporary system unit. It forced a health failure after a candidate database write, verified the snapshot before restart, restored the previous binary, checked the service health and confirmed the candidate table was absent. Output included:The test unit, state/backup paths and labeled Traefik container were confirmed removed. Rollback clears failed-database SQLite sidecars before installing the snapshot; SQLite documents WAL as persistent transaction state alongside the main database (SQLite WAL file format).
Exact deploy command for the orchestrator:
scripts/deploy-o2(defaults toorigin/dev). This job did not connect to o2.The #17 stale-dependency fix is in
aa9b575; I added a pointer there to this deploy validation comment.Independent review started. Branch: deploy-o2. Base:
7bda001e5b. Head:de04859878. Scope: read-only review of deployment change; no connection to o2.Independent review of deploy-o2 at
de04859878(base origin/dev). No connection to o2; no source edits.Findings
[blocks-merge] Rollback can erase legitimate writes made after the live snapshot, even when the candidate never ran. upgrade.sh:99-108 takes the .backup while the old service is live; :158-177 and :222-228 restore it on any installer failure. install.sh:127-128, :135-148 have pre-install checks that can fail while the old service continues accepting writes. A failure there stops the old service and replaces its database with an earlier snapshot. SQLite documents .backup as a consistent snapshot, not a freeze of subsequent writes: https://www.sqlite.org/backup.html . Quiesce writes before the snapshot or avoid database restore until a candidate could have migrated it.
[blocks-merge] A failed rollback can still restart on damaged/mismatched data. upgrade.sh:171-185 removes the live -wal/-shm/-journal before mv of the restored database. If mv then fails, the candidate main DB remains without its possibly uncheckpointed WAL; :201-209 restarts even after restore_tmp backup, chown, chmod, sidecar removal, or mv failed. If the candidate migrated the DB, the previous binary can start against it. SQLite warns that WAL files may contain needed transactions: https://www.sqlite.org/wal.html . Keep the service stopped on incomplete restore, and switch the binary/database only after verified replacements are ready.
[blocks-merge] Low disk can prevent the safety backup and leave incomplete backup directories. upgrade.sh:101-108 creates and fills the backup before pruning (:110-123); the 1 GiB free-space check is only in install.sh:127-128, after the backup. On o2's 94%-full root filesystem, .backup or the binary copy can exhaust space; the EXIT trap is not installed until upgrade.sh:222, so a failed backup leaves its partial timestamp directory. Repeated retries can consume more space. Reserve capacity or prune/validate before writing, and clean incomplete backups without touching prior valid snapshots.
[blocks-merge] Uninstall can delete the database without requiring a snapshot. install.sh:189-195 records new_state_dir=true when the directory did not exist at initial install; :93 and :104-105 then recursively delete the whole state directory on a later --uninstall. Once that directory contains the owner's database, its creation history is not a safe deletion authorization. This conflicts with the requirement that the production database never be lost.
[later] Temporary artifacts are not cleaned on every failure. install.sh:180-183 and :198-200 create a binary_tmp but have no EXIT cleanup if install/mv fails; :318-320 creates dns_state.tmp without a failure cleanup. upgrade.sh:170-185 can leave database.rollback.PID after a failed restore. The initial mktemp files are mode 0600 under umask 077 (install.sh:4, upgrade.sh:4), but the binary temp is changed to 0755 during install. Cleanup traps should cover these paths.
[later] Uninstall's manifest ownership check is weaker than install's. install.sh:87-103 accepts any absolute traefik_path ending in /infraternal.yml and unlinks it; install verifies the exact path from the Traefik /dynamic bind mount at :135-147. A stale/corrupt manifest can therefore point uninstall outside the owned Traefik directory. Validate the exact path again before unlinking.
[later] Cloudflare curl calls inherit root's default .curlrc because cf_request uses
curl --configwithout leading-q(install.sh:251-260). A root curl trace/verbose setting can affect the assertion that authorization headers never reach logs. curl documents default config loading and-qto disable it: https://curl.se/docs/tutorial.html . Token discovery, regex validation and mode-0600 curl config (install.sh:226-272) otherwise keep the token out of argv and scripted output; no real token was handled in this review.Verified controls and scope
tar_status=2,tar: Unexpected EOF in archive, andchecksum_status=1. The test did not connect to o2.bash scripts/tests/js-dependencies.sh:PASS shared JavaScript dependency freshness. scripts/lib/js-dependencies.sh:38-71 compares bun.lock hash, records it only after successful install, and returns without bun when all sentinels/hash match; preview sources the same helper at scripts/preview:60-61.sudo bash scripts/tests/service-hardening.sh: baseline and PrivateDevices=yes units each reported PASS Docker socket, rootless Podman socket, Docker exec PTY, and rootless Podman exec PTY; final output:PASS system unit reached Docker and rootless Podman sockets and allocated exec PTYs with PrivateDevices=yes.diff -u /mnt/hdd/infraternal/bin/launch-job.sh scripts/jobs/launch-job.shhad no output. scripts/jobs/README.md:7-11 states openai1 only and /home/kayg/wt. The sole shellcheck warning across the requested shell files is SC2155 at scripts/jobs/launch-job.sh:7, a line unchanged by this diff; shellcheck on deployment, helper, preview and verify scripts passed.Gate output: initial
scripts/verify fastreached Clippy but failed witherror: failed to create directory .../target/debug/Permission denied (os error 13)because this worktree's target is root-owned. Rerun with an external CARGO_TARGET_DIR passed; its exact stdout was:PASS fast: migrations are append-only
PASS fast: JavaScript dependencies
PASS fast: build web app
PASS fast: cargo fmt --check
PASS fast: cargo clippy
PASS fast: cargo test
PASS fast: svelte-check
PASS fast: UI package svelte-check
PASS fast: stream package typecheck
PASS fast: Vitest
PASS fast: UI package Vitest
PASS fast: stream package Vitest
PASS fast: OpenAPI and TypeScript client drift
PASS fast: terminal TypeScript
PASS fast: terminal Vitest
Review head:
de04859878. scripts/tests/deploy-o2-local.sh was not run because no x86_64 release binary is present in this worktree; the requested hardening test was run. Worktree remains clean.Fix round started on branch deploy-o2 at
de04859878. Scope: deploy/, scripts/deploy-o2, scripts/lib/, docs/DEPLOY.md. Read the independent review. All tests stay on the Debian VM using labeled local resources; no connection to o2.Fix round started on branch deploy-o2 at
de04859878. Scope: deploy/, scripts/deploy-o2, scripts/lib/, docs/DEPLOY.md. Read the independent review. All tests stay on the Debian VM using labeled local resources; no connection to o2.Fix decision and evidence:
upgrade.shnow stops the unit before the SQLite snapshot, checks required free space before backup creation, verifies the snapshot's integrity and_sqlx_migrations, and marks candidate start immediately beforesystemctl restart. A failure before that marker restores only the binary; a post-start failure stages and verifies a replacement database before replacing the main file and clearing stale sidecars. Failed restore leaves the unit stopped with printed recovery paths.install.sh --uninstallretains data by default;--delete-datatakes a fresh verified snapshot after stopping the unit. Local systemd tests and gates are in progress. Commits836b3fdand246bb8d.Fix round complete on branch
deploy-o2; head5bee3516b4e1d22cf2715d728b28c9a0af58318f. No push and no connection to o2.Per finding:
systemctl restartsucceeds on aType=execunit; an olderType=simpleunit is rejected before stopping. Pre-start rollback leaves the database untouched, including a local write made after the upgrade was planned. Systemd documents thatType=execwaits for the binary's exec whileType=simplecan report success before it: https://man7.org/linux/man-pages/man5/systemd.service.5.html ._sqlx_migrations, retains the failed main database, atomically renames the verified snapshot, then moves stale WAL/SHM/journal to the backup directory. A simulated restore failure left the local unit stopped, candidate database and snapshot present, and recovery instructions printed. SQLite documents WAL as persistent database state: https://www.sqlite.org/wal.html .--delete-datastops the unit and takes a verified backup before removing state. Both modes passed locally./dynamicbind mount; a forged same-suffix path was rejected in the local test.curl -q, disabling root's.curlrc.Local x86_64 release build output:
Finished release profile [optimized] target(s) in 3m 35s(CARGO_TARGET_DIR=/home/kayg/wt/deploy-o2-target). The labeled local systemd/Docker run reported verbatim:The test unit, binary, state, metadata, backups and labeled Traefik container were removed afterward.
scripts/verify fastoutput, verbatim:All deploy and scripts shell files passed
shellcheck -x -e SC2155; SC2155 is the unchangedscripts/jobs/launch-job.shline 7, preserved byte-for-byte from the job-tooling source.scripts/deploy-o2 --dry-runpassed and printed:Starting verify-deploy on branch at . Fetched at ; merging it before verification. I will not connect to o2.
Starting verify-deploy on branch
deploy-o2at5bee3516b4e1d22cf2715d728b28c9a0af58318f. Fetcheddevat7bda001e5bd8ff42c3768fb1e6cbba317d22789e; merging it before verification. I will not connect to o2.Verification result
Branch
deploy-o2, HEADbbf0d73da882671117f03e5417b5d25ae24d07ff. No connection to o2 was made.PASS — backup while stopped and pre-start failure.
upgrade.shstops the unit before SQLite.backup. The real local harness wrote a row before deployment, forced failure before candidate start, then verified the row and service remained healthy; output:Rollback complete without database restore; candidate never startedandPASS pre-start failure preserved live write.PASS — rollback staging, integrity and recovery. Candidate-start rollback restores the verified snapshot through a staged SQLite backup and same-filesystem rename. The local harness verifies the failed candidate write is absent after rollback; forced restore failure leaves the unit stopped, snapshot/prior binary/database retained and recovery instructions printed. Its next ordinary upgrade resumes the marked restore before installing a candidate. The SIGKILL case below exercised the rename-to-sidecar window.
The first SIGKILL run exposed a real gap: a plain rerun recovered health but the still-live candidate WAL replayed the failed write. Fixed by writing
.rollback-in-progressbefore stopping for candidate rollback, preserving the failed main copy and sidecars, and making reruns re-verify and finish the snapshot restore before a new candidate. After the fix, killing immediately after the snapshot rename left the unit stopped, the previous binary installed, the main DB hash equal to the verified snapshot, the marker/snapshot/failed DB copy retained, and WAL/SHM still present. The candidate probe was visible through that WAL while stopped; the rerun moved the sidecars into the recovery backup, removed the probe, cleared the marker and completed healthy.PASS — space, partial backups and retention. Forced low-space refusal happened before backup creation. SIGTERM and SIGINT during backup removed the partial directory. With 11 timestamp backups seeded, a failed candidate left 12 backups; the next successful upgrade pruned to exactly 10.
PASS — uninstall data handling. Default uninstall retained the database and backups.
--delete-datacreated a fresh*-uninstall-*verified snapshot, then removed the state directory while retaining backups.PASS — temporary files, mount path and curl. Installer temporary binary files and partial backup directories were removed after both SIGTERM and SIGINT. Uninstall rejected a manifest path outside the actual Docker
/dynamicmount. Source review confirmed each token-bearing curl begins with-q.PASS — shared JavaScript helper (#17).
scripts/tests/js-dependencies.shoutput:PASS shared JavaScript dependency freshness; it exercises lockfile-change reinstall, failure retry and unchanged no-network behavior.scripts/previewuses the same helper.PASS — service hardening and launcher match (#17).
service-hardening.shoutput:PASS system unit reached Docker and rootless Podman sockets and allocated exec PTYs with PrivateDevices=yes.cmppassed forscripts/jobs/launch-job.shand/mnt/hdd/infraternal/bin/launch-job.sh.Gates
Shellcheck on the deployment, helper and local test scripts: exit 0, no diagnostics.
scripts/deploy-o2 --dry-runoutput:CARGO_TARGET_DIR=/home/kayg/wt/deploy-o2-target scripts/verify fastoutput:Full local x86_64 upgrade test output included:
The test unit, data, backups, labeled Traefik containers, and temporary SIGKILL script were removed. Commits:
4c47e71andbbf0d73(plus harness setup3fd91e7). No push.Orchestrator: the deploy tooling is merged to
devat9688a24(the merged tree is identical to the verifiedbbf0d73). The path was buildops-deploy, an independent Sol review (4 data-safety blockers), one fix round (fix-deploy), and a verify that found and fixed an interrupted-rollback WAL replay with its ownkill -9scenario. Everything PASS. The first real run ofscripts/deploy-o2against o2 is in progress now, deploying the same app code o2 already runs, as a proof of the tool.Orchestrator: the first real
scripts/deploy-o2run failed in the build container, before touching o2:curl: (35) Recv failure: Connection reset by peerwhile downloadingbun-linux-x64.zipfrom GitHub's latest release. Later, small: pin the bun version (the one the repo uses) and cache the download under/mnt/hdd/infraternal/artifacts/deploy-o2/, so each deploy is reproducible and does not depend on GitHub availability. Also: the orchestrator's shell lacks thedockergroup, so deploys run throughsystemd-run --userwith the absolute script path.Starting issue #24 job in branch deploy-build-fix at base
9688a24(dev). Scope: deploy/o2/build.sh, scripts/deploy-o2, deploy local tests, and docs/DEPLOY.md. I will add pinned, checksum-verified build downloads and reproduce the build-only bundle path without connecting to o2.Finding for issue #24: ip route get 1.1.1.1 selects eth0, whose MTU is 1450; docker0 is 1500. Docker documents com.docker.network.driver.mtu as a per-user-defined-bridge option.
Decision: create a temporary bridge at the detected egress MTU, inspect the applied option, and remove it after the build. This keeps container network isolation and leaves daemon defaults unchanged; host networking would share the host network namespace.
Pins: Bun 1.4.2 checksum 36368faef7527875d5ffa52e53cd48021741f2a83eb6208a8dd64068d422a913 matches the release SHASUMS256.txt; rustup-init 1.29.1 checksum dda7234360b7f578ca8b0ddcb80145646fa61a67c1720a5abc7051b35c9fcb71 matches its published .sha256. Rust toolchain is pinned to 1.98.1.
Added --build-only to assemble and verify a bundle without running any NetBird command. Full real-container build and repeat-cache proof are next.
Real build finding for issue #24: Docker created and inspected infraternal-o2-build-390917 at MTU 1450; curl downloaded the pinned Bun 1.4.2 archive (34.9 MB) from GitHub inside the Ubuntu container and the checksum check passed because the build proceeded into bun install and Rustup.
The AArch64 link failed in openssl-sys with missing bits/libc-header-start.h. Ubuntu listed libc6-dev-arm64-cross as a recommendation, but the builder uses --no-install-recommends. Added libc6-dev-arm64-cross explicitly; retry is using the warmed Bun, Rustup, apt and Cargo caches.
Additional issue #24 finding: the official Ubuntu 24.04 image includes /etc/apt/apt.conf.d/docker-clean, whose post-update and post-install hooks delete /var/cache/apt/archives/*.deb. Despite the archive bind mount, the host cache remained empty and the retry downloaded 130 MB of apt packages again.
Fix: the disposable builder removes that cleanup hook before apt update so package archives can persist under the deploy cache. This does not change the host or Docker daemon; the next final-ref build will verify the retained cache.
Completed the deploy build fix.
Branch:
deploy-build-fixHEAD:
906254dcad07d9f66d44d97e32350fadcef06678The build uses a short-lived user-defined Docker bridge with
com.docker.network.driver.mtuset to the egress interface MTU (1450 here). Docker documents this bridge option for user-defined networks. This keeps the builder isolated, avoids host networking, and does not change daemon configuration. A build inside Ubuntu 24.04 downloaded Bun from GitHub on this network and verified its checksum.Bun 1.4.2 is pinned and its x64 archive hash matches the release
SHASUMS256.txt; rustup 1.29.1 is pinned and its hash matches the versioned official checksum. Rust 1.98.1 is pinned. Downloads and apt archives persist under/mnt/hdd/infraternal/artifacts/deploy-o2/cache/; the apt cache contains 103 archives (125 MB).The final build-only bundle is an AArch64 ELF, has a maximum GLIBC symbol requirement of 2.39, and embeds
_app/immutable. The second build-only run printed:Using verified bundle cache: /mnt/hdd/infraternal/artifacts/deploy-o2/bundles/906254dcad07d9f66d44d97e32350fadcef06678 (no build downloads).scripts/verify fastoutput:Local deploy test output:
ShellCheck,
bash -n, andgit diff --checkpassed. No command connected to o2.Starting deploy unit migration work on branch
deploy-unit-migrate, based on35f73c56efb886f3183e6ce577f10cc88088f909.Started deploy-unit-migrate on branch deploy-unit-migrate at base
35f73c56ef. I am tracing the unit rendering, manifest ownership checks, and local deployment tests before implementing the migration path.Finding: the installed manifest records only
bin_path,unit_path, andtraefik_path; the upgrade's earlyType=execcheck therefore blocks an otherwise owned unit. Commit74b91c3changed the service template fromType=simpletoType=exec; the previous template matches the current template with that one line changed. The migration accepts that exact legacy render when no unit hash exists and adds a SHA-256 baseline for future edit detection.Completed deploy-unit-migrate on branch
deploy-unit-migrate.Head:
08ddb53a7e04a8a84953a0003c899b697a49abacscripts/verify fastoutput:Local deployment test output (x86-64 binary, labelled Traefik container and system unit):
The test also checked that the rendered unit diff showed only
Type=simple→Type=exec, the Traefik file hash stayed unchanged, and no DNS state was created. Shellcheck, Bash syntax, and diff whitespace checks passed.Orchestrator: the first successful
scripts/deploy-o2run: o2 now runsf1bedf9(steps 11b and 12f). The database was backed up with the unit stopped to/var/lib/infraternal-backups/20260924T112546Z/along with the previous binary, it migrated from version 9 to 11, and the account is intact. The direct and HTTPS health checks pass, and the Traefik file and DNS are unchanged.One-time manual step before it. The live unit came from the very first
install.shdraft (d0428e0). The orchestrator verified it byte-identical to that template rendered for o2, so it had no hand edits. The migration check only recognizes the current template withType=execreverted, so it refused. The orchestrator installed the current template atomically (diff:Type=simple→exec,PrivateDevices=no→yes), randaemon-reloadwithout a restart, and kept the old unit at/root/infraternal.service.pre-migrate-20260924-1125. This was the only install from before unit hashes, so it will not recur.