Move production from monitor01 to ssh

kayg commented

2026-10-31 17:45:37 +00:00

Owner

Owner-requested, read-only planning for moving infraternal production from monitor01 (10.69.69.50) to ssh (10.69.69.89). Inventory both hosts and relevant repo history, then post a migration plan with measured sizing, data integrity, service, network, CA, rollback, and verification steps plus owner grill questions. No cutover, deploy, install, or host changes are authorized in this issue.

kayg commented

2026-10-31 17:45:59 +00:00

Author

Owner

Started read-only research in worktree plan/prod-move at base origin/dev ec7b10111e. I will inspect repository history and the two hosts without changing either one, then post the owner plan and grill questions here.

Started read-only research in worktree plan/prod-move at base origin/dev ec7b10111e97a368d13becfca7eb3ac3df0195f6. I will inspect repository history and the two hosts without changing either one, then post the owner plan and grill questions here.

kayg commented

2026-10-31 18:12:30 +00:00

Author

Owner

Read-only findings (2026-10-31, about 23:40 IST)

No host was changed. Secret contents and hashes were not printed or posted; the CA key is referenced only by its path and mode.

Current production: monitor01 (10.69.69.50)

Debian 13, 4 cores, 7.76 GiB RAM, 5.31 GiB available at the sample. infraternal-server.service is active on release 9.5 (0c8639496) as root. The unit uses KillMode=process, starts after network-online and NetBird, and runs /usr/local/bin/infraternal.
HTTP is bound to 172.18.0.1:7011, with --behind-proxy --trusted-proxy 172.18.0.0/16; remote hosts use 10.69.69.50:7443 and the CA directory /var/lib/infraternal/remote-ca. The web bundle is /usr/local/share/infraternal/web.
Persistent state is /var/lib/infraternal (4.6 MiB, 0700 root:root): the SQLite database, CA state, and SSH trust data. The database, CA key (/var/lib/infraternal/remote-ca/active/remote.key), and trust files are 0600 root:root. Canvas documents are held in the SQLite database. /run/infraternal is 0700 root:root and holds volatile session/shim state.
There are six live session scopes. Three shims use a deleted old binary. KillMode=process means stopping only the server does not stop those scopes. The Oct 30 database snapshot has 15 host rows (4 active, 11 revoked) and 68 terminal rows with closed_at unset; those rows do not map one-to-one to the six currently live scopes and need reconciliation before cutover. The four active labels include kayg@10.69.69.125, kayg@o1, root@10.69.69.217, and one unlabeled rootless host. The 10.69.69.125 record is the old control host and was not contacted.
/var/backups/infraternal contains 43 directories / 188 MiB at inspection. Latest is 20261030-113220-0c8639496 (4.6 MiB; SQLite integrity_check was ok). The backup root is 0755; the latest snapshot directory is 0700 and its files are 0600.
infraternal.kayg.org resolves to 37.157.255.66 (TTL 300). Traefik (kayg-traefik) on monitor01 routes public HTTPS to 172.18.0.1:7011; its Cloudflare DNS-01 certificate is valid until 2027-01-02. Public /healthz returned 200. prometheus-node-exporter.service is active on 10.69.69.50:9100.

Memory and sizing

Read-only /proc sample: server plus embedded process = 73.1 MiB RSS / 65.7 MiB PSS; six session shims = 26.7 MiB RSS / 16.3 MiB PSS; all eight processes together = 99.7 MiB RSS / 82.0 MiB PSS.

The 50-shim release 9.5 measurement on #24 reports idle totals of 196.3 MiB RSS / 58.8 MiB PSS / 56.1 MiB USS. One of four 9.5 fixtures had about 17 shims gain 2 MiB USS each while idle (92 MiB total USS rather than 59 MiB); the follow-up remains open. This leaves ample memory for 50+ mostly idle shims on ssh, but does not establish CPU capacity for 50+ active workloads.

New target: ssh (10.69.69.89)

AlmaLinux 10.2, x86_64, 4 cores, 7.68 GiB RAM (7.19 GiB available), 47 GiB free disk.
SELinux is enforcing. Nix, Podman/Docker, nft, iptables, and firewalld are absent; no local firewall service is active. sshd currently listens on all addresses. Interfaces include private 10.69.0.89 and NetBird 10.69.69.89. Outbound public IP observed: 49.205.172.130; that does not prove inbound public routing. No node exporter is running there.

Deployment and code evidence

deploy-prod.sh is absent from this checkout. #254 reconstructs its contract: deploy-prod.sh [--bin x86] [--bundle dir] [--web dist], refuses a SHA without release-prod-live, makes a database backup, and rolls back when the 20-second health check fails. #24 comment 70483 records the latest 9.5 deployment and backup; #24 comment 70351 records the 50-shim measurements. #262 records current production and D316.
The checked-in scripts/release/deploy still targets root@10.69.69.50, uses /usr/local/share/infraternal and /usr/local/bin/infraternal, and checks http://172.18.0.1:7011. docs/operations/deploy-pipeline.md documents /var/backups/infraternal and rollback behavior.
docs/operations/deploy.md says every cold start locks the CA until an administrator unlocks it with a passkey. crates/infraternal/src/remote.rs:205-208 allows the same enrolled host identity and server name to use a changed server endpoint when the stored CA still matches.
D316 leaves proxy-engine selection to kayg after the #261 measurement. This checkout has no Nix flake/package definition, so a pinned Nix package for the signed release and runtime tools is a staging prerequisite under the all-packages-via-Nix rule.

## Read-only findings (2026-10-31, about 23:40 IST) No host was changed. Secret contents and hashes were not printed or posted; the CA key is referenced only by its path and mode. ### Current production: monitor01 (10.69.69.50) - Debian 13, 4 cores, 7.76 GiB RAM, 5.31 GiB available at the sample. infraternal-server.service is active on release 9.5 (0c8639496) as root. The unit uses KillMode=process, starts after network-online and NetBird, and runs /usr/local/bin/infraternal. - HTTP is bound to 172.18.0.1:7011, with --behind-proxy --trusted-proxy 172.18.0.0/16; remote hosts use 10.69.69.50:7443 and the CA directory /var/lib/infraternal/remote-ca. The web bundle is /usr/local/share/infraternal/web. - Persistent state is /var/lib/infraternal (4.6 MiB, 0700 root:root): the SQLite database, CA state, and SSH trust data. The database, CA key (/var/lib/infraternal/remote-ca/active/remote.key), and trust files are 0600 root:root. Canvas documents are held in the SQLite database. /run/infraternal is 0700 root:root and holds volatile session/shim state. - There are six live session scopes. Three shims use a deleted old binary. KillMode=process means stopping only the server does not stop those scopes. The Oct 30 database snapshot has 15 host rows (4 active, 11 revoked) and 68 terminal rows with closed_at unset; those rows do not map one-to-one to the six currently live scopes and need reconciliation before cutover. The four active labels include kayg@10.69.69.125, kayg@o1, root@10.69.69.217, and one unlabeled rootless host. The 10.69.69.125 record is the old control host and was not contacted. - /var/backups/infraternal contains 43 directories / 188 MiB at inspection. Latest is 20261030-113220-0c8639496 (4.6 MiB; SQLite integrity_check was ok). The backup root is 0755; the latest snapshot directory is 0700 and its files are 0600. - infraternal.kayg.org resolves to 37.157.255.66 (TTL 300). Traefik (kayg-traefik) on monitor01 routes public HTTPS to 172.18.0.1:7011; its Cloudflare DNS-01 certificate is valid until 2027-01-02. Public /healthz returned 200. prometheus-node-exporter.service is active on 10.69.69.50:9100. ### Memory and sizing Read-only /proc sample: server plus embedded process = 73.1 MiB RSS / 65.7 MiB PSS; six session shims = 26.7 MiB RSS / 16.3 MiB PSS; all eight processes together = 99.7 MiB RSS / 82.0 MiB PSS. The 50-shim release 9.5 measurement on #24 reports idle totals of 196.3 MiB RSS / 58.8 MiB PSS / 56.1 MiB USS. One of four 9.5 fixtures had about 17 shims gain 2 MiB USS each while idle (92 MiB total USS rather than 59 MiB); the follow-up remains open. This leaves ample memory for 50+ mostly idle shims on ssh, but does not establish CPU capacity for 50+ active workloads. ### New target: ssh (10.69.69.89) - AlmaLinux 10.2, x86_64, 4 cores, 7.68 GiB RAM (7.19 GiB available), 47 GiB free disk. - SELinux is enforcing. Nix, Podman/Docker, nft, iptables, and firewalld are absent; no local firewall service is active. sshd currently listens on all addresses. Interfaces include private 10.69.0.89 and NetBird 10.69.69.89. Outbound public IP observed: 49.205.172.130; that does not prove inbound public routing. No node exporter is running there. ### Deployment and code evidence - deploy-prod.sh is absent from this checkout. #254 reconstructs its contract: deploy-prod.sh <sha> [--bin x86] [--bundle dir] [--web dist], refuses a SHA without release-prod-live, makes a database backup, and rolls back when the 20-second health check fails. #24 comment 70483 records the latest 9.5 deployment and backup; #24 comment 70351 records the 50-shim measurements. #262 records current production and D316. - The checked-in scripts/release/deploy still targets root@10.69.69.50, uses /usr/local/share/infraternal and /usr/local/bin/infraternal, and checks http://172.18.0.1:7011. docs/operations/deploy-pipeline.md documents /var/backups/infraternal and rollback behavior. - docs/operations/deploy.md says every cold start locks the CA until an administrator unlocks it with a passkey. crates/infraternal/src/remote.rs:205-208 allows the same enrolled host identity and server name to use a changed server endpoint when the stored CA still matches. - D316 leaves proxy-engine selection to kayg after the #261 measurement. This checkout has no Nix flake/package definition, so a pinned Nix package for the signed release and runtime tools is a staging prerequisite under the all-packages-via-Nix rule.

kayg commented

2026-10-31 18:12:31 +00:00

Author

Owner

Recommended migration plan (owner decision required)

Recommendation: move the server process and persistent data to ssh, while keeping monitor01 only as a short-lived Traefik edge and rollback point. Leave infraternal.kayg.org DNS and TLS untouched in this first phase. The new VM has no confirmed inbound public path, and D316 leaves the eventual proxy engine to kayg after #261. This makes ssh the production server host without pretending the public edge has moved.

Owner grill

Public edge
- Recommendation: temporarily keep Traefik on monitor01 and change only its backend to the new server after testing reachability from the Traefik container to 10.69.69.89:7011 over NetBird. Keep DNS 37.157.255.66 and the existing certificate.
- Question: is that temporary edge bridge acceptable, or should ssh receive public ingress in the same window? If direct ingress is required, what inbound 80/443 path should be used? D316 says the proxy engine stays an owner choice after #261 measurement.
Nix, service, SELinux, and firewall
- Recommendation: after the go, bootstrap multi-user Nix; add a pinned Nix package for the exact signed 9.5 binary and web bundle plus runtime tools (there is no flake in this checkout); use an explicit root infraternal-server.service whose ExecStart resolves to the immutable Nix store. Keep mutable state at /var/lib/infraternal, runtime state at /run/infraternal, and new backups at /var/backups/infraternal (0700 root:root).
- Keep SELinux enforcing. Persist standard state labels (var_lib_t for /var/lib/infraternal, var_run_t for /run/infraternal) with semanage fcontext/restorecon; use an Alma-supported Nix store executable type or a narrow local service policy, then check for AVC denials after cold start. Do not use permissive mode or broad labeling.
- Use a Nix-managed nftables policy with default-deny inbound: SSH only via the management/NetBird path; TCP and UDP 7443 only on NetBird; TCP 7011 only from the measured Traefik source on NetBird; no public 80/443 until the edge decision. Verify access from a second SSH session before applying any firewall policy because sshd currently binds all addresses.
- Question: approve Nix packaging as a prerequisite issue before host prep? The host package rule is clear, but this repo currently has no Nix expression to install the signed release reproducibly.
Agents and Mesh identity
- Recommendation: copy the same CA and preserve server name/identity; update only the endpoint for the four active host records to 10.69.69.89:7443. Leave the 11 revoked records untouched. Verify the 10.69.69.125 control-host record with kayg before deciding whether it remains active; do not contact that host as part of this task.
- Question: should the product Mesh node keep its current identity and only change its display name to ssh, or should it retain monitor01 until a later cleanup? The checked-in re-enrollment contract supports an endpoint change without rotating identity or CA.
Live sessions
- Recommendation: accept that all six current browser terminal attachments disconnect when the old server stops; their PTYs cannot move to the new host. Reserve about 15–20 minutes for UI/API/terminal unavailability. Keep the old scopes on monitor01 for the 24-hour rollback hold (KillMode=process leaves them running); they will be unreachable from ssh, and commands in those shells may continue. Retire the scopes after the hold.
- Question: is that disposition acceptable, or should kayg explicitly stop the six old scopes at cutover and interrupt any commands still running?
Window and owner availability
- Recommendation: Nov 1, 2026, 09:00–11:00 IST: 09:00–10:00 stage Nix/service/SELinux/firewall and prove the proxy route using a separate scratch data root; 10:00–10:05 final preflight and backup; about 10:05–10:25 outage/copy/start; 10:25–10:45 verification; 10:45–11:00 rollback reserve. If the go arrives after 09:00, slide the window by the staging time instead of compressing checks. Kayg must be present at new-service cold start to unlock the CA with the owner passkey.
- Question: approve that window and the 24-hour rollback hold, or name a different IST window?

Runbook after approval

Pre-stage without production data. Bootstrap Nix and a pinned package for the exact signed 9.5 (0c8639496) binary/web artifacts and needed support tools. Install the root systemd unit with After=network-online.target netbird.service, Restart=on-failure, and KillMode=process. Bind HTTP to 10.69.69.89:7011, set --public-origin https://infraternal.kayg.org, --behind-proxy, --trusted-proxy to the measured single proxy source, --remote-listen 10.69.69.89:7443, and --remote-ca-dir /var/lib/infraternal/remote-ca. Keep /var/lib/infraternal 0700, database/CA/trust files 0600; create /run/infraternal 0700. Test the unit with a separate empty scratch database and high ports only. Confirm the Nix store executable and writable data/runtime paths under enforcing SELinux; apply persistent file-context rules and check AVC logs. Stage firewall rules and prove SSH still works via NetBird. Test proxy-container routing before maintenance; if that path fails, stop and ask kayg before choosing another edge arrangement.
Freeze and make the final copy. Announce maintenance and record current active hosts/scopes. Make a final SQLite-consistent backup on monitor01, verify its integrity, then stop infraternal-server.service. Do not start the new instance until the old one is stopped. Keep monitor01's data and its current backup unchanged for rollback. Copy /var/lib/infraternal over NetBird, preserving numeric owner and modes; exclude /run and sockets. Confirm equal bytes with private pass/fail comparisons (do not print a CA-key hash), verify the copied database with PRAGMA integrity_check = ok, compare the public CA fingerprint, check root ownership/modes, and run restorecon on the destination. The dataset is only 4.6 MiB, so the transfer itself should take under a minute; allow 5–10 minutes for snapshot, copy, and checks.
Start on ssh and unlock. Start the new service only after the copy checks pass. Kayg unlocks the CA with the passkey after the cold start. Verify the CA status is unlocked, the process uses the exact signed release, /healthz returns 200 on the NetBird bind, and no SELinux denials appear.
Route and reconnect. Point monitor01 Traefik's backend to http://10.69.69.89:7011 only after the container-to-NetBird path is proven; keep public DNS/TLS unchanged. Repoint the four active host agents to 10.69.69.89:7443 while retaining the same host IDs, server name, and CA; verify each reconnects. Install/enable the node exporter through Nix on ssh and update the existing scrape target from 10.69.69.50:9100 to 10.69.69.89:9100. Do not change revoked records. Keep the old server and edge available for a 24-hour rollback period.
Verification before accepting the move. Check systemd active state/restart count and executable digest; internal and public /healthz; public TLS name/expiry; kayg passkey sign-in and unlocked CA status; all four active agents online; opening a new terminal and attaching to it; reading existing canvas documents; the release-suite smoke; node-exporter scrape; firewall source restrictions; SELinux still enforcing with no relevant AVC denials. Existing six terminals are not expected to reappear on ssh.
Rollback if any check fails by 11:00 IST. Stop the new service. If it accepted any writes, take a consistent target backup and copy the updated database/CA state back to monitor01 while both servers are stopped; verify integrity and permissions. Start the original monitor01 service, have kayg unlock its cold CA, route Traefik back to 172.18.0.1:7011, and return active agents to 10.69.69.50:7443. Never run two writable servers with the same identity/CA. Retain both backup trees until kayg accepts the 24-hour hold.

Deployment script changes

Rebuild the missing control-host deploy-prod.sh from #254's documented contract, and align it with scripts/release/deploy:

Target root@10.69.69.89 over NetBird, with strict host-key checking.
Keep signed-release and release-prod-live admission; do not build on the target.
Make each new target backup under /var/backups/infraternal/- on ssh, with root-only mode, before replacement. Keep the old monitor01 backups in place for rollback.
Replace the old remote /usr/local/bin, /usr/local/share, and 172.18.0.1:7011 assumptions with the pinned Nix store/profile paths, the new unit, and http://10.69.69.89:7011/healthz; retain the public health check through https://infraternal.kayg.org/healthz.
Preserve automatic rollback on failed health/release identity checks and preserve the existing persistent data path /var/lib/infraternal. Add a preflight for enforcing SELinux, NetBird address, and target backup permissions.

If the public edge cannot reach ssh from Traefik's container, do not cut over until kayg chooses an alternate ingress. The direct inbound route for ssh is not established by its observed outbound public IP.

## Recommended migration plan (owner decision required) **Recommendation:** move the server process and persistent data to ssh, while keeping monitor01 only as a short-lived Traefik edge and rollback point. Leave infraternal.kayg.org DNS and TLS untouched in this first phase. The new VM has no confirmed inbound public path, and D316 leaves the eventual proxy engine to kayg after #261. This makes ssh the production server host without pretending the public edge has moved. ### Owner grill 1. **Public edge** - **Recommendation:** temporarily keep Traefik on monitor01 and change only its backend to the new server after testing reachability from the Traefik container to 10.69.69.89:7011 over NetBird. Keep DNS 37.157.255.66 and the existing certificate. - **Question:** is that temporary edge bridge acceptable, or should ssh receive public ingress in the same window? If direct ingress is required, what inbound 80/443 path should be used? D316 says the proxy engine stays an owner choice after #261 measurement. 2. **Nix, service, SELinux, and firewall** - **Recommendation:** after the go, bootstrap multi-user Nix; add a pinned Nix package for the exact signed 9.5 binary and web bundle plus runtime tools (there is no flake in this checkout); use an explicit root infraternal-server.service whose ExecStart resolves to the immutable Nix store. Keep mutable state at /var/lib/infraternal, runtime state at /run/infraternal, and new backups at /var/backups/infraternal (0700 root:root). - Keep SELinux enforcing. Persist standard state labels (var_lib_t for /var/lib/infraternal, var_run_t for /run/infraternal) with semanage fcontext/restorecon; use an Alma-supported Nix store executable type or a narrow local service policy, then check for AVC denials after cold start. Do not use permissive mode or broad labeling. - Use a Nix-managed nftables policy with default-deny inbound: SSH only via the management/NetBird path; TCP and UDP 7443 only on NetBird; TCP 7011 only from the measured Traefik source on NetBird; no public 80/443 until the edge decision. Verify access from a second SSH session before applying any firewall policy because sshd currently binds all addresses. - **Question:** approve Nix packaging as a prerequisite issue before host prep? The host package rule is clear, but this repo currently has no Nix expression to install the signed release reproducibly. 3. **Agents and Mesh identity** - **Recommendation:** copy the same CA and preserve server name/identity; update only the endpoint for the four active host records to 10.69.69.89:7443. Leave the 11 revoked records untouched. Verify the 10.69.69.125 control-host record with kayg before deciding whether it remains active; do not contact that host as part of this task. - **Question:** should the product Mesh node keep its current identity and only change its display name to ssh, or should it retain monitor01 until a later cleanup? The checked-in re-enrollment contract supports an endpoint change without rotating identity or CA. 4. **Live sessions** - **Recommendation:** accept that all six current browser terminal attachments disconnect when the old server stops; their PTYs cannot move to the new host. Reserve about 15–20 minutes for UI/API/terminal unavailability. Keep the old scopes on monitor01 for the 24-hour rollback hold (KillMode=process leaves them running); they will be unreachable from ssh, and commands in those shells may continue. Retire the scopes after the hold. - **Question:** is that disposition acceptable, or should kayg explicitly stop the six old scopes at cutover and interrupt any commands still running? 5. **Window and owner availability** - **Recommendation:** Nov 1, 2026, 09:00–11:00 IST: 09:00–10:00 stage Nix/service/SELinux/firewall and prove the proxy route using a separate scratch data root; 10:00–10:05 final preflight and backup; about 10:05–10:25 outage/copy/start; 10:25–10:45 verification; 10:45–11:00 rollback reserve. If the go arrives after 09:00, slide the window by the staging time instead of compressing checks. Kayg must be present at new-service cold start to unlock the CA with the owner passkey. - **Question:** approve that window and the 24-hour rollback hold, or name a different IST window? ### Runbook after approval 1. **Pre-stage without production data.** Bootstrap Nix and a pinned package for the exact signed 9.5 (0c8639496) binary/web artifacts and needed support tools. Install the root systemd unit with After=network-online.target netbird.service, Restart=on-failure, and KillMode=process. Bind HTTP to 10.69.69.89:7011, set --public-origin https://infraternal.kayg.org, --behind-proxy, --trusted-proxy to the measured single proxy source, --remote-listen 10.69.69.89:7443, and --remote-ca-dir /var/lib/infraternal/remote-ca. Keep /var/lib/infraternal 0700, database/CA/trust files 0600; create /run/infraternal 0700. Test the unit with a separate empty scratch database and high ports only. Confirm the Nix store executable and writable data/runtime paths under enforcing SELinux; apply persistent file-context rules and check AVC logs. Stage firewall rules and prove SSH still works via NetBird. Test proxy-container routing before maintenance; if that path fails, stop and ask kayg before choosing another edge arrangement. 2. **Freeze and make the final copy.** Announce maintenance and record current active hosts/scopes. Make a final SQLite-consistent backup on monitor01, verify its integrity, then stop infraternal-server.service. Do not start the new instance until the old one is stopped. Keep monitor01's data and its current backup unchanged for rollback. Copy /var/lib/infraternal over NetBird, preserving numeric owner and modes; exclude /run and sockets. Confirm equal bytes with private pass/fail comparisons (do not print a CA-key hash), verify the copied database with PRAGMA integrity_check = ok, compare the public CA fingerprint, check root ownership/modes, and run restorecon on the destination. The dataset is only 4.6 MiB, so the transfer itself should take under a minute; allow 5–10 minutes for snapshot, copy, and checks. 3. **Start on ssh and unlock.** Start the new service only after the copy checks pass. Kayg unlocks the CA with the passkey after the cold start. Verify the CA status is unlocked, the process uses the exact signed release, /healthz returns 200 on the NetBird bind, and no SELinux denials appear. 4. **Route and reconnect.** Point monitor01 Traefik's backend to http://10.69.69.89:7011 only after the container-to-NetBird path is proven; keep public DNS/TLS unchanged. Repoint the four active host agents to 10.69.69.89:7443 while retaining the same host IDs, server name, and CA; verify each reconnects. Install/enable the node exporter through Nix on ssh and update the existing scrape target from 10.69.69.50:9100 to 10.69.69.89:9100. Do not change revoked records. Keep the old server and edge available for a 24-hour rollback period. 5. **Verification before accepting the move.** Check systemd active state/restart count and executable digest; internal and public /healthz; public TLS name/expiry; kayg passkey sign-in and unlocked CA status; all four active agents online; opening a new terminal and attaching to it; reading existing canvas documents; the release-suite smoke; node-exporter scrape; firewall source restrictions; SELinux still enforcing with no relevant AVC denials. Existing six terminals are not expected to reappear on ssh. 6. **Rollback if any check fails by 11:00 IST.** Stop the new service. If it accepted any writes, take a consistent target backup and copy the updated database/CA state back to monitor01 while both servers are stopped; verify integrity and permissions. Start the original monitor01 service, have kayg unlock its cold CA, route Traefik back to 172.18.0.1:7011, and return active agents to 10.69.69.50:7443. Never run two writable servers with the same identity/CA. Retain both backup trees until kayg accepts the 24-hour hold. ### Deployment script changes Rebuild the missing control-host deploy-prod.sh from #254's documented contract, and align it with scripts/release/deploy: - Target root@10.69.69.89 over NetBird, with strict host-key checking. - Keep signed-release and release-prod-live admission; do not build on the target. - Make each new target backup under /var/backups/infraternal/<timestamp>-<sha> on ssh, with root-only mode, before replacement. Keep the old monitor01 backups in place for rollback. - Replace the old remote /usr/local/bin, /usr/local/share, and 172.18.0.1:7011 assumptions with the pinned Nix store/profile paths, the new unit, and http://10.69.69.89:7011/healthz; retain the public health check through https://infraternal.kayg.org/healthz. - Preserve automatic rollback on failed health/release identity checks and preserve the existing persistent data path /var/lib/infraternal. Add a preflight for enforcing SELinux, NetBird address, and target backup permissions. If the public edge cannot reach ssh from Traefik's container, do not cut over until kayg chooses an alternate ingress. The direct inbound route for ssh is not established by its observed outbound public IP.

kayg commented

2026-10-31 18:12:31 +00:00

Author

Owner

Finished research-only work at worktree plan/prod-move, head ec7b10111e (the requested origin/dev base). No tracked or untracked files changed; neither host was modified. Findings and the recommended plan with owner grill questions are posted above.

Gate output: scripts/verify fast was not run because this task was planning-only and made no source changes. No builds, tests, or browser work were run.

Finished research-only work at worktree plan/prod-move, head ec7b10111e97a368d13becfca7eb3ac3df0195f6 (the requested origin/dev base). No tracked or untracked files changed; neither host was modified. Findings and the recommended plan with owner grill questions are posted above. Gate output: scripts/verify fast was not run because this task was planning-only and made no source changes. No builds, tests, or browser work were run.

kayg commented

2026-10-31 18:16:59 +00:00

Author

Owner

J27 started: branch nix-package, base nix-devshell at 72afb3edbd24761dfffd65957d5416093fe72e21 (advanced from the brief’s 380179a60). Worktree /home/kayg/Developer/Worktrees/infraternal-nix-package. Adding a reproducible Nix package for the static release bundle and tracer web assets; builds/comparisons only on compute01 through build-slot. No production host access or changes.

J27 started: branch `nix-package`, base `nix-devshell` at `72afb3edbd24761dfffd65957d5416093fe72e21` (advanced from the brief’s 380179a60). Worktree `/home/kayg/Developer/Worktrees/infraternal-nix-package`. Adding a reproducible Nix package for the static release bundle and tracer web assets; builds/comparisons only on compute01 through build-slot. No production host access or changes.

kayg commented

2026-10-31 18:26:41 +00:00

Author

Owner

J27 packaging findings:

scripts/package-release produces five executables: server for the selected architecture, full host payloads for x86_64/aarch64, and session-only payloads for x86_64/aarch64. bootstrap_runtime.rs and release_payload.rs discover payloads beside the running executable; the Nix layout preserves that directory.
scripts/release/dev_builder.py adds the apps/tracer production web output and SOURCE/VERSION/GATE/checksums. GATE and signed MANIFEST/signature are release admission records, not values a source package may invent. The package supplies SOURCE/VERSION/checksums and documents the signing/admission boundary.
apps/tracer/build.ts embeds wall-clock build time and obtains HEAD through git, so its current normal output is not reproducible. Added explicit optional build-date/commit inputs; ordinary builds retain their existing defaults. Nix derives these inputs from the source revision/time.
Both x86_64-linux and aarch64-linux package derivations evaluate successfully on compute01. No architecture build is claimed yet. Dependency build plus fresh fast-gate/reference commands are queued through build-slot; all four shared slots are currently busy.

No production host was contacted or changed.

J27 packaging findings: - `scripts/package-release` produces five executables: server for the selected architecture, full host payloads for x86_64/aarch64, and session-only payloads for x86_64/aarch64. `bootstrap_runtime.rs` and `release_payload.rs` discover payloads beside the running executable; the Nix layout preserves that directory. - `scripts/release/dev_builder.py` adds the `apps/tracer` production web output and SOURCE/VERSION/GATE/checksums. GATE and signed MANIFEST/signature are release admission records, not values a source package may invent. The package supplies SOURCE/VERSION/checksums and documents the signing/admission boundary. - `apps/tracer/build.ts` embeds wall-clock build time and obtains HEAD through git, so its current normal output is not reproducible. Added explicit optional build-date/commit inputs; ordinary builds retain their existing defaults. Nix derives these inputs from the source revision/time. - Both x86_64-linux and aarch64-linux package derivations evaluate successfully on compute01. No architecture build is claimed yet. Dependency build plus fresh fast-gate/reference commands are queued through build-slot; all four shared slots are currently busy. No production host was contacted or changed.

kayg commented

2026-10-31 18:32:28 +00:00

Author

Owner

J27 service boundary finding: on compute01, stat reports /nix/store as 1775 root:nixbld (/nix is 0755 root:root). crates/infraternal/src/restart_binary.rs:134-142 rejects group-writable executable ancestry and marks it unprotected for warm handoff. A direct store path therefore does not establish signed warm-replacement eligibility, even after avoiding the bin/infraternal symlink. This source package also has no signed MANIFEST/signature by design.

The docs-only systemd example explicitly supports cold stop/start and retained-output rollback. It does not add ExecReload or weaken the security validator. #274's eventual production update/admission path needs to account for this existing boundary; no host permission changes were attempted.

J27 service boundary finding: on compute01, `stat` reports `/nix/store` as `1775 root:nixbld` (`/nix` is 0755 root:root). `crates/infraternal/src/restart_binary.rs:134-142` rejects group-writable executable ancestry and marks it unprotected for warm handoff. A direct store path therefore does not establish signed warm-replacement eligibility, even after avoiding the `bin/infraternal` symlink. This source package also has no signed MANIFEST/signature by design. The docs-only systemd example explicitly supports cold stop/start and retained-output rollback. It does not add ExecReload or weaken the security validator. #274's eventual production update/admission path needs to account for this existing boundary; no host permission changes were attempted.

kayg commented

2026-10-31 18:41:12 +00:00

Author

Owner

J27 base correction before any build ran: the local nix-devshell branch was stale at 72afb3edb, while origin/nix-devshell contains the brief's requested 380179a602d6af185674971b7ae090137fd70ed5. Reset only my nix-package branch to that intended base and reapplied the packaging changes. The compute01 source is updated to the same base; both build-slot jobs are still queued, so no results were produced against the stale base. The earlier start comment's base is superseded by 380179a60. This includes #264's reviewed ARM fixture/tool portability changes unchanged.

J27 base correction before any build ran: the local `nix-devshell` branch was stale at 72afb3edb, while `origin/nix-devshell` contains the brief's requested `380179a602d6af185674971b7ae090137fd70ed5`. Reset only my `nix-package` branch to that intended base and reapplied the packaging changes. The compute01 source is updated to the same base; both build-slot jobs are still queued, so no results were produced against the stale base. The earlier start comment's base is superseded by 380179a60. This includes #264's reviewed ARM fixture/tool portability changes unchanged.

kayg commented

2026-10-31 18:46:12 +00:00

Author

Owner

J27 validation is waiting for compute01 capacity: the dependency build has spent about 27 minutes in ~/infraternal-ops/build-slot; the separate fast-gate/reference build about 21 minutes. Both logs still contain only all 4 slots busy, waiting.... Free disk is approximately 118 GiB, so this is slot occupancy, not the disk-floor wait. No job-owned compile has started, no target directory exists yet, and no checksums or passing gate are claimed. Source work is ready on the requested 380179a60 base; I am leaving both jobs queued under the mandatory wrapper, without changing slot configuration or touching other jobs.

J27 validation is waiting for compute01 capacity: the dependency build has spent about 27 minutes in `~/infraternal-ops/build-slot`; the separate fast-gate/reference build about 21 minutes. Both logs still contain only `all 4 slots busy, waiting...`. Free disk is approximately 118 GiB, so this is slot occupancy, not the disk-floor wait. No job-owned compile has started, no target directory exists yet, and no checksums or passing gate are claimed. Source work is ready on the requested 380179a60 base; I am leaving both jobs queued under the mandatory wrapper, without changing slot configuration or touching other jobs.

kayg commented

2026-10-31 19:22:59 +00:00

Author

Owner

J27 normal reference bundle completed on compute01 through build-slot at source base 380179a60 plus the packaging working tree (no Rust source/profile changes). Independent readelf inspection confirms all five have no PT_INTERP; SHA256SUMS verifies.

normal artifact	bytes	sha256
infraternal	21,431,472	b527c33d9df30659c816b31546c9583b05c85bae87fa7b95eec4e207d589ea3c
infraternal-x86_64-musl	10,121,376	2176ca2af1a3d66711e1a6b9dc200d678f40e7ce32fa6046ba741cffc27086c8
infraternal-aarch64-musl	9,512,056	521c14d33d3650ef5f02783622cfacaa8441f186661917ec7baf8f2784c3a935
infraternal-x86_64-session-musl	1,863,592	58238b490c27ad69f319111e906318d4719458ad2bd7e24b25fc0ed1d02d2ed2
infraternal-aarch64-session-musl	1,622,400	de6e2d24164b96327269d0238b1ccb9b7fee63947f592e70e008ca2fd4d357f0

Found a dev-shell omission: package-release invokes rg at line 43, but rg was missing. Its conditional pipeline prints command-not-found and still allows exit 0. Added pkgs.ripgrep to this branch's dev shell (the package derivation already explicitly included it), and independently inspected the actual reference ELF headers rather than trusting that failed-open check. No release helper or security validator was changed. Logs: ~/j27-evidence/reference-build.log and reference-checks.txt.

First Nix dependency attempt fetched the exact Zig/native archives successfully and measured Bun's initial tree; native unpacking failed because fetchurl's output name lacked an archive extension. The derivation now gives it a .tar.gz name. Also omitted unused optional native peer tooling from the Bun fetch so its fixed-output tree can be shared across the two Linux architectures. These changes await the queued retry; final hash pins and Nix equivalence are still outstanding.

The first fast-gate attempt stopped at repository metadata (scratch checkout lacked origin/dev); that setup fault is corrected. A fresh full gate is queued with the real origin/dev reference, private target, both existing short-state-root overrides, default SIGINT, and debug-symbol/incremental suppression for disposable test artifacts. Release optimization and assertions are unchanged. No passing full gate or Nix byte equivalence is claimed yet.

J27 normal reference bundle completed on compute01 through build-slot at source base 380179a60 plus the packaging working tree (no Rust source/profile changes). Independent readelf inspection confirms all five have no PT_INTERP; SHA256SUMS verifies. | normal artifact | bytes | sha256 | |---|---:|---| | infraternal | 21,431,472 | b527c33d9df30659c816b31546c9583b05c85bae87fa7b95eec4e207d589ea3c | | infraternal-x86_64-musl | 10,121,376 | 2176ca2af1a3d66711e1a6b9dc200d678f40e7ce32fa6046ba741cffc27086c8 | | infraternal-aarch64-musl | 9,512,056 | 521c14d33d3650ef5f02783622cfacaa8441f186661917ec7baf8f2784c3a935 | | infraternal-x86_64-session-musl | 1,863,592 | 58238b490c27ad69f319111e906318d4719458ad2bd7e24b25fc0ed1d02d2ed2 | | infraternal-aarch64-session-musl | 1,622,400 | de6e2d24164b96327269d0238b1ccb9b7fee63947f592e70e008ca2fd4d357f0 | Found a dev-shell omission: package-release invokes rg at line 43, but rg was missing. Its conditional pipeline prints command-not-found and still allows exit 0. Added pkgs.ripgrep to this branch's dev shell (the package derivation already explicitly included it), and independently inspected the actual reference ELF headers rather than trusting that failed-open check. No release helper or security validator was changed. Logs: ~/j27-evidence/reference-build.log and reference-checks.txt. First Nix dependency attempt fetched the exact Zig/native archives successfully and measured Bun's initial tree; native unpacking failed because fetchurl's output name lacked an archive extension. The derivation now gives it a .tar.gz name. Also omitted unused optional native peer tooling from the Bun fetch so its fixed-output tree can be shared across the two Linux architectures. These changes await the queued retry; final hash pins and Nix equivalence are still outstanding. The first fast-gate attempt stopped at repository metadata (scratch checkout lacked origin/dev); that setup fault is corrected. A fresh full gate is queued with the real origin/dev reference, private target, both existing short-state-root overrides, default SIGINT, and debug-symbol/incremental suppression for disposable test artifacts. Release optimization and assertions are unchanged. No passing full gate or Nix byte equivalence is claimed yet.

reference-checks.txt

968 B

kayg commented

2026-10-31 19:45:15 +00:00

Author

Owner

J27 crash-safety checkpoint: pushed nix-package at f50f1e381 (WIP, not build-qualified). Static reference build is complete and matches D243; Nix dependency hash acquisition and the fresh fast gate are still queued through compute01 build-slot. Two fake hashes deliberately remain pending measurement. Logs: /home/claude/j27-evidence/{package-batch.log,deps-current.log,full-gate-launch.log,reference-checks.txt}; working source /home/claude/worktrees/nix-package, control worktree ~/Developer/Worktrees/infraternal-nix-package. No production host touched. Next: pin measured dependency hashes, compare all five binaries and web assets, finish gate, then replace WIP with qualified atomic commit.

J27 crash-safety checkpoint: pushed `nix-package` at `f50f1e381` (WIP, not build-qualified). Static reference build is complete and matches D243; Nix dependency hash acquisition and the fresh fast gate are still queued through compute01 build-slot. Two fake hashes deliberately remain pending measurement. Logs: `/home/claude/j27-evidence/{package-batch.log,deps-current.log,full-gate-launch.log,reference-checks.txt}`; working source `/home/claude/worktrees/nix-package`, control worktree `~/Developer/Worktrees/infraternal-nix-package`. No production host touched. Next: pin measured dependency hashes, compare all five binaries and web assets, finish gate, then replace WIP with qualified atomic commit.

kayg commented

2026-10-31 20:11:33 +00:00

Author

Owner

J27 progress checkpoint: nix-package pushed at 76f5c0ae5 (still WIP). Added the full binary/web byte-comparison procedure. Package dependency acquisition has waited roughly 69 minutes and the fresh fast gate roughly 50 minutes for compute01 build-slot admission; neither has run. Current slot holders are active, not stale locks. Free disk has recovered to 85 GiB; this job keeps one 6 GiB target. No new qualification claim, and no production host touched. Evidence and continuation paths remain as recorded above.

J27 progress checkpoint: `nix-package` pushed at `76f5c0ae5` (still WIP). Added the full binary/web byte-comparison procedure. Package dependency acquisition has waited roughly 69 minutes and the fresh fast gate roughly 50 minutes for compute01 build-slot admission; neither has run. Current slot holders are active, not stale locks. Free disk has recovered to 85 GiB; this job keeps one 6 GiB target. No new qualification claim, and no production host touched. Evidence and continuation paths remain as recorded above.

kayg commented

2026-10-31 20:21:09 +00:00

Author

Owner

J27: after 77 minutes queued, the package job obtained a compute01 slot. Both fixed-output dependency derivations completed and produced measured recursive hashes: Bun sha256-o9hhp0aYmmnLsfo1u9ZaA81ldO8hnF1ZSbhrkyB5d4Y=, native Zig package cache sha256-Zz3ldGMFajbjnsXyJ1OGtvYRj1FmvyF/Gm3sGYhcLJc=. Both are now pinned; no fake hashes remain. The sandboxed full package build is running through the same slot. Fast gate is still queued. Dependency logs: /home/claude/j27-evidence/deps-current.log; package log: /home/claude/j27-evidence/nix-build.log. The reference static-linkage/checksum receipt is attached to the earlier comparison comment.

J27: after 77 minutes queued, the package job obtained a compute01 slot. Both fixed-output dependency derivations completed and produced measured recursive hashes: Bun `sha256-o9hhp0aYmmnLsfo1u9ZaA81ldO8hnF1ZSbhrkyB5d4Y=`, native Zig package cache `sha256-Zz3ldGMFajbjnsXyJ1OGtvYRj1FmvyF/Gm3sGYhcLJc=`. Both are now pinned; no fake hashes remain. The sandboxed full package build is running through the same slot. Fast gate is still queued. Dependency logs: `/home/claude/j27-evidence/deps-current.log`; package log: `/home/claude/j27-evidence/nix-build.log`. The reference static-linkage/checksum receipt is attached to the earlier comparison comment.

kayg commented

2026-10-31 20:25:59 +00:00

Author

Owner

J27 sandbox finding: the first full package build failed in the native terminal build because Zig fetch-only mode seeded non-lazy sources but omitted dependencies requested by the actual build graph (including oniguruma and the upstream graph's other lazy packages). Cargo vendoring and Zig/archive pins worked; sandbox DNS blocked the attempted native downloads, as intended. Evidence: /home/claude/j27-evidence/nix-build.log, failing derivation mn8wxbrrr6sqds37m2k7rv59wrdsyzvs-infraternal-static-0.1.0. Fix under test: the fixed-output native dependency derivation runs the selected upstream native build graph, discards every compiled object, and retains only package sources. Its cache hash must be measured again; the earlier native cache hash is insufficient. New batch queued through build-slot: PID 3196022; logs package-retry.log, deps-v3.log, nix-build-v3-*.log. No shipping native recipe, features, compiler flags or size gates changed.

J27 sandbox finding: the first full package build failed in the native terminal build because Zig fetch-only mode seeded non-lazy sources but omitted dependencies requested by the actual build graph (including oniguruma and the upstream graph's other lazy packages). Cargo vendoring and Zig/archive pins worked; sandbox DNS blocked the attempted native downloads, as intended. Evidence: `/home/claude/j27-evidence/nix-build.log`, failing derivation `mn8wxbrrr6sqds37m2k7rv59wrdsyzvs-infraternal-static-0.1.0`. Fix under test: the fixed-output native dependency derivation runs the selected upstream native build graph, discards every compiled object, and retains only package sources. Its cache hash must be measured again; the earlier native cache hash is insufficient. New batch queued through build-slot: PID 3196022; logs `package-retry.log`, `deps-v3.log`, `nix-build-v3-*.log`. No shipping native recipe, features, compiler flags or size gates changed.

nix-build-incomplete-cache.log

160 KiB

kayg commented

2026-10-31 20:42:06 +00:00

Author

Owner

J27 checkpoint: pushed native lazy-dependency fix as WIP a0b74157c. Its replacement fixed-output hash remains deliberately pending measurement; retry PID 3196022 is queued through build-slot. The fresh fast gate obtained slot 2 at 02:02:08 IST and is running (/home/claude/j27-evidence/verify-fast.log); separate scripts/test-package-release.py passed. Failed offline native-cache log is attached to the finding above. No production host touched.

J27 checkpoint: pushed native lazy-dependency fix as WIP `a0b74157c`. Its replacement fixed-output hash remains deliberately pending measurement; retry PID 3196022 is queued through build-slot. The fresh fast gate obtained slot 2 at 02:02:08 IST and is running (`/home/claude/j27-evidence/verify-fast.log`); separate `scripts/test-package-release.py` passed. Failed offline native-cache log is attached to the finding above. No production host touched.

kayg commented

2026-10-31 21:22:59 +00:00

Author

Owner

J27 interim verification: scripts/verify fast exited 0 on compute01. The complete unmodified output is attached; excerpts below are verbatim. The separate release-bundle fixture also passed. The final clean package commit will receive its own gate run.

+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
x86_64-unknown-linux-musl: exact define removed; build directories remapped; explicit paths preserved; target override refused
aarch64-unknown-linux-musl: exact define removed; build directories remapped; explicit paths preserved; target override refused
PT_INTERP: absent; size: 21431472 bytes; cap: 21431472 bytes; delta: +0 bytes
key memory: x86_64-unknown-linux-musl; memlock=8388608 bytes
PT_INTERP: absent; size: 19605896 bytes; cap: 19605896 bytes; delta: +0 bytes
key memory: aarch64-unknown-linux-musl runtime not checked (host x86_64; native execution required)
x86_64-unknown-linux-musl: exact 21431472-byte cap PASS; 21431473 bytes rejected PASS; smaller artifact requires ratchet PASS
aarch64-unknown-linux-musl: exact 19605896-byte cap PASS; 19605897 bytes rejected PASS; smaller artifact requires ratchet PASS

The new native dependency cache measured sha256-BfU58ccEGY7L08XD3VPCr05VWHyDxQ+kb+LDQiCC2l0=. The actual graph also exposed the stock compiler's /usr/bin/env host ABI probe inside the sandbox. Keep the shipping compiler executable/LLVM 20.1.2 unchanged; adapt only its build-library host discovery and explicit native generator queries. Cross-musl flags stay on the shipping branch of the build library. Native compatibility retry queued through build-slot as PID 3886534; outer log package-v4-launch.log. No production host touched.

J27 interim verification: `scripts/verify fast` exited **0** on compute01. The complete unmodified output is attached; excerpts below are verbatim. The separate release-bundle fixture also passed. The final clean package commit will receive its own gate run. ```text + | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | static builds | PASS | +---------------------------+--------------+ x86_64-unknown-linux-musl: exact define removed; build directories remapped; explicit paths preserved; target override refused aarch64-unknown-linux-musl: exact define removed; build directories remapped; explicit paths preserved; target override refused PT_INTERP: absent; size: 21431472 bytes; cap: 21431472 bytes; delta: +0 bytes key memory: x86_64-unknown-linux-musl; memlock=8388608 bytes PT_INTERP: absent; size: 19605896 bytes; cap: 19605896 bytes; delta: +0 bytes key memory: aarch64-unknown-linux-musl runtime not checked (host x86_64; native execution required) x86_64-unknown-linux-musl: exact 21431472-byte cap PASS; 21431473 bytes rejected PASS; smaller artifact requires ratchet PASS aarch64-unknown-linux-musl: exact 19605896-byte cap PASS; 19605897 bytes rejected PASS; smaller artifact requires ratchet PASS ``` The new native dependency cache measured `sha256-BfU58ccEGY7L08XD3VPCr05VWHyDxQ+kb+LDQiCC2l0=`. The actual graph also exposed the stock compiler's `/usr/bin/env` host ABI probe inside the sandbox. Keep the shipping compiler executable/LLVM 20.1.2 unchanged; adapt only its build-library host discovery and explicit native generator queries. Cross-musl flags stay on the shipping branch of the build library. Native compatibility retry queued through build-slot as PID 3886534; outer log `package-v4-launch.log`. No production host touched.

verify-fast.log

502 KiB

package-release-test.log

398 B

native-graph-stock-compiler.log

27 KiB

kayg commented

2026-10-31 21:51:20 +00:00

Author

Owner

J27 checkpoint: nix-package is pushed at 5d8460b2a (WIP; native compatibility/build comparison still pending). No uncommitted source work remains. Updated package attributes evaluate for both Linux architectures. The stock compiler sandbox-probe log is now attached to the interim gate comment above, alongside the full passing fast-gate output.

Retry PID 3886534 has waited about 30 minutes through compute01 build-slot; outer log /home/claude/j27-evidence/package-v4-launch.log. All four inspected slot holders had active compiler/test descendants; no other job or lock touched. Free disk has recovered to 95 GiB. Continue from /tmp/infraternal-j27/CONTINUE.md on control; the retry waits at most 120 seconds at each handoff. Next: qualify native-host build-library adaptation with the untouched archive compiler, compare all five payloads and complete web tree at the final clean commit, run the final head gate, then cleanup. No production access.

J27 checkpoint: `nix-package` is pushed at `5d8460b2a` (WIP; native compatibility/build comparison still pending). No uncommitted source work remains. Updated package attributes evaluate for both Linux architectures. The stock compiler sandbox-probe log is now attached to the interim gate comment above, alongside the full passing fast-gate output. Retry PID 3886534 has waited about 30 minutes through compute01 build-slot; outer log `/home/claude/j27-evidence/package-v4-launch.log`. All four inspected slot holders had active compiler/test descendants; no other job or lock touched. Free disk has recovered to 95 GiB. Continue from `/tmp/infraternal-j27/CONTINUE.md` on control; the retry waits at most 120 seconds at each handoff. Next: qualify native-host build-library adaptation with the untouched archive compiler, compare all five payloads and complete web tree at the final clean commit, run the final head gate, then cleanup. No production access.

kayg commented

2026-10-31 22:23:34 +00:00

Author

Owner

J27 capacity checkpoint: unchanged pushed WIP head 5d8460b2a; no uncommitted source work. Native compatibility retry PID 3886534 has now waited about 62 minutes for compute01 build-slot admission. Current slot holders were inspected read-only and had active compiler/test descendants, including newly admitted work; no locks or other jobs changed. Free disk is 68 GiB. The current wrapper has no FIFO queue, so admission order can differ from arrival.

The interim full fast gate passed (complete verbatim output attached above), and normal reference binaries match exact D243 size. Updated Nix package/build-library compatibility and the final byte comparison remain unverified until this retry runs. Continue from /tmp/infraternal-j27/CONTINUE.md; outer log /home/claude/j27-evidence/package-v4-launch.log. All source and evidence are preserved; no production access.

J27 capacity checkpoint: unchanged pushed WIP head `5d8460b2a`; no uncommitted source work. Native compatibility retry PID 3886534 has now waited about 62 minutes for compute01 build-slot admission. Current slot holders were inspected read-only and had active compiler/test descendants, including newly admitted work; no locks or other jobs changed. Free disk is 68 GiB. The current wrapper has no FIFO queue, so admission order can differ from arrival. The interim full fast gate passed (complete verbatim output attached above), and normal reference binaries match exact D243 size. Updated Nix package/build-library compatibility and the final byte comparison remain unverified until this retry runs. Continue from `/tmp/infraternal-j27/CONTINUE.md`; outer log `/home/claude/j27-evidence/package-v4-launch.log`. All source and evidence are preserved; no production access.

kayg commented

2026-10-31 22:42:56 +00:00

Author

Owner

J27 capacity finding: after confirming PID 3886534 was still queued for over 80 minutes, stopped only that own waiter and refreshed the build-slot invocation. The VM helper has since changed to FIFO tickets (5-second polling, oldest live ticket only), so the fresh invocation uses the updated fairness rule. New PID 671083, outer log /home/claude/j27-evidence/package-v5-launch.log, initial queue position 8. No other job, lock, shared helper or capacity limit changed by J27. Source remains pushed WIP 5d8460b2a; native compatibility and byte comparison still pending. Admission-time private guard also refuses heavy work below the existing 25 GiB free-disk floor.

J27 capacity finding: after confirming PID 3886534 was still queued for over 80 minutes, stopped only that own waiter and refreshed the build-slot invocation. The VM helper has since changed to FIFO tickets (5-second polling, oldest live ticket only), so the fresh invocation uses the updated fairness rule. New PID **671083**, outer log `/home/claude/j27-evidence/package-v5-launch.log`, initial queue position **8**. No other job, lock, shared helper or capacity limit changed by J27. Source remains pushed WIP `5d8460b2a`; native compatibility and byte comparison still pending. Admission-time private guard also refuses heavy work below the existing 25 GiB free-disk floor.

kayg commented

2026-10-31 23:17:33 +00:00

Author

Owner

J27 checkpoint d2366379c pushed on nix-package. The Nix daemon did not inherit build-slot client niceness: its builder was nice 0. Stopped only my first package client and retried with builder-local lowering to nice 10 (preserving stricter existing priority). Verified the new nixbld builder, package-release, Cargo and rustc processes at nice 10. No daemon settings, shared limits, or production hosts changed. Build 2 is active in the existing allocated slot; log ~/j27-evidence/nix-build-v3-2.log. The interim fast gate passed; final-head comparison and gate remain pending. One own target: /home/claude/.cache/codex-scratch/j27/target, to be removed at finish.

J27 checkpoint `d2366379c` pushed on `nix-package`. The Nix daemon did not inherit build-slot client niceness: its builder was nice 0. Stopped only my first package client and retried with builder-local lowering to nice 10 (preserving stricter existing priority). Verified the new nixbld builder, package-release, Cargo and rustc processes at nice 10. No daemon settings, shared limits, or production hosts changed. Build 2 is active in the existing allocated slot; log `~/j27-evidence/nix-build-v3-2.log`. The interim fast gate passed; final-head comparison and gate remain pending. One own target: `/home/claude/.cache/codex-scratch/j27/target`, to be removed at finish.

kayg commented

2026-10-31 23:40:52 +00:00

Author

Owner

J27 first complete Nix package build succeeded. Both session payloads are byte-identical; all five sizes match the normal build exactly (server 21,431,472 B, delta 0). The server and full host payloads had hash differences. First differing string is OpenSSL build date: Nix package explicitly set SOURCE_DATE_EPOCH=0 (1970), while normal nix develop exports 315532800 (1980). Measured both environment and embedded strings. Aligning the package epoch with the normal dev shell, then rebuilding to establish whether this explains all differences. Clean consolidated candidate 774403508 is local pending qualification; latest pushed crash checkpoint remains d2366379c. New allocated-slot request PID1460331, logs ~/j27-evidence/epoch-launch.log and nix-epoch.log; old wrapper671083 has exited. No production access.

J27 first complete Nix package build succeeded. Both session payloads are byte-identical; all five sizes match the normal build exactly (server 21,431,472 B, delta 0). The server and full host payloads had hash differences. First differing string is OpenSSL build date: Nix package explicitly set SOURCE_DATE_EPOCH=0 (1970), while normal `nix develop` exports 315532800 (1980). Measured both environment and embedded strings. Aligning the package epoch with the normal dev shell, then rebuilding to establish whether this explains all differences. Clean consolidated candidate `774403508` is local pending qualification; latest pushed crash checkpoint remains `d2366379c`. New allocated-slot request PID1460331, logs `~/j27-evidence/epoch-launch.log` and `nix-epoch.log`; old wrapper671083 has exited. No production access.

reference-checks.txt

968 B

comparison-initial.md

1.1 KiB

kayg commented

2026-10-31 23:45:00 +00:00

Author

Owner

Candidate 77440350841cce0f6a6748326064e6d5e135815e is now pushed on nix-package as a pending-qualification crash checkpoint. It consolidates the WIP history into one logical package change; the source tree includes the measured 1980 epoch correction. Own wrapper1460331 is queued through build-slot (initial position11), nice10. No final success is claimed. The first complete comparison is attached to the preceding finding; all sizes match and both session hashes match. Next: corrected package build, full same-commit web/binary comparison, final fast gate, cleanup.

Candidate `77440350841cce0f6a6748326064e6d5e135815e` is now pushed on `nix-package` as a pending-qualification crash checkpoint. It consolidates the WIP history into one logical package change; the source tree includes the measured 1980 epoch correction. Own wrapper1460331 is queued through build-slot (initial position11), nice10. No final success is claimed. The first complete comparison is attached to the preceding finding; all sizes match and both session hashes match. Next: corrected package build, full same-commit web/binary comparison, final fast gate, cleanup.

kayg commented

2026-11-01 00:00:53 +00:00

Author

Owner

J27 candidate 77440350841cce0f6a6748326064e6d5e135815e remains pushed, pending final qualification. The epoch-corrected rebuild has been admitted through build-slot (wrapper1460331, slot1/4) after FIFO wait. Active log ~/j27-evidence/nix-epoch.log. The same allocated slot now automatically compares all five executables, switches only this job checkout to the exact committed candidate if tracked content matches, then runs full same-commit bundle/web comparison, packaged key-memory/runtime closure checks and final scripts/verify fast. Any mismatch stops the sequence. One normal target remains /home/claude/.cache/codex-scratch/j27/target; keep it until J27 cleanup. No production access.

J27 candidate `77440350841cce0f6a6748326064e6d5e135815e` remains pushed, pending final qualification. The epoch-corrected rebuild has been admitted through build-slot (wrapper1460331, slot1/4) after FIFO wait. Active log `~/j27-evidence/nix-epoch.log`. The same allocated slot now automatically compares all five executables, switches only this job checkout to the exact committed candidate if tracked content matches, then runs full same-commit bundle/web comparison, packaged key-memory/runtime closure checks and final `scripts/verify fast`. Any mismatch stops the sequence. One normal target remains `/home/claude/.cache/codex-scratch/j27/target`; keep it until J27 cleanup. No production access.

kayg commented

2026-11-01 00:11:46 +00:00

Author

Owner

J27 30-minute checkpoint: branch nix-package still at pushed candidate 77440350841cce0f6a6748326064e6d5e135815e, no further source edits. Corrected sandbox build is active in the existing slot; full server completed in 10m03s and host payloads are compiling. Verified actual nixbld builder/rustc nice10. Wrapper1460331 remains live; current log ~/j27-evidence/nix-epoch.log. Final verification is automatic in the same slot after exact preliminary binary comparison. External final-verify.sh clears comparison-only web metadata before the ordinary final fast gate. One normal target ~/.cache/codex-scratch/j27/target remains needed, then will be deleted. No production access.

J27 30-minute checkpoint: branch `nix-package` still at pushed candidate `77440350841cce0f6a6748326064e6d5e135815e`, no further source edits. Corrected sandbox build is active in the existing slot; full server completed in 10m03s and host payloads are compiling. Verified actual nixbld builder/rustc nice10. Wrapper1460331 remains live; current log `~/j27-evidence/nix-epoch.log`. Final verification is automatic in the same slot after exact preliminary binary comparison. External final-verify.sh clears comparison-only web metadata before the ordinary final fast gate. One normal target `~/.cache/codex-scratch/j27/target` remains needed, then will be deleted. No production access.

kayg commented

2026-11-01 00:25:38 +00:00

Author

Owner

J27 clean candidate 77440350841cce0f6a6748326064e6d5e135815e comparison succeeded. Both builds use the same committed tree and pinned Nix dev shell; the ordinary build uses unmodified scripts/package-release and frozen Bun check/build. All five executables and all 284 web files (including precompressed assets) are byte-identical. The epoch correction accounts for every native difference from the first build. Web commit/date inputs are aligned as documented. Nix adds SOURCE/VERSION and expands SHA256SUMS to all bundle files; the reference assembles those records with the same documented schema. Admission/signing records remain external.

artifact	normal bytes	nix bytes	normal sha256	nix sha256	identical
infraternal	21,431,472	21,431,472	`b527c33d9df30659c816b31546c9583b05c85bae87fa7b95eec4e207d589ea3c`	`b527c33d9df30659c816b31546c9583b05c85bae87fa7b95eec4e207d589ea3c`	True
infraternal-x86_64-musl	10,121,376	10,121,376	`2176ca2af1a3d66711e1a6b9dc200d678f40e7ce32fa6046ba741cffc27086c8`	`2176ca2af1a3d66711e1a6b9dc200d678f40e7ce32fa6046ba741cffc27086c8`	True
infraternal-aarch64-musl	9,512,056	9,512,056	`521c14d33d3650ef5f02783622cfacaa8441f186661917ec7baf8f2784c3a935`	`521c14d33d3650ef5f02783622cfacaa8441f186661917ec7baf8f2784c3a935`	True
infraternal-x86_64-session-musl	1,863,592	1,863,592	`58238b490c27ad69f319111e906318d4719458ad2bd7e24b25fc0ed1d02d2ed2`	`58238b490c27ad69f319111e906318d4719458ad2bd7e24b25fc0ed1d02d2ed2`	True
infraternal-aarch64-session-musl	1,622,400	1,622,400	`de6e2d24164b96327269d0238b1ccb9b7fee63947f592e70e008ca2fd4d357f0`	`de6e2d24164b96327269d0238b1ccb9b7fee63947f592e70e008ca2fd4d357f0`	True

web: 284 files; 0 differences: []

package SHA256SUMS: PASS; all five executables: PT_INTERP absent

Raw Zig compiler executable also compares identical to the original checksum-pinned archive. Packaged key-memory check under 8 MiB memlock: Key memory initialization=1; locked pages and dump exclusion enabled. Runtime closure contains only the package itself: 48,141,272 bytes. Both Linux attributes evaluate; the native aarch64 package was not built, while its host/session payloads were actually cross-built and compared above.

Final scripts/verify fast is now running on this clean commit in the same allocated slot. No final gate success claimed yet. No production access. Target/result cleanup follows the gate.

J27 clean candidate `77440350841cce0f6a6748326064e6d5e135815e` comparison succeeded. Both builds use the same committed tree and pinned Nix dev shell; the ordinary build uses unmodified `scripts/package-release` and frozen Bun check/build. All five executables and all 284 web files (including precompressed assets) are byte-identical. The epoch correction accounts for every native difference from the first build. Web commit/date inputs are aligned as documented. Nix adds SOURCE/VERSION and expands SHA256SUMS to all bundle files; the reference assembles those records with the same documented schema. Admission/signing records remain external. | artifact | normal bytes | nix bytes | normal sha256 | nix sha256 | identical | |---|---:|---:|---|---|---| | infraternal | 21,431,472 | 21,431,472 | `b527c33d9df30659c816b31546c9583b05c85bae87fa7b95eec4e207d589ea3c` | `b527c33d9df30659c816b31546c9583b05c85bae87fa7b95eec4e207d589ea3c` | True | | infraternal-x86_64-musl | 10,121,376 | 10,121,376 | `2176ca2af1a3d66711e1a6b9dc200d678f40e7ce32fa6046ba741cffc27086c8` | `2176ca2af1a3d66711e1a6b9dc200d678f40e7ce32fa6046ba741cffc27086c8` | True | | infraternal-aarch64-musl | 9,512,056 | 9,512,056 | `521c14d33d3650ef5f02783622cfacaa8441f186661917ec7baf8f2784c3a935` | `521c14d33d3650ef5f02783622cfacaa8441f186661917ec7baf8f2784c3a935` | True | | infraternal-x86_64-session-musl | 1,863,592 | 1,863,592 | `58238b490c27ad69f319111e906318d4719458ad2bd7e24b25fc0ed1d02d2ed2` | `58238b490c27ad69f319111e906318d4719458ad2bd7e24b25fc0ed1d02d2ed2` | True | | infraternal-aarch64-session-musl | 1,622,400 | 1,622,400 | `de6e2d24164b96327269d0238b1ccb9b7fee63947f592e70e008ca2fd4d357f0` | `de6e2d24164b96327269d0238b1ccb9b7fee63947f592e70e008ca2fd4d357f0` | True | web: 284 files; 0 differences: [] package SHA256SUMS: PASS; all five executables: PT_INTERP absent Raw Zig compiler executable also compares identical to the original checksum-pinned archive. Packaged key-memory check under 8 MiB memlock: `Key memory initialization=1; locked pages and dump exclusion enabled`. Runtime closure contains only the package itself: 48,141,272 bytes. Both Linux attributes evaluate; the native aarch64 package was not built, while its host/session payloads were actually cross-built and compared above. Final `scripts/verify fast` is now running on this clean commit in the same allocated slot. No final gate success claimed yet. No production access. Target/result cleanup follows the gate.

comparison.md

1.2 KiB

zig-compiler-comparison.txt

267 B

runtime-closure.txt

136 B

kayg commented

2026-11-01 00:40:39 +00:00

Author

Owner

J27 final-head gate completed nonzero: Rust test101, specifically faithful_snapshot_preserves_public_mouse_encoder_reports. Static builds/linkage/key-memory/sizes, fmt, clippy, cargo-deny and other completed rows passed. The probe's retained mouse-build.log identifies the cause: fatal error: 'ghostty/vt.h' file not found.

This came from my earlier private native-cache trim, not a product-source change: I retained installed archives/headers but removed ghostty-source, and scripts/verify-vt-native.py explicitly derives its include directory from that source subtree. Those probe/test files are unchanged from unmodified origin/dev ec7b10111. A new single-slot retry will run the baseline probe against the same missing include tree, restore only source include directories from the existing SHA-256-verified pinned archive, rerun the mouse fixture, then rerun the full final-head fast gate. No code or gate weakening is involved.

Only own new wrapper2140012 is queued/running through build-slot; old1460331 is confirmed exited. Logs ~/j27-evidence/gate-retry-launch.log, header-restoration.log, mouse-retry.log; failed full transcript preserved as verify-fast-final-missing-headers.log. All complete package/native/web comparison receipts remain valid and unchanged at pushed head77440350841cce0f6a6748326064e6d5e135815e. One normal target still needed at ~/.cache/codex-scratch/j27/target, then cleanup.

J27 final-head gate completed nonzero: Rust test101, specifically `faithful_snapshot_preserves_public_mouse_encoder_reports`. Static builds/linkage/key-memory/sizes, fmt, clippy, cargo-deny and other completed rows passed. The probe's retained `mouse-build.log` identifies the cause: `fatal error: 'ghostty/vt.h' file not found`. This came from my earlier private native-cache trim, not a product-source change: I retained installed archives/headers but removed `ghostty-source`, and `scripts/verify-vt-native.py` explicitly derives its include directory from that source subtree. Those probe/test files are unchanged from unmodified origin/dev `ec7b10111`. A new single-slot retry will run the baseline probe against the same missing include tree, restore only source include directories from the existing SHA-256-verified pinned archive, rerun the mouse fixture, then rerun the full final-head fast gate. No code or gate weakening is involved. Only own new wrapper2140012 is queued/running through build-slot; old1460331 is confirmed exited. Logs `~/j27-evidence/gate-retry-launch.log`, `header-restoration.log`, `mouse-retry.log`; failed full transcript preserved as `verify-fast-final-missing-headers.log`. All complete package/native/web comparison receipts remain valid and unchanged at pushed head77440350841cce0f6a6748326064e6d5e135815e. One normal target still needed at `~/.cache/codex-scratch/j27/target`, then cleanup.

verify-fast-final-missing-headers.log

392 KiB

missing-header-build.log

183 B

kayg commented

2026-11-01 01:09:45 +00:00

Author

Owner

J27 30-minute checkpoint: pushed clean head remains 77440350841cce0f6a6748326064e6d5e135815e, with no further repository changes. Complete native/web comparison, packaged key-memory and self-only runtime closure remain PASS (receipts on comment72115). Final gate retry wrapper2140012 is still FIFO queued, now position7 after about29minutes; log ~/j27-evidence/gate-retry-launch.log. It will prove the missing-header prerequisite failure with unmodified dev probe sources, restore only headers from the checksum-pinned archive, run the mouse fixture and then the full fast gate. The previous failure and raw transcript are on comment72145. Checked that the previous run left zero own legacy test processes. Exactly one normal target ~/.cache/codex-scratch/j27/target is retained for this pending retry; target/result/reference cleanup follows final qualification. No production access, no other jobs touched.

J27 30-minute checkpoint: pushed clean head remains `77440350841cce0f6a6748326064e6d5e135815e`, with no further repository changes. Complete native/web comparison, packaged key-memory and self-only runtime closure remain PASS (receipts on comment72115). Final gate retry wrapper2140012 is still FIFO queued, now position7 after about29minutes; log `~/j27-evidence/gate-retry-launch.log`. It will prove the missing-header prerequisite failure with unmodified dev probe sources, restore only headers from the checksum-pinned archive, run the mouse fixture and then the full fast gate. The previous failure and raw transcript are on comment72145. Checked that the previous run left zero own legacy test processes. Exactly one normal target `~/.cache/codex-scratch/j27/target` is retained for this pending retry; target/result/reference cleanup follows final qualification. No production access, no other jobs touched.

kayg commented

2026-11-01 01:26:17 +00:00

Author

Owner

J27 retry admitted in the existing FIFO request (wrapper2140012, slot2/4 after about45minutes). Root cause confirmed: the unmodified origin/dev ec7b10111 C probe reproduces the missing ghostty/vt.h prerequisite failure against the same retained native archive (exit1). Verified pinned source archive SHA-256 fb4b2f9ffa0af125983041fdbe4ef94d3fa79fb9f2d22b9c213c0e3847a866b6, restored nine include directories, and the focused mouse fixture now passes: 1 passed, 0 failed. Final-head release-bundle fixture also PASS.

The full scripts/verify fast retry is now running on clean pushed head77440350841cce0f6a6748326064e6d5e135815e, same slot and one normal target. Current raw log ~/j27-evidence/verify-fast-final.log is the fresh retry, not the old failed log (preserved separately). No repository edits, no production access. Cleanup follows successful final qualification.

J27 retry admitted in the existing FIFO request (wrapper2140012, slot2/4 after about45minutes). Root cause confirmed: the unmodified origin/dev ec7b10111 C probe reproduces the missing `ghostty/vt.h` prerequisite failure against the same retained native archive (exit1). Verified pinned source archive SHA-256 `fb4b2f9ffa0af125983041fdbe4ef94d3fa79fb9f2d22b9c213c0e3847a866b6`, restored nine include directories, and the focused mouse fixture now passes: 1 passed, 0 failed. Final-head release-bundle fixture also PASS. The full `scripts/verify fast` retry is now running on clean pushed head77440350841cce0f6a6748326064e6d5e135815e, same slot and one normal target. Current raw log `~/j27-evidence/verify-fast-final.log` is the fresh retry, not the old failed log (preserved separately). No repository edits, no production access. Cleanup follows successful final qualification.

header-restoration.log

367 B

mouse-retry.log

449 B

kayg commented

2026-11-01 01:55:02 +00:00

Author

Owner

J27 finished. Branch nix-package, pushed head 77440350841cce0f6a6748326064e6d5e135815e, direct parent 380179a602d6af185674971b7ae090137fd70ed5 from #264. Control and VM worktrees are clean; remote branch matches this head.

Added packages.x86_64-linux.default and packages.aarch64-linux.default, using the locked Rust toolchain, Cargo.lock vendoring, checksum-pinned original Zig compiler and fixed-output native/Bun dependency trees. The package calls the existing shipping scripts, disables executable fixups, and includes the five release executables and production web assets. Documentation includes reproduction instructions, the external signing boundary, and a cold systemd example. No apps or NixOS module, no production access or changes.

Same clean-commit comparison on compute01, through build-slot:

artifact	normal bytes	nix bytes	normal sha256	nix sha256	identical
infraternal	21,431,472	21,431,472	`b527c33d9df30659c816b31546c9583b05c85bae87fa7b95eec4e207d589ea3c`	`b527c33d9df30659c816b31546c9583b05c85bae87fa7b95eec4e207d589ea3c`	True
infraternal-x86_64-musl	10,121,376	10,121,376	`2176ca2af1a3d66711e1a6b9dc200d678f40e7ce32fa6046ba741cffc27086c8`	`2176ca2af1a3d66711e1a6b9dc200d678f40e7ce32fa6046ba741cffc27086c8`	True
infraternal-aarch64-musl	9,512,056	9,512,056	`521c14d33d3650ef5f02783622cfacaa8441f186661917ec7baf8f2784c3a935`	`521c14d33d3650ef5f02783622cfacaa8441f186661917ec7baf8f2784c3a935`	True
infraternal-x86_64-session-musl	1,863,592	1,863,592	`58238b490c27ad69f319111e906318d4719458ad2bd7e24b25fc0ed1d02d2ed2`	`58238b490c27ad69f319111e906318d4719458ad2bd7e24b25fc0ed1d02d2ed2`	True
infraternal-aarch64-session-musl	1,622,400	1,622,400	`de6e2d24164b96327269d0238b1ccb9b7fee63947f592e70e008ca2fd4d357f0`	`de6e2d24164b96327269d0238b1ccb9b7fee63947f592e70e008ca2fd4d357f0`	True

web: 284 files; 0 differences: []

package SHA256SUMS: PASS; all five executables: PT_INTERP absent

All native sizes have zero growth. The raw Zig compiler executable is also byte-identical to the original pinned archive (SHA-256 2858dc89dbbfdd08cceda1b841e7fd0a793a1a67b49f150bc3d0d1de44ed7f51). Packaged key-memory check passes with 8 MiB memlock. Runtime closure contains only the package itself, 48,141,272 bytes.

Assembly differences from a plain source scripts/package-release directory are documented: Nix installs under share/infraternal with a bin symlink, adds SOURCE/workspace VERSION, and expands SHA256SUMS to cover metadata and web files. Normal web build revision/date inputs were aligned with the flake source; the reference assembles those records with the same schema. The default remains an unsigned source build. Release identity override uses the existing release inputs; GATE/MANIFEST/signatures and production admission remain external.

Both Linux package attributes evaluate on this head. The native aarch64 package was not built; its full host/session payloads were actually cross-built and compared above.

Final scripts/verify fast passed on this exact clean head, through build-slot, with CARGO_BUILD_JOBS=4, private short state roots, and debug/test symbols disabled as in #264 (release flags unchanged). The first final attempt failed because my earlier cache trim removed the source include tree required by the mouse probe. Unmodified dev probe sources reproduced that missing-header prerequisite failure; restoring nine include directories from the verified pinned archive made the focused mouse test pass, then this full retry passed. No repository change or gate weakening was needed. The final release-bundle fixture also passed.

Verbatim final gate summaries (complete raw transcript attached):

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+

PT_INTERP: absent; size: 21431472 bytes; cap: 21431472 bytes; delta: +0 bytes
PT_INTERP: absent; size: 19605896 bytes; cap: 19605896 bytes; delta: +0 bytes

 2 pass
 0 fail
 121 expect() calls
Ran 2 tests across 1 file. [585.00ms]

 363 pass
 0 fail
 21695 expect() calls
Ran 363 tests across 60 files. [11.09s]

 105 pass
 0 fail
 413 expect() calls
Ran 105 tests across 13 files. [275.00ms]

 17 pass
 0 fail
 1794 expect() calls
Ran 17 tests across 4 files. [324.00ms]

 39 pass
 0 fail
 50325 expect() calls
Ran 39 tests across 10 files. [4.12s]

Driver output, verbatim:

FINAL_VERIFY_EXIT=0

Cleanup verified: own build runner exited, zero remaining own test processes; both target directories, result symlink, temporary binary reference bundles/probe output, and private test state removed. Small logs/receipts are retained outside the checkout. Cleanup receipt attached. No blockers or new owner decisions for this implementation; #274 still retains its migration/admission decisions.

J27 finished. Branch `nix-package`, pushed head `77440350841cce0f6a6748326064e6d5e135815e`, direct parent `380179a602d6af185674971b7ae090137fd70ed5` from #264. Control and VM worktrees are clean; remote branch matches this head. Added `packages.x86_64-linux.default` and `packages.aarch64-linux.default`, using the locked Rust toolchain, Cargo.lock vendoring, checksum-pinned original Zig compiler and fixed-output native/Bun dependency trees. The package calls the existing shipping scripts, disables executable fixups, and includes the five release executables and production web assets. Documentation includes reproduction instructions, the external signing boundary, and a cold systemd example. No apps or NixOS module, no production access or changes. Same clean-commit comparison on compute01, through build-slot: | artifact | normal bytes | nix bytes | normal sha256 | nix sha256 | identical | |---|---:|---:|---|---|---| | infraternal | 21,431,472 | 21,431,472 | `b527c33d9df30659c816b31546c9583b05c85bae87fa7b95eec4e207d589ea3c` | `b527c33d9df30659c816b31546c9583b05c85bae87fa7b95eec4e207d589ea3c` | True | | infraternal-x86_64-musl | 10,121,376 | 10,121,376 | `2176ca2af1a3d66711e1a6b9dc200d678f40e7ce32fa6046ba741cffc27086c8` | `2176ca2af1a3d66711e1a6b9dc200d678f40e7ce32fa6046ba741cffc27086c8` | True | | infraternal-aarch64-musl | 9,512,056 | 9,512,056 | `521c14d33d3650ef5f02783622cfacaa8441f186661917ec7baf8f2784c3a935` | `521c14d33d3650ef5f02783622cfacaa8441f186661917ec7baf8f2784c3a935` | True | | infraternal-x86_64-session-musl | 1,863,592 | 1,863,592 | `58238b490c27ad69f319111e906318d4719458ad2bd7e24b25fc0ed1d02d2ed2` | `58238b490c27ad69f319111e906318d4719458ad2bd7e24b25fc0ed1d02d2ed2` | True | | infraternal-aarch64-session-musl | 1,622,400 | 1,622,400 | `de6e2d24164b96327269d0238b1ccb9b7fee63947f592e70e008ca2fd4d357f0` | `de6e2d24164b96327269d0238b1ccb9b7fee63947f592e70e008ca2fd4d357f0` | True | web: 284 files; 0 differences: [] package SHA256SUMS: PASS; all five executables: PT_INTERP absent All native sizes have zero growth. The raw Zig compiler executable is also byte-identical to the original pinned archive (SHA-256 `2858dc89dbbfdd08cceda1b841e7fd0a793a1a67b49f150bc3d0d1de44ed7f51`). Packaged key-memory check passes with 8 MiB memlock. Runtime closure contains only the package itself, 48,141,272 bytes. Assembly differences from a plain source `scripts/package-release` directory are documented: Nix installs under `share/infraternal` with a `bin` symlink, adds SOURCE/workspace VERSION, and expands SHA256SUMS to cover metadata and web files. Normal web build revision/date inputs were aligned with the flake source; the reference assembles those records with the same schema. The default remains an unsigned source build. Release identity override uses the existing release inputs; GATE/MANIFEST/signatures and production admission remain external. Both Linux package attributes evaluate on this head. The native aarch64 package was not built; its full host/session payloads were actually cross-built and compared above. Final `scripts/verify fast` passed on this exact clean head, through build-slot, with CARGO_BUILD_JOBS=4, private short state roots, and debug/test symbols disabled as in #264 (release flags unchanged). The first final attempt failed because my earlier cache trim removed the source include tree required by the mouse probe. Unmodified dev probe sources reproduced that missing-header prerequisite failure; restoring nine include directories from the verified pinned archive made the focused mouse test pass, then this full retry passed. No repository change or gate weakening was needed. The final release-bundle fixture also passed. Verbatim final gate summaries (complete raw transcript attached): ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | static builds | PASS | +---------------------------+--------------+ PT_INTERP: absent; size: 21431472 bytes; cap: 21431472 bytes; delta: +0 bytes PT_INTERP: absent; size: 19605896 bytes; cap: 19605896 bytes; delta: +0 bytes 2 pass 0 fail 121 expect() calls Ran 2 tests across 1 file. [585.00ms] 363 pass 0 fail 21695 expect() calls Ran 363 tests across 60 files. [11.09s] 105 pass 0 fail 413 expect() calls Ran 105 tests across 13 files. [275.00ms] 17 pass 0 fail 1794 expect() calls Ran 17 tests across 4 files. [324.00ms] 39 pass 0 fail 50325 expect() calls Ran 39 tests across 10 files. [4.12s] ``` Driver output, verbatim: ``` FINAL_VERIFY_EXIT=0 ``` Cleanup verified: own build runner exited, zero remaining own test processes; both target directories, result symlink, temporary binary reference bundles/probe output, and private test state removed. Small logs/receipts are retained outside the checkout. Cleanup receipt attached. No blockers or new owner decisions for this implementation; #274 still retains its migration/admission decisions.

verify-fast-final.log

468 KiB

cleanup.log

686 B

package-release-test-final.log

398 B

package-platforms-final.json

296 B

key-memory.log

69 B

kayg commented

2026-11-01 06:19:06 +00:00

Author

Owner

J45 start (2026-11-01 11:49 IST). Branch prod-on-ssh, base origin/dev at 2b2869274325c633fabff9579300647e5e13c624; worktree /home/kayg/Developer/Worktrees/infraternal-prod-on-ssh.

Scope follows kayg's newer explicit approval: install a fresh server on ssh from the current origin/dev Nix package, with no data migration; route the public Traefik backend only after the new instance is healthy; then remove only the old infraternal service installation from monitor01, retaining /var/backups/infraternal and Traefik. I will keep SELinux enforcing, limit firewall openings to measured service paths, stop for the owner CA unlock if requested, and post evidence for each stage. No production host has been changed yet.

J45 start (2026-11-01 11:49 IST). Branch `prod-on-ssh`, base `origin/dev` at `2b2869274325c633fabff9579300647e5e13c624`; worktree `/home/kayg/Developer/Worktrees/infraternal-prod-on-ssh`. Scope follows kayg's newer explicit approval: install a fresh server on `ssh` from the current `origin/dev` Nix package, with no data migration; route the public Traefik backend only after the new instance is healthy; then remove only the old infraternal service installation from `monitor01`, retaining `/var/backups/infraternal` and Traefik. I will keep SELinux enforcing, limit firewall openings to measured service paths, stop for the owner CA unlock if requested, and post evidence for each stage. No production host has been changed yet.

kayg commented

2026-11-01 06:22:02 +00:00

Author

Owner

J45 read-only preflight (2026-11-01 11:52 IST). Evidence:

ssh (10.69.69.89) is AlmaLinux 10.2, x86_64, SELinux Enforcing, with NetBird interface wt0 at 10.69.69.89 and 47 GiB free. Nix and nftables/firewalld tooling are absent; NetBird is active. No service or firewall changes have been made.
monitor01 still has infraternal-server.service active/enabled, root-run with KillMode=process. Its unit is /etc/systemd/system/infraternal-server.service; it runs /usr/local/bin/infraternal, serves HTTP on 172.18.0.1:7011, and listens for remote hosts on 10.69.69.50:7443. Its mutable state is /var/lib/infraternal and /run/infraternal; backups are under /var/backups/infraternal.
The public edge container is kayg-traefik. Existing unrelated monitor01 services are present; none have been or will be changed. The checked-in release deploy script also still targets 10.69.69.50 and the legacy /usr/local layout, which is part of the tooling update.

The current package commit to deploy is 2b2869274325c633fabff9579300647e5e13c624. Next: install Nix on ssh, build that flake package on compute01 through build-slot, and copy its closure to ssh.

J45 read-only preflight (2026-11-01 11:52 IST). Evidence: - `ssh` (10.69.69.89) is AlmaLinux 10.2, x86_64, SELinux Enforcing, with NetBird interface `wt0` at 10.69.69.89 and 47 GiB free. Nix and nftables/firewalld tooling are absent; NetBird is active. No service or firewall changes have been made. - `monitor01` still has `infraternal-server.service` active/enabled, root-run with `KillMode=process`. Its unit is `/etc/systemd/system/infraternal-server.service`; it runs `/usr/local/bin/infraternal`, serves HTTP on 172.18.0.1:7011, and listens for remote hosts on 10.69.69.50:7443. Its mutable state is `/var/lib/infraternal` and `/run/infraternal`; backups are under `/var/backups/infraternal`. - The public edge container is `kayg-traefik`. Existing unrelated monitor01 services are present; none have been or will be changed. The checked-in release deploy script also still targets 10.69.69.50 and the legacy `/usr/local` layout, which is part of the tooling update. The current package commit to deploy is `2b2869274325c633fabff9579300647e5e13c624`. Next: install Nix on `ssh`, build that flake package on compute01 through `build-slot`, and copy its closure to `ssh`.

kayg commented

2026-11-01 06:24:40 +00:00

Author

Owner

J45 finding (2026-11-01 11:54 IST): the official multi-user Nix installer refuses to proceed while SELinux is enabled (Nix does not work with selinux enabled yet!). It exited before installation. Rechecked ssh: SELinux is still Enforcing, /nix is absent, and nix-daemon.service is inactive. No target state changed. I am checking the alternative Determinate installer for SELinux handling; I will not disable SELinux or bypass the installer check.

J45 finding (2026-11-01 11:54 IST): the official multi-user Nix installer refuses to proceed while SELinux is enabled (`Nix does not work with selinux enabled yet!`). It exited before installation. Rechecked `ssh`: SELinux is still Enforcing, `/nix` is absent, and `nix-daemon.service` is inactive. No target state changed. I am checking the alternative Determinate installer for SELinux handling; I will not disable SELinux or bypass the installer check.

kayg commented

2026-11-01 06:26:25 +00:00

Author

Owner

J45 step 1, Nix install (2026-11-01 11:56 IST). The official multi-user installer refused SELinux, so I used the permitted Determinate installer v3.24.1 with the Linux/systemd planner, upstream package preference left default, four build users, no profile changes, and diagnostics disabled. Nix 2.35.3 is installed and both nix-daemon.service and its socket are active. SELinux remains Enforcing. Target disk still has 47 GiB free. No application package or service has been installed or started yet; next I am checking SELinux labels and the Nix daemon before the package transfer.

J45 step 1, Nix install (2026-11-01 11:56 IST). The official multi-user installer refused SELinux, so I used the permitted Determinate installer v3.24.1 with the Linux/systemd planner, upstream package preference left default, four build users, no profile changes, and diagnostics disabled. Nix 2.35.3 is installed and both `nix-daemon.service` and its socket are active. SELinux remains Enforcing. Target disk still has 47 GiB free. No application package or service has been installed or started yet; next I am checking SELinux labels and the Nix daemon before the package transfer.

kayg commented

2026-11-01 06:28:31 +00:00

Author

Owner

J45 revision update (2026-11-01 11:58 IST): git ls-remote origin refs/heads/dev now reports f4b61cd6c3f52078cad7573c5367d7a6e75f84f0 (merge #237), advanced from the 2b286927 head recorded at J45 start. The compute01 fetch observed the same new SHA; its expected-SHA guard stopped before creating the build worktree, so no stale package was built or copied. I am advancing the clean prod-on-ssh branch to f4b61cd6c3f52078cad7573c5367d7a6e75f84f0; this will be the deployed source/version.

J45 revision update (2026-11-01 11:58 IST): `git ls-remote origin refs/heads/dev` now reports `f4b61cd6c3f52078cad7573c5367d7a6e75f84f0` (merge #237), advanced from the `2b286927` head recorded at J45 start. The compute01 fetch observed the same new SHA; its expected-SHA guard stopped before creating the build worktree, so no stale package was built or copied. I am advancing the clean `prod-on-ssh` branch to `f4b61cd6c3f52078cad7573c5367d7a6e75f84f0`; this will be the deployed source/version.

kayg commented

2026-11-01 06:39:08 +00:00

Author

Owner

J45 edge finding (2026-11-01 12:08 IST). kayg-traefik reads dynamic configuration from the host directory /opt/infra/traefik/dynamic; its infraternal route is the single file /opt/infra/traefik/dynamic/infraternal.yml, currently forwarding to http://172.18.0.1:7011. ip route get 10.69.69.89 on monitor01 selects wt0 with source 10.69.69.50, so the new backend and trusted-proxy/firewall source can be restricted to that NetBird peer. No Traefik config has been edited. I will save a copy of this one file before changing only its backend URL; Traefik's other routes and container remain untouched.

J45 edge finding (2026-11-01 12:08 IST). `kayg-traefik` reads dynamic configuration from the host directory `/opt/infra/traefik/dynamic`; its infraternal route is the single file `/opt/infra/traefik/dynamic/infraternal.yml`, currently forwarding to `http://172.18.0.1:7011`. `ip route get 10.69.69.89` on monitor01 selects `wt0` with source 10.69.69.50, so the new backend and trusted-proxy/firewall source can be restricted to that NetBird peer. No Traefik config has been edited. I will save a copy of this one file before changing only its backend URL; Traefik's other routes and container remain untouched.

kayg commented

2026-11-01 06:42:17 +00:00

Author

Owner

J45 continuation (2026-11-01 12:11 IST). I located the clean worktree at source . Nix is present on and SELinux remains Enforcing. The compute01 package build is active via build-slot (PID 3452073; log ); no new service, firewall or Traefik changes have been made. Next: finish and verify the package, transfer its closure, then stage the fresh service with SELinux enforcing and a source-limited firewall.

kayg commented

2026-11-01 06:42:23 +00:00

Author

Owner

J45 continuation (2026-11-01 12:11 IST). I located the clean prod-on-ssh worktree at source f4b61cd6c3. Nix is present on ssh and SELinux remains Enforcing. The compute01 package build is active via build-slot (PID 3452073; log ~/j45-evidence/package-build.log); no new service, firewall or Traefik changes have been made. Next: finish and verify the package, transfer its closure, then stage the fresh service with SELinux enforcing and a source-limited firewall.

J45 continuation (2026-11-01 12:11 IST). I located the clean prod-on-ssh worktree at source f4b61cd6c3f52078cad7573c5367d7a6e75f84f0. Nix is present on ssh and SELinux remains Enforcing. The compute01 package build is active via build-slot (PID 3452073; log ~/j45-evidence/package-build.log); no new service, firewall or Traefik changes have been made. Next: finish and verify the package, transfer its closure, then stage the fresh service with SELinux enforcing and a source-limited firewall.

kayg commented

2026-11-01 06:43:13 +00:00

Author

Owner

Correction to the previous 12:11 IST continuation note: one duplicate was created with empty interpolated fields. The immediately following 12:11 note has the correct branch, source SHA, host, build PID, and log path; disregard the malformed duplicate.

kayg commented

2026-11-01 06:48:13 +00:00

Author

Owner

J45 progress (2026-11-01 12:18 IST). The release source remains the current origin/dev head f4b61cd6c3. Its package build is still running on compute01 through build-slot (PID 3452073; log ~/j45-evidence/package-build.log). On ssh, nftables 1.1.7 is installed via Nix from the locked nixpkgs revision 885ba4d038f10c3d17d7a92e8c3cc185b99c9158; no firewall policy is loaded yet. The only current nft tables are NetBird-owned ip netbird and ip6 netbird, which I will preserve. No application package, service, firewall policy or Traefik route has been started/changed; SELinux remains Enforcing.

J45 progress (2026-11-01 12:18 IST). The release source remains the current origin/dev head f4b61cd6c3f52078cad7573c5367d7a6e75f84f0. Its package build is still running on compute01 through build-slot (PID 3452073; log ~/j45-evidence/package-build.log). On ssh, nftables 1.1.7 is installed via Nix from the locked nixpkgs revision 885ba4d038f10c3d17d7a92e8c3cc185b99c9158; no firewall policy is loaded yet. The only current nft tables are NetBird-owned ip netbird and ip6 netbird, which I will preserve. No application package, service, firewall policy or Traefik route has been started/changed; SELinux remains Enforcing.

kayg commented

2026-11-01 06:50:33 +00:00

Author

Owner

J45 finding (2026-11-01 12:20 IST). Inspecting the existing NetBird ACL on ssh shows chain netbird-acl-input-filter drops unmatched packets arriving on wt0. Its allow rules include TCP 80/443 and UDP 443 from the peer set containing monitor01 10.69.69.50; they do not include the required backend TCP 7011. It also has no TCP/UDP 7443 rule for the fresh remote listener. I have not changed NetBird policy, nft rules, or Traefik. A local nft allow cannot override the existing NetBird drop; I am holding the public route change until this is resolved.

kayg commented

2026-11-01 06:52:30 +00:00

Author

Owner

Kayg decision (2026-11-01 12:22 IST): hold the public route switch and monitor01 cleanup until the backend path works. Exact NetBird rule requested: inbound allow on ssh wt0, source peer monitor01 10.69.69.50, destination peer ssh 10.69.69.89, protocol TCP, destination port 7011. No broader access is needed for the Traefik backend. I will continue service staging, local/mesh health checks, deploy tooling and the rollback note, then poll the monitor01-to-ssh backend path every five minutes. Traefik and the old service remain untouched until the connection succeeds.

kayg commented

2026-11-01 07:07:28 +00:00

Author

Owner

J45 package step complete (2026-11-01 12:37 IST). The compute01 Nix build finished through build-slot with exit 0. Source is current origin/dev f4b61cd6c3; package VERSION is 0.1.0. Output is /nix/store/1qhq717g6v249l5kjxm10fcn9vbhj20m-infraternal-0.1.0; NAR hash sha256-CGRws/LSzfGKmZdZjrNg8duF+KEmrQOZgvUxxdDtKwE=, 48,187,552 bytes. Package verification checked all 291 regular bundle files; server is 21,443,824 bytes, SHA-256 b2cec7a205c522dff4f0f68c07bb2e1da93a9f81f234111ef29420bb2a5d4ed1. Source package is unsigned by design and the owner-approved fresh install path checks SOURCE plus SHA256SUMS. Six targeted deploy tests pass on compute01; generated nft input rules pass nft -c. On ssh, SELinux is Enforcing and nftables 1.1.7 is installed via Nix; no host filter rules, application service, or Traefik route changed. Next: copy the closure, install the fresh unit and host firewall, verify the service locally and over the mesh, then finish tooling/docs and the fast gate.

J45 package step complete (2026-11-01 12:37 IST). The compute01 Nix build finished through build-slot with exit 0. Source is current origin/dev f4b61cd6c3f52078cad7573c5367d7a6e75f84f0; package VERSION is 0.1.0. Output is /nix/store/1qhq717g6v249l5kjxm10fcn9vbhj20m-infraternal-0.1.0; NAR hash sha256-CGRws/LSzfGKmZdZjrNg8duF+KEmrQOZgvUxxdDtKwE=, 48,187,552 bytes. Package verification checked all 291 regular bundle files; server is 21,443,824 bytes, SHA-256 b2cec7a205c522dff4f0f68c07bb2e1da93a9f81f234111ef29420bb2a5d4ed1. Source package is unsigned by design and the owner-approved fresh install path checks SOURCE plus SHA256SUMS. Six targeted deploy tests pass on compute01; generated nft input rules pass nft -c. On ssh, SELinux is Enforcing and nftables 1.1.7 is installed via Nix; no host filter rules, application service, or Traefik route changed. Next: copy the closure, install the fresh unit and host firewall, verify the service locally and over the mesh, then finish tooling/docs and the fast gate.

kayg commented

2026-11-01 07:17:27 +00:00

Author

Owner

J45 checkpoint (2026-11-01 13:18 IST). The fresh package for current origin/dev f4b61cd6c3f52078cad7573c5367d7a6e75f84f0 is built and verified on compute01; Nix is installed on ssh and SELinux remains Enforcing. The package closure has not yet been copied, and no application service, local host firewall table, Traefik route, or monitor01 service has been changed. I am tightening the generated deployment firewall and removing an unnecessary target health-check package dependency before installation.

Exact NetBird allow requested for the backend path: inbound on ssh wt0, source peer monitor01 10.69.69.50, destination peer ssh 10.69.69.89, protocol TCP, destination port 7011. Local nftables cannot override the current NetBird ACL drop. Public route and old-instance cleanup remain held. Next: install and verify the fresh service on ssh, document the CA passkey step and rollback on this issue, and poll monitor01 to ssh TCP 7011 every five minutes once the listener is healthy.

J45 checkpoint (2026-11-01 13:18 IST). The fresh package for current origin/dev `f4b61cd6c3f52078cad7573c5367d7a6e75f84f0` is built and verified on compute01; Nix is installed on ssh and SELinux remains Enforcing. The package closure has not yet been copied, and no application service, local host firewall table, Traefik route, or monitor01 service has been changed. I am tightening the generated deployment firewall and removing an unnecessary target health-check package dependency before installation. Exact NetBird allow requested for the backend path: inbound on ssh `wt0`, source peer monitor01 `10.69.69.50`, destination peer ssh `10.69.69.89`, protocol TCP, destination port `7011`. Local nftables cannot override the current NetBird ACL drop. Public route and old-instance cleanup remain held. Next: install and verify the fresh service on ssh, document the CA passkey step and rollback on this issue, and poll monitor01 to ssh TCP `7011` every five minutes once the listener is healthy.

kayg commented

2026-11-01 07:22:44 +00:00

Author

Owner

Correction: the timestamp in my preceding J45 checkpoint was inaccurate; its status details are otherwise current.

Owner direction update for #274: the replacement public name is ssh.kayg.org, A 49.205.172.130; fridge-ingress (10.69.69.99 on NetBird) will terminate DNS-01 TLS and proxy to ssh. The monitor01-to-ssh allow request for source 10.69.69.50 is withdrawn and must not be added.

If fridge-ingress cannot reach the new listener, the exact NetBird rule needed is inbound on ssh wt0, source peer fridge-ingress 10.69.69.99, destination peer ssh 10.69.69.89, protocol TCP, destination port 7011. I will hold the new Traefik route until that path connects. Keep the existing infraternal.kayg.org DNS/route and monitor01 service/backups until ssh.kayg.org is verified end to end; then remove only its old route and DNS record and retire only the old infraternal service installation.

Correction: the timestamp in my preceding J45 checkpoint was inaccurate; its status details are otherwise current. Owner direction update for #274: the replacement public name is `ssh.kayg.org`, A `49.205.172.130`; fridge-ingress (`10.69.69.99` on NetBird) will terminate DNS-01 TLS and proxy to ssh. The monitor01-to-ssh allow request for source `10.69.69.50` is withdrawn and must not be added. If fridge-ingress cannot reach the new listener, the exact NetBird rule needed is inbound on ssh `wt0`, source peer fridge-ingress `10.69.69.99`, destination peer ssh `10.69.69.89`, protocol TCP, destination port `7011`. I will hold the new Traefik route until that path connects. Keep the existing `infraternal.kayg.org` DNS/route and monitor01 service/backups until `ssh.kayg.org` is verified end to end; then remove only its old route and DNS record and retire only the old infraternal service installation.

kayg commented

2026-11-01 07:27:46 +00:00

Author

Owner

J45 read-only ingress preflight (2026-11-01 12:53 IST). fridge-ingress is NetBird 10.69.69.99; its route to ssh 10.69.69.89 uses wt0 with source 10.69.69.99. Traefik is the systemd service on that host. Its file provider watches /etc/traefik/dynamic, the HTTPS entry point is websecure, and the existing DNS-01 resolver is cf with Cloudflare. I did not change its configuration or expose the API token.

Cloudflare read-only inspection found no ssh.kayg.org record. The existing infraternal.kayg.org A record is 37.157.255.66, DNS-only (proxied=false); the zone currently has 9 proxied and 65 DNS-only records. I will make the new A record ssh.kayg.org -> 49.205.172.130 DNS-only to match the existing infraternal record.

The old .50 -> .89:7011 request is withdrawn. If the new source is blocked, the exact required NetBird allow is inbound on ssh wt0, source fridge-ingress 10.69.69.99, destination ssh 10.69.69.89, TCP destination port 7011. I will create the new route only after that backend connection succeeds. The old public name, route, and monitor01 service remain in place until ssh.kayg.org passes end-to-end TLS, sign-in page, and WebSocket checks.

J45 read-only ingress preflight (2026-11-01 12:53 IST). `fridge-ingress` is NetBird `10.69.69.99`; its route to ssh `10.69.69.89` uses `wt0` with source `10.69.69.99`. Traefik is the systemd service on that host. Its file provider watches `/etc/traefik/dynamic`, the HTTPS entry point is `websecure`, and the existing DNS-01 resolver is `cf` with Cloudflare. I did not change its configuration or expose the API token. Cloudflare read-only inspection found no `ssh.kayg.org` record. The existing `infraternal.kayg.org` A record is `37.157.255.66`, DNS-only (`proxied=false`); the zone currently has 9 proxied and 65 DNS-only records. I will make the new A record `ssh.kayg.org -> 49.205.172.130` DNS-only to match the existing infraternal record. The old `.50 -> .89:7011` request is withdrawn. If the new source is blocked, the exact required NetBird allow is inbound on ssh `wt0`, source fridge-ingress `10.69.69.99`, destination ssh `10.69.69.89`, TCP destination port `7011`. I will create the new route only after that backend connection succeeds. The old public name, route, and monitor01 service remain in place until `ssh.kayg.org` passes end-to-end TLS, sign-in page, and WebSocket checks.

kayg commented

2026-11-01 07:28:53 +00:00

Author

Owner

J45 DNS step (2026-11-01 12:58 IST). Created Cloudflare A record ssh.kayg.org -> 49.205.172.130, DNS-only (proxied=false, automatic TTL). This matches the existing DNS-only infraternal.kayg.org record. The credential was read from Traefik's active token file on fridge-ingress and used in a Python process there; it was not printed, copied, or placed in argv. No Traefik route or monitor01 setting has changed yet. The new route stays held until fridge-ingress can connect to ssh TCP 7011.

J45 DNS step (2026-11-01 12:58 IST). Created Cloudflare A record `ssh.kayg.org -> 49.205.172.130`, DNS-only (`proxied=false`, automatic TTL). This matches the existing DNS-only `infraternal.kayg.org` record. The credential was read from Traefik's active token file on fridge-ingress and used in a Python process there; it was not printed, copied, or placed in argv. No Traefik route or monitor01 setting has changed yet. The new route stays held until fridge-ingress can connect to ssh TCP 7011.

kayg commented

2026-11-01 07:31:30 +00:00

Author

Owner

J45 tooling step (2026-11-01 13:01 IST). Pushed prod-on-ssh commit 4847c4819. scripts/release/deploy now requires a fresh Nix store output from the current origin/dev head for production; the package verifier checks SOURCE and every regular bundle file. The new systemd/firewall renderer targets ssh.kayg.org and fridge-ingress 10.69.69.99. Existing perf-VM staging remains separate. Targeted unit run on compute01: 22 passed, 0 failed; Python compilation passed. The generated nft rules passed nft -c on ssh. Documentation updates are in progress. No app service or host firewall was installed yet; no monitor01 configuration changed. Next I will copy the verified closure, install the fresh service, and test fridge-ingress reachability before creating its Traefik route.

J45 tooling step (2026-11-01 13:01 IST). Pushed `prod-on-ssh` commit `4847c4819`. `scripts/release/deploy` now requires a fresh Nix store output from the current `origin/dev` head for production; the package verifier checks SOURCE and every regular bundle file. The new systemd/firewall renderer targets `ssh.kayg.org` and fridge-ingress `10.69.69.99`. Existing perf-VM staging remains separate. Targeted unit run on compute01: 22 passed, 0 failed; Python compilation passed. The generated nft rules passed `nft -c` on ssh. Documentation updates are in progress. No app service or host firewall was installed yet; no monitor01 configuration changed. Next I will copy the verified closure, install the fresh service, and test fridge-ingress reachability before creating its Traefik route.

kayg commented

2026-11-01 07:42:57 +00:00

Author

Owner

J45 Nix copy finding (2026-11-01, IST). The focused regression on compute01 reproduced the failure: DeploymentTests.test_fresh_nix_copy_uses_strict_job_known_hosts_for_both_ssh_clients fails because the nix copy --to URI is only ssh://root@10.69.69.89, and does not select the installed remote store program. On ssh, root's non-login PATH does not contain nix-store, while /nix/var/nix/profiles/default/bin/nix-store exists and is executable. The SSH store supports remote-program; I will use that absolute installed path in the destination URI, keeping strict host-key checking and avoiding a broad PATH change. No service or public route was started/changed while the copy was failing.

Current backend ACL request under kayg's latest public-edge direction: inbound on ssh wt0, source peer fridge-ingress 10.69.69.99, destination ssh 10.69.69.89, protocol TCP, destination port 7011. The earlier monitor01 .50 request is withdrawn. Keep the infraternal.kayg.org route/DNS and monitor01 service intact until ssh.kayg.org passes TLS, sign-in, and WebSocket checks.

J45 Nix copy finding (2026-11-01, IST). The focused regression on compute01 reproduced the failure: `DeploymentTests.test_fresh_nix_copy_uses_strict_job_known_hosts_for_both_ssh_clients` fails because the `nix copy --to` URI is only `ssh://root@10.69.69.89`, and does not select the installed remote store program. On ssh, root's non-login PATH does not contain `nix-store`, while `/nix/var/nix/profiles/default/bin/nix-store` exists and is executable. The SSH store supports `remote-program`; I will use that absolute installed path in the destination URI, keeping strict host-key checking and avoiding a broad PATH change. No service or public route was started/changed while the copy was failing. Current backend ACL request under kayg's latest public-edge direction: inbound on ssh `wt0`, source peer fridge-ingress `10.69.69.99`, destination ssh `10.69.69.89`, protocol TCP, destination port `7011`. The earlier monitor01 `.50` request is withdrawn. Keep the `infraternal.kayg.org` route/DNS and monitor01 service intact until `ssh.kayg.org` passes TLS, sign-in, and WebSocket checks.

kayg commented

2026-11-01 07:44:48 +00:00

Author

Owner

J45 Nix copy fix (2026-11-01 13:14 IST), pushed prod-on-ssh head c549e36db. The regression failed before the fix because nix copy used ssh://root@10.69.69.89 without remote-program; after setting the destination URI to use /nix/var/nix/profiles/default/bin/nix-store, the same focused test passed. Full compute01 deploy suite: Ran 23 tests in 1.075s / OK; Python AST syntax check: deploy_nix.py syntax: PASS. The test retains strict host-key checks for both SSH clients.

Next I will retry the verified package closure copy and fresh service install on ssh. No application unit/service or host firewall policy is installed yet; the new public Traefik route and all monitor01 state remain untouched. Latest exact NetBird allow request remains inbound to ssh wt0, from fridge-ingress 10.69.69.99 to ssh 10.69.69.89, TCP destination port 7011; the withdrawn monitor01 .50 rule is not needed.

J45 Nix copy fix (2026-11-01 13:14 IST), pushed `prod-on-ssh` head `c549e36db`. The regression failed before the fix because `nix copy` used `ssh://root@10.69.69.89` without `remote-program`; after setting the destination URI to use `/nix/var/nix/profiles/default/bin/nix-store`, the same focused test passed. Full compute01 deploy suite: `Ran 23 tests in 1.075s` / `OK`; Python AST syntax check: `deploy_nix.py syntax: PASS`. The test retains strict host-key checks for both SSH clients. Next I will retry the verified package closure copy and fresh service install on ssh. No application unit/service or host firewall policy is installed yet; the new public Traefik route and all monitor01 state remain untouched. Latest exact NetBird allow request remains inbound to ssh `wt0`, from fridge-ingress `10.69.69.99` to ssh `10.69.69.89`, TCP destination port `7011`; the withdrawn monitor01 `.50` rule is not needed.

kayg commented

2026-11-01 07:46:53 +00:00

Author

Owner

J45 firewall finding (2026-11-01, IST). Before loading any host filter rules, I checked the actual management path: the control host route to 10.69.69.89 is dev wt0 src 10.69.69.92; ssh -G root@10.69.69.89 selects port 22, and sshd on ssh listens on 0.0.0.0:22 and [::]:22. The generated fresh-host nft policy instead permitted wt0 TCP 22022. A new focused renderer regression fails against the current output because the TCP 22 rule is absent. I will change that one rule to permit TCP 22 only on wt0 from the NetBird subnet, retain NetBird ACL enforcement, and keep public SSH closed. No firewall policy has been loaded.

J45 firewall finding (2026-11-01, IST). Before loading any host filter rules, I checked the actual management path: the control host route to `10.69.69.89` is `dev wt0 src 10.69.69.92`; `ssh -G root@10.69.69.89` selects port `22`, and sshd on ssh listens on `0.0.0.0:22` and `[::]:22`. The generated fresh-host nft policy instead permitted `wt0` TCP `22022`. A new focused renderer regression fails against the current output because the TCP 22 rule is absent. I will change that one rule to permit TCP 22 only on `wt0` from the NetBird subnet, retain NetBird ACL enforcement, and keep public SSH closed. No firewall policy has been loaded.

kayg commented

2026-11-01 07:48:40 +00:00

Author

Owner

J45 firewall correction (2026-11-01 13:18 IST), pushed prod-on-ssh head d7ee85dc5. The focused firewall regression was RED against TCP 22022, then GREEN after the generated rule was changed to TCP 22 on wt0 from 10.69.69.0/24. The full compute01 deploy suite passes: Ran 23 tests in 1.025s / OK. The rendered nft policy passed nft -c -f - on ssh (NFT_CHECK=PASS). No host filter table has been installed yet; this prevents locking out the active NetBird SSH management path.

Next: retry closure copy and install the fresh systemd service on ssh, then verify health and SELinux. The current ingress request remains inbound on ssh wt0, source fridge-ingress 10.69.69.99, destination ssh 10.69.69.89, TCP destination port 7011. No public route switch or old-instance cleanup until that path and end-to-end checks pass.

J45 firewall correction (2026-11-01 13:18 IST), pushed `prod-on-ssh` head `d7ee85dc5`. The focused firewall regression was RED against TCP `22022`, then GREEN after the generated rule was changed to TCP `22` on `wt0` from `10.69.69.0/24`. The full compute01 deploy suite passes: `Ran 23 tests in 1.025s` / `OK`. The rendered nft policy passed `nft -c -f -` on ssh (`NFT_CHECK=PASS`). No host filter table has been installed yet; this prevents locking out the active NetBird SSH management path. Next: retry closure copy and install the fresh systemd service on ssh, then verify health and SELinux. The current ingress request remains inbound on ssh `wt0`, source fridge-ingress `10.69.69.99`, destination ssh `10.69.69.89`, TCP destination port `7011`. No public route switch or old-instance cleanup until that path and end-to-end checks pass.

kayg commented

2026-11-01 07:50:59 +00:00

Author

Owner

J45 install attempt note (2026-11-01, IST). The first deploy invocation stopped during strict SSH preflight with exit 255 before nix copy: I misspelled the job-local known-hosts environment variable, so the configured pinned host-key file was not selected. Target inspection confirms the package is not valid in ssh's store and no infraternal service/unit, state directory, runtime directory, or host nft table exists. I corrected the invocation to use the exact INFRATERNAL_SSH_KNOWN_HOSTS name and am retrying; strict host-key checking remains enabled. No repo change was needed.

J45 install attempt note (2026-11-01, IST). The first deploy invocation stopped during strict SSH preflight with exit 255 before `nix copy`: I misspelled the job-local known-hosts environment variable, so the configured pinned host-key file was not selected. Target inspection confirms the package is not valid in ssh's store and no infraternal service/unit, state directory, runtime directory, or host nft table exists. I corrected the invocation to use the exact `INFRATERNAL_SSH_KNOWN_HOSTS` name and am retrying; strict host-key checking remains enabled. No repo change was needed.

kayg commented

2026-11-01 07:55:51 +00:00

Author

Owner

J45 SELinux finding (2026-11-01 13:25 IST). The strict preflight passed and nix copy transferred the verified package to ssh. The installer created only the fresh empty /var/lib/infraternal plus this job's systemd/nft config; no production data was copied. Starting infraternal-firewall.service failed with systemd status=203/EXEC / Permission denied for /root/.nix-profile/bin/nft. SELinux remains Enforcing; the profile symlink and resolved Nix store nft currently have default_t, and the host nft table is absent. infraternal-server.service is inactive and no public route changed. I will point the firewall unit directly at the resolved immutable Nix store binary, assign that exact binary a persistent bin_t context, restore its context, then validate the unit before starting it. No SELinux mode change or broad store labeling will be used.

J45 SELinux finding (2026-11-01 13:25 IST). The strict preflight passed and `nix copy` transferred the verified package to ssh. The installer created only the fresh empty `/var/lib/infraternal` plus this job's systemd/nft config; no production data was copied. Starting `infraternal-firewall.service` failed with systemd `status=203/EXEC` / `Permission denied` for `/root/.nix-profile/bin/nft`. SELinux remains Enforcing; the profile symlink and resolved Nix store `nft` currently have `default_t`, and the host nft table is absent. `infraternal-server.service` is inactive and no public route changed. I will point the firewall unit directly at the resolved immutable Nix store binary, assign that exact binary a persistent `bin_t` context, restore its context, then validate the unit before starting it. No SELinux mode change or broad store labeling will be used.

kayg commented

2026-11-01 07:59:02 +00:00

Author

Owner

J45 SELinux firewall fix (2026-11-01 13:29 IST), pushed prod-on-ssh head 0fe3570e0. The new unit regression failed against the profile-path renderer, then passed after the renderer took an immutable nft store path. The installer now resolves /root/.nix-profile/bin/nft, verifies it resolves beneath /nix/store, adds a persistent bin_t context for that exact executable, restores its label, and writes the unit with the direct store path. No broad /nix label or SELinux mode change.

Compute01 deploy suite: Ran 24 tests in 1.138s / OK. The target's first install attempt left only this fresh install's empty state directory and unit files; infraternal-firewall.service failed before loading any table and infraternal-server.service remains inactive. I will apply the exact-path label/unit correction to that partial install, validate and start the firewall, then start and health-check the server. No public route or monitor01 setting changed.

J45 SELinux firewall fix (2026-11-01 13:29 IST), pushed `prod-on-ssh` head `0fe3570e0`. The new unit regression failed against the profile-path renderer, then passed after the renderer took an immutable nft store path. The installer now resolves `/root/.nix-profile/bin/nft`, verifies it resolves beneath `/nix/store`, adds a persistent `bin_t` context for that exact executable, restores its label, and writes the unit with the direct store path. No broad `/nix` label or SELinux mode change. Compute01 deploy suite: `Ran 24 tests in 1.138s` / `OK`. The target's first install attempt left only this fresh install's empty state directory and unit files; `infraternal-firewall.service` failed before loading any table and `infraternal-server.service` remains inactive. I will apply the exact-path label/unit correction to that partial install, validate and start the firewall, then start and health-check the server. No public route or monitor01 setting changed.

kayg commented

2026-11-01 08:05:45 +00:00

Author

Owner

J45 SELinux parent-path correction (2026-11-01 13:35 IST), pushed prod-on-ssh head 465396f1c. The target confirmed /usr is usr_t and /usr/bin is bin_t; /nix, /nix/store, and the nft package's parent directories were default_t. Labeling only the executable file bin_t still produced systemd 203/EXEC, so no nft table was loaded and the infraternal server remains stopped.

The installer now queries the Nix runtime closures for the infraternal package and nft, sets usr_t on only /nix and /nix/store plus those closure roots, and sets bin_t on the five package executables and nft. It restores those exact paths. This does not relabel unrelated Nix store outputs or change SELinux mode. The release deploy suite passes: Ran 24 tests in 1.063s / OK. I will apply this scoped context to the current partial install and check systemd execution again.

J45 SELinux parent-path correction (2026-11-01 13:35 IST), pushed `prod-on-ssh` head `465396f1c`. The target confirmed `/usr` is `usr_t` and `/usr/bin` is `bin_t`; `/nix`, `/nix/store`, and the nft package's parent directories were `default_t`. Labeling only the executable file `bin_t` still produced systemd `203/EXEC`, so no nft table was loaded and the infraternal server remains stopped. The installer now queries the Nix runtime closures for the infraternal package and nft, sets `usr_t` on only `/nix` and `/nix/store` plus those closure roots, and sets `bin_t` on the five package executables and nft. It restores those exact paths. This does not relabel unrelated Nix store outputs or change SELinux mode. The release deploy suite passes: `Ran 24 tests in 1.063s` / `OK`. I will apply this scoped context to the current partial install and check systemd execution again.

kayg commented

2026-11-01 08:12:45 +00:00

Author

Owner

J45 checkpoint (2026-11-01 13:42 IST). A new batch-mode SSH connection from the control host to root@10.69.69.89 succeeds. On ssh, SELinux is still Enforcing; infraternal-server.service is inactive and disabled, and infraternal-firewall.service is active and enabled. nft list tables returned no tables, so I am checking the firewall unit and loaded rules before starting the application. I have not read the application journal, which may contain first-run setup material. The new public route, old DNS/route/service, and monitor01 remain untouched. The backend ACL needed for the new route is inbound on ssh wt0: source fridge-ingress 10.69.69.99, destination ssh 10.69.69.89, TCP destination port 7011; the withdrawn monitor01 .50 rule is not needed.

J45 checkpoint (2026-11-01 13:42 IST). A new batch-mode SSH connection from the control host to `root@10.69.69.89` succeeds. On `ssh`, SELinux is still Enforcing; `infraternal-server.service` is inactive and disabled, and `infraternal-firewall.service` is active and enabled. `nft list tables` returned no tables, so I am checking the firewall unit and loaded rules before starting the application. I have not read the application journal, which may contain first-run setup material. The new public route, old DNS/route/service, and monitor01 remain untouched. The backend ACL needed for the new route is inbound on ssh `wt0`: source fridge-ingress `10.69.69.99`, destination ssh `10.69.69.89`, TCP destination port `7011`; the withdrawn monitor01 `.50` rule is not needed.

kayg commented

2026-11-01 08:13:16 +00:00

Author

Owner

J45 firewall finding and correction to my 13:42 checkpoint (2026-11-01 13:43:16 IST). Root's non-login PATH has no nft command, so the bare nft list tables invocation returned no output; it did not show the ruleset. The explicit installed Nix binary /root/.nix-profile/bin/nft confirms the infraternal_host table is loaded. Inspecting its input chain exposed a real renderer issue: an unconditional iifname "wt0" accept precedes the per-port rules, so the later source/port restrictions are unreachable. The server is still stopped. I am adding a focused regression and will remove the blanket accept before starting the service. The intended firewall remains SSH TCP 22 on wt0, TCP/UDP 7443 on wt0, TCP 7011 from fridge-ingress 10.69.69.99 on wt0, and established/local/network necessities; the separate NetBird ACL is still required.

J45 firewall finding and correction to my 13:42 checkpoint (2026-11-01 13:43:16 IST). Root's non-login PATH has no `nft` command, so the bare `nft list tables` invocation returned no output; it did not show the ruleset. The explicit installed Nix binary `/root/.nix-profile/bin/nft` confirms the `infraternal_host` table is loaded. Inspecting its input chain exposed a real renderer issue: an unconditional `iifname "wt0" accept` precedes the per-port rules, so the later source/port restrictions are unreachable. The server is still stopped. I am adding a focused regression and will remove the blanket accept before starting the service. The intended firewall remains SSH TCP 22 on `wt0`, TCP/UDP 7443 on `wt0`, TCP 7011 from fridge-ingress `10.69.69.99` on `wt0`, and established/local/network necessities; the separate NetBird ACL is still required.

kayg commented

2026-11-01 08:13:58 +00:00

Author

Owner

Correction to the firewall finding above (2026-11-01 13:43:58 IST). The renderer and /etc/nftables/infraternal-host.nft both contain only the intended port/source rules; the blanket wt0 accept was present only in the active infraternal_host table. This is a stale loaded table, not a renderer defect, so no code change or regression is needed. I will validate the on-disk rules and restart only infraternal-firewall.service (it owns only inet infraternal_host), then verify the live table and a new SSH connection before starting the server.

Correction to the firewall finding above (2026-11-01 13:43:58 IST). The renderer and `/etc/nftables/infraternal-host.nft` both contain only the intended port/source rules; the blanket `wt0` accept was present only in the active `infraternal_host` table. This is a stale loaded table, not a renderer defect, so no code change or regression is needed. I will validate the on-disk rules and restart only `infraternal-firewall.service` (it owns only `inet infraternal_host`), then verify the live table and a new SSH connection before starting the server.

kayg commented

2026-11-01 08:18:57 +00:00

Author

Owner

J45 service and mesh checkpoint (2026-11-01 13:48:57 IST). The Nix package from origin/dev f4b61cd6c3f52078cad7573c5367d7a6e75f84f0 is running on ssh as root. Local /healthz returns HTTP 200 {"status":"ok"}; / returns HTTP 200 with title infraternal. The server is active/running with zero restarts. /var/lib/infraternal is 0700 root:root with var_lib_t; /run/infraternal is 0700 root:root with var_run_t. SELinux remains Enforcing; the recent AVC query returned no records. The firewall table now contains only its intended port/source rules, and a fresh control-host SSH session succeeds.

The first backend probe from fridge-ingress 10.69.69.99 over wt0 used curl http://10.69.69.89:7011/healthz and timed out after 5.002 s (HTTP 000, connect time 0). The target firewall already permits that exact source/port; this is the NetBird ACL boundary. Please add the allow with source peer fridge-ingress (10.69.69.99), destination peer ssh (10.69.69.89), protocol TCP, destination port 7011, inbound on ssh wt0. The old monitor01 .50 rule remains withdrawn. The public route, old DNS/route/service, and all monitor01 files remain untouched while this is blocked.

J45 service and mesh checkpoint (2026-11-01 13:48:57 IST). The Nix package from origin/dev `f4b61cd6c3f52078cad7573c5367d7a6e75f84f0` is running on `ssh` as root. Local `/healthz` returns HTTP 200 `{"status":"ok"}`; `/` returns HTTP 200 with title `infraternal`. The server is active/running with zero restarts. `/var/lib/infraternal` is 0700 root:root with `var_lib_t`; `/run/infraternal` is 0700 root:root with `var_run_t`. SELinux remains Enforcing; the recent AVC query returned no records. The firewall table now contains only its intended port/source rules, and a fresh control-host SSH session succeeds. The first backend probe from fridge-ingress `10.69.69.99` over `wt0` used `curl http://10.69.69.89:7011/healthz` and timed out after 5.002 s (`HTTP 000`, connect time 0). The target firewall already permits that exact source/port; this is the NetBird ACL boundary. Please add the allow with source peer **fridge-ingress (10.69.69.99)**, destination peer **ssh (10.69.69.89)**, protocol **TCP**, destination port **7011**, inbound on ssh `wt0`. The old monitor01 `.50` rule remains withdrawn. The public route, old DNS/route/service, and all monitor01 files remain untouched while this is blocked.

kayg commented

2026-11-01 08:25:15 +00:00

Author

Owner

J45 first-owner/CA instruction (2026-11-01 13:55:15 IST). The fresh instance reports setup_required=true; I checked only the public auth status and did not read or print the one-time code. The /auth/status response also confirms the first-run page is waiting for owner enrollment. After ssh.kayg.org becomes reachable, kayg must run this on ssh (the current immutable package path; it reads the code from the private local socket):

sudo /nix/store/1qhq717g6v249l5kjxm10fcn9vbhj20m-infraternal-0.1.0/bin/infraternal server setup-token --data /var/lib/infraternal

Then open https://ssh.kayg.org, enter the returned code only on the first-run screen, create the kayg owner account/passkey, and save the recovery words. After signing in, use Add host → Set up host connections → Confirm with passkey to create the new instance's fresh CA. No old account or CA was copied. I will stop before retrieving the code or completing this passkey confirmation. This is first-owner/CA creation for the fresh install; the normal “Unlock host connections” action applies to later cold starts. The public route remains held for the missing fridge-ingress-to-ssh NetBird allow recorded above.

J45 first-owner/CA instruction (2026-11-01 13:55:15 IST). The fresh instance reports `setup_required=true`; I checked only the public auth status and did not read or print the one-time code. The `/auth/status` response also confirms the first-run page is waiting for owner enrollment. After `ssh.kayg.org` becomes reachable, kayg must run this on `ssh` (the current immutable package path; it reads the code from the private local socket): `sudo /nix/store/1qhq717g6v249l5kjxm10fcn9vbhj20m-infraternal-0.1.0/bin/infraternal server setup-token --data /var/lib/infraternal` Then open `https://ssh.kayg.org`, enter the returned code only on the first-run screen, create the kayg owner account/passkey, and save the recovery words. After signing in, use **Add host → Set up host connections → Confirm with passkey** to create the new instance's fresh CA. No old account or CA was copied. I will stop before retrieving the code or completing this passkey confirmation. This is first-owner/CA creation for the fresh install; the normal “Unlock host connections” action applies to later cold starts. The public route remains held for the missing fridge-ingress-to-ssh NetBird allow recorded above.

kayg commented

2026-11-01 08:26:40 +00:00

Author

Owner

J45 release-tooling checkpoint (2026-11-01 13:56:40 IST), pushed prod-on-ssh head 65afd9eb1. The new SELinux regression failed for the expected reason: an existing /nix(/.*)? rule made substring detection choose -m for /nix, while the intended exact rule was absent ('-a' != '-m'). After exact-pattern matching, the same test passed (Ran 1 test / OK). Full release deploy suite output on compute01:

.....Release build failed for aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa; gate log retained
.........Build deferred: host capacity limit reached
...........
----------------------------------------------------------------------
Ran 25 tests in 0.903s

OK

I also reloaded only the managed inet infraternal_host table on ssh; the live chain now has no blanket wt0 accept, and a fresh SSH connection still works. The server remains active with /healthz 200 and first-owner setup required. The new public route remains held because the required NetBird allow is still missing; the old name, monitor01 route/service, and backups remain untouched. The current owner setup command/URL and passkey steps are recorded in the preceding comment.

J45 release-tooling checkpoint (2026-11-01 13:56:40 IST), pushed `prod-on-ssh` head `65afd9eb1`. The new SELinux regression failed for the expected reason: an existing `/nix(/.*)?` rule made substring detection choose `-m` for `/nix`, while the intended exact rule was absent (`'-a' != '-m'`). After exact-pattern matching, the same test passed (`Ran 1 test` / `OK`). Full release deploy suite output on compute01: ```text .....Release build failed for aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa; gate log retained .........Build deferred: host capacity limit reached ........... ---------------------------------------------------------------------- Ran 25 tests in 0.903s OK ``` I also reloaded only the managed `inet infraternal_host` table on ssh; the live chain now has no blanket `wt0` accept, and a fresh SSH connection still works. The server remains active with `/healthz` 200 and first-owner setup required. The new public route remains held because the required NetBird allow is still missing; the old name, monitor01 route/service, and backups remain untouched. The current owner setup command/URL and passkey steps are recorded in the preceding comment.

kayg commented

2026-11-01 08:27:41 +00:00

Author

Owner

J45 rollback note (2026-11-01 13:57:41 IST). The rollback point is still live: infraternal.kayg.org DNS/Traefik route and monitor01's service/data are untouched, and no state was copied. If the new public checks fail before retirement, keep the old URL and leave monitor01 as-is.

Before any later retirement, after stopping the old service, make a root-only pre-retirement backup directory under /var/backups/infraternal/<timestamp>-j45-pre-retirement/ with the consistent /var/lib/infraternal, /etc/systemd/system/infraternal-server.service, /usr/local/bin/infraternal, /usr/local/share/infraternal, and the old Traefik dynamic route file. Keep the existing backup tree too; its latest inventory entry is 20261030-113220-0c8639496, but the new snapshot is the cut-over rollback point. Record the actual backup and route backup paths and exact files removed here before deletion.

To roll back after retirement, copy the pre-retirement backup's var/lib/infraternal, etc/systemd/system/infraternal-server.service, usr/local/bin/infraternal, and usr/local/share/infraternal back to the same absolute paths with ownership/modes preserved; restore the backed-up old route, recreate infraternal.kayg.org as DNS-only A 37.157.255.66 using the Cloudflare token in-place on fridge-ingress, run systemctl daemon-reload and systemctl enable --now infraternal-server.service, and verify the old public health endpoint. Do not restore any new-instance database or CA to monitor01. The old name and service remain in place while fridge-ingress-to-ssh TCP 7011 is blocked, so none of these monitor01 changes have been made yet.

J45 rollback note (2026-11-01 13:57:41 IST). The rollback point is still live: `infraternal.kayg.org` DNS/Traefik route and monitor01's service/data are untouched, and no state was copied. If the new public checks fail before retirement, keep the old URL and leave monitor01 as-is. Before any later retirement, after stopping the old service, make a root-only pre-retirement backup directory under `/var/backups/infraternal/<timestamp>-j45-pre-retirement/` with the consistent `/var/lib/infraternal`, `/etc/systemd/system/infraternal-server.service`, `/usr/local/bin/infraternal`, `/usr/local/share/infraternal`, and the old Traefik dynamic route file. Keep the existing backup tree too; its latest inventory entry is `20261030-113220-0c8639496`, but the new snapshot is the cut-over rollback point. Record the actual backup and route backup paths and exact files removed here before deletion. To roll back after retirement, copy the pre-retirement backup's `var/lib/infraternal`, `etc/systemd/system/infraternal-server.service`, `usr/local/bin/infraternal`, and `usr/local/share/infraternal` back to the same absolute paths with ownership/modes preserved; restore the backed-up old route, recreate `infraternal.kayg.org` as DNS-only A `37.157.255.66` using the Cloudflare token in-place on fridge-ingress, run `systemctl daemon-reload` and `systemctl enable --now infraternal-server.service`, and verify the old public health endpoint. Do not restore any new-instance database or CA to monitor01. The old name and service remain in place while fridge-ingress-to-ssh TCP 7011 is blocked, so none of these monitor01 changes have been made yet.

kayg commented

2026-11-01 08:28:10 +00:00

Author

Owner

J45 tooling/docs checkpoint (2026-11-01 13:58:10 IST), pushed prod-on-ssh head a3b9f3b96. The branch now documents the production unit's exact NetBird listeners, DNS-only ssh.kayg.org, fridge-ingress websecure/cf route and its ACL prerequisite, first-owner setup command/URL, fresh CA passkey confirmation, and pre-retirement rollback/restore steps. It still keeps all old-name/service removal behind end-to-end public checks. The new server remains healthy locally and the backend ACL is still pending. I am syncing this head to compute01 and running the required scripts/verify fast; output will be recorded here verbatim when complete.

J45 tooling/docs checkpoint (2026-11-01 13:58:10 IST), pushed `prod-on-ssh` head `a3b9f3b96`. The branch now documents the production unit's exact NetBird listeners, DNS-only `ssh.kayg.org`, fridge-ingress `websecure`/`cf` route and its ACL prerequisite, first-owner setup command/URL, fresh CA passkey confirmation, and pre-retirement rollback/restore steps. It still keeps all old-name/service removal behind end-to-end public checks. The new server remains healthy locally and the backend ACL is still pending. I am syncing this head to compute01 and running the required `scripts/verify fast`; output will be recorded here verbatim when complete.

kayg commented

2026-11-01 08:30:43 +00:00

Author

Owner

J45 fridge-ingress reachability poll (2026-11-01 13:59:57 IST). From verified fridge-ingress (10.69.69.99) on wt0, curl --connect-timeout 5 --max-time 8 http://10.69.69.89:7011/healthz timed out after 5.001 s (HTTP 000, connect time 0). The local service is healthy, and the ssh firewall permits this exact source; NetBird still blocks the path. Hold the new Traefik route and old-instance cleanup. Required rule remains inbound on ssh wt0: source fridge-ingress 10.69.69.99, destination ssh 10.69.69.89, TCP destination port 7011.

J45 fridge-ingress reachability poll (2026-11-01 13:59:57 IST). From verified `fridge-ingress` (`10.69.69.99`) on `wt0`, `curl --connect-timeout 5 --max-time 8 http://10.69.69.89:7011/healthz` timed out after 5.001 s (`HTTP 000`, connect time 0). The local service is healthy, and the ssh firewall permits this exact source; NetBird still blocks the path. Hold the new Traefik route and old-instance cleanup. Required rule remains inbound on ssh `wt0`: source fridge-ingress `10.69.69.99`, destination ssh `10.69.69.89`, TCP destination port `7011`.

kayg commented

2026-11-01 08:31:48 +00:00

Author

Owner

J45 edge preflight (2026-11-01 14:01:47 IST). The control host resolves ssh.kayg.org to 49.205.172.130 and the old name to 37.157.255.66. On verified fridge-ingress (10.69.69.99), traefik.service is active and its file provider directory is /etc/traefik/dynamic. A filename-only scan found no existing ssh.kayg.org or infraternal.kayg.org route there; I did not read static config or any Cloudflare credential. No Traefik file has been copied or changed. The new route remains held until the backend TCP 7011 probe succeeds.

J45 edge preflight (2026-11-01 14:01:47 IST). The control host resolves `ssh.kayg.org` to `49.205.172.130` and the old name to `37.157.255.66`. On verified `fridge-ingress` (`10.69.69.99`), `traefik.service` is active and its file provider directory is `/etc/traefik/dynamic`. A filename-only scan found no existing `ssh.kayg.org` or `infraternal.kayg.org` route there; I did not read static config or any Cloudflare credential. No Traefik file has been copied or changed. The new route remains held until the backend TCP 7011 probe succeeds.

kayg commented

2026-11-01 08:38:54 +00:00

Author

Owner

J45 fresh-host verification and reachability poll (2026-11-01 14:08 IST). On ssh, infraternal-server.service is active/running and now enabled, with NRestarts=0 and ExecMainStatus=0; the listener-bound /healthz and / both return HTTP 200, and SELinux remains Enforcing. The service had been active but disabled, so I enabled it for boot persistence.

At 14:07 IST, a verified SSH session to fridge-ingress (10.69.69.99) ran curl to http://10.69.69.89:7011/healthz; it timed out (HTTP 000, connect timeout 3 s). The ssh host firewall already permits this exact source/port. The required NetBird allow remains inbound on ssh wt0: source fridge-ingress 10.69.69.99, destination ssh 10.69.69.89, protocol TCP, destination port 7011.

The new Traefik route remains held, and the old infraternal.kayg.org DNS/route, monitor01 service, and backups remain untouched. No secrets were read or posted.

J45 fresh-host verification and reachability poll (2026-11-01 14:08 IST). On `ssh`, `infraternal-server.service` is active/running and now enabled, with `NRestarts=0` and `ExecMainStatus=0`; the listener-bound `/healthz` and `/` both return HTTP 200, and SELinux remains Enforcing. The service had been active but disabled, so I enabled it for boot persistence. At 14:07 IST, a verified SSH session to fridge-ingress (`10.69.69.99`) ran `curl` to `http://10.69.69.89:7011/healthz`; it timed out (`HTTP 000`, connect timeout 3 s). The ssh host firewall already permits this exact source/port. The required NetBird allow remains inbound on ssh `wt0`: source fridge-ingress `10.69.69.99`, destination ssh `10.69.69.89`, protocol TCP, destination port `7011`. The new Traefik route remains held, and the old `infraternal.kayg.org` DNS/route, monitor01 service, and backups remain untouched. No secrets were read or posted.

kayg commented

2026-11-01 08:56:53 +00:00

Author

Owner

J45 update-path implementation start (2026-11-01, about 14:25 IST). Branch prod-on-ssh starts from origin/dev base f4b61cd6c3; current branch head is a3b9f3b96. I am adding a recurring Nix update mode separate from --fresh-nix-package: timestamped online state backup with retention, side-by-side rooted package output, atomic unit/web switch, health check, and automatic package + state restore on failure. R2/D217 handover remains the preference, but the current Nix source package has no signed MANIFEST/MANIFEST.sig; restart admission refuses it, so this deployment must cold-start and will require the owner passkey unlock unless the candidate becomes handover-eligible. The fridge-ingress TCP 7011 probe timed out at 14:24 IST; no route or monitor01 changes made. Next: add failing focused tests, implement the updater, document both paths, then run the full gate via compute01 build-slot.

J45 update-path implementation start (2026-11-01, about 14:25 IST). Branch prod-on-ssh starts from origin/dev base f4b61cd6c3f52078cad7573c5367d7a6e75f84f0; current branch head is a3b9f3b96. I am adding a recurring Nix update mode separate from --fresh-nix-package: timestamped online state backup with retention, side-by-side rooted package output, atomic unit/web switch, health check, and automatic package + state restore on failure. R2/D217 handover remains the preference, but the current Nix source package has no signed MANIFEST/MANIFEST.sig; restart admission refuses it, so this deployment must cold-start and will require the owner passkey unlock unless the candidate becomes handover-eligible. The fridge-ingress TCP 7011 probe timed out at 14:24 IST; no route or monitor01 changes made. Next: add failing focused tests, implement the updater, document both paths, then run the full gate via compute01 build-slot.

kayg commented

2026-11-01 09:09:52 +00:00

Author

Owner

J45 update-path finding (2026-11-01, about 14:36 IST). The focused deployment suite ran 29 tests and found an automatic-rollback cleanup failure: the staged Nix release is intentionally read-only (0555 directories/0444 files), and pruning it with plain shutil.rmtree raised PermissionError while removing the failed candidate. The state restore itself completed before cleanup failed. I am changing only managed version-directory cleanup to add owner write permission before removal; the Nix store remains untouched. The fridge-ingress path is still blocked, so no route or monitor01 changes have been made.

kayg commented

2026-11-01 09:11:41 +00:00

Author

Owner

J45 update-path finding resolved (2026-11-01, about 14:42 IST). Cleanup now retries only PermissionError by making the owned managed package path writable before deleting it; it never changes Nix store permissions. The full focused deployment suite on compute01 reran after the change: Ran 29 tests in 1.493s; OK. I am continuing with CLI/update transport tests and review of atomic current-package switching.

kayg commented

2026-11-01 09:19:40 +00:00

Author

Owner

J45 update-path checkpoint (2026-11-01 14:49 IST). I resumed from branch prod-on-ssh at a3b9f3b96, reviewed the updater and #274 history, and am finishing the documented update path plus rollback failure reporting. Fridge-ingress was verified through its pinned NetBird IP 10.69.69.99 because its DNS name has no matching known-host entry. Probe to http://10.69.69.89:7011/healthz timed out after 5.003 s (HTTP 000, connect 0.000 s). The backend allow is still missing; keep the new public route held and monitor01 untouched. Required rule remains source fridge-ingress 10.69.69.99 to destination ssh 10.69.69.89, TCP destination port 7011.

J45 update-path checkpoint (2026-11-01 14:49 IST). I resumed from branch prod-on-ssh at a3b9f3b96, reviewed the updater and #274 history, and am finishing the documented update path plus rollback failure reporting. Fridge-ingress was verified through its pinned NetBird IP 10.69.69.99 because its DNS name has no matching known-host entry. Probe to http://10.69.69.89:7011/healthz timed out after 5.003 s (HTTP 000, connect 0.000 s). The backend allow is still missing; keep the new public route held and monitor01 untouched. Required rule remains source fridge-ingress 10.69.69.99 to destination ssh 10.69.69.89, TCP destination port 7011.

kayg commented

2026-11-01 09:21:22 +00:00

Author

Owner

J45 update-path rollback-reporting finding (2026-11-01 14:50 IST). A new focused regression on compute01 fails as expected: test_nix_update_reports_owner_unlock_if_automatic_rollback_fails reaches the generic failed result, which omits ca_handover and the owner passkey action if automatic rollback itself fails. I will return an explicit cold-rollback failure record with the same exact unlock instructions while keeping service/backup diagnostics secret-free. The ingress probe at 14:49 IST still timed out (HTTP 000, connect 0.000 s); no route or monitor01 changes.

J45 update-path rollback-reporting finding (2026-11-01 14:50 IST). A new focused regression on compute01 fails as expected: `test_nix_update_reports_owner_unlock_if_automatic_rollback_fails` reaches the generic failed result, which omits `ca_handover` and the owner passkey action if automatic rollback itself fails. I will return an explicit cold-rollback failure record with the same exact unlock instructions while keeping service/backup diagnostics secret-free. The ingress probe at 14:49 IST still timed out (HTTP 000, connect 0.000 s); no route or monitor01 changes.

kayg commented

2026-11-01 09:21:55 +00:00

Author

Owner

J45 rollback-reporting finding resolved (2026-11-01 14:52 IST). The new regression is GREEN on compute01 (Ran 1 test / OK). If health-check rollback cannot be completed, the remote helper now returns a secret-free failed record that still marks the path cold, names the attempted previous package/state restore, and prints the owner passkey unlock action plus the recovery step. I am adding the fresh-bootstrap and recurring-update commands to the deploy docs next.

J45 rollback-reporting finding resolved (2026-11-01 14:52 IST). The new regression is GREEN on compute01 (`Ran 1 test` / `OK`). If health-check rollback cannot be completed, the remote helper now returns a secret-free failed record that still marks the path cold, names the attempted previous package/state restore, and prints the owner passkey unlock action plus the recovery step. I am adding the fresh-bootstrap and recurring-update commands to the deploy docs next.

kayg commented

2026-11-01 09:24:53 +00:00

Author

Owner

J45 backup crash-durability finding (2026-11-01 14:55 IST). A focused regression on compute01 fails: create_backup writes COMPLETE without first fsyncing every nested snapshot file/directory, and does not fsync the temporary snapshot directory before its final rename. A power loss could therefore leave a completion marker for data that was not made durable. I am adding bottom-up snapshot fsync and syncing the marker directory before publication; the source snapshot and final rename remain on the backup filesystem.

J45 backup crash-durability finding (2026-11-01 14:55 IST). A focused regression on compute01 fails: `create_backup` writes `COMPLETE` without first fsyncing every nested snapshot file/directory, and does not fsync the temporary snapshot directory before its final rename. A power loss could therefore leave a completion marker for data that was not made durable. I am adding bottom-up snapshot fsync and syncing the marker directory before publication; the source snapshot and final rename remain on the backup filesystem.

kayg commented

2026-11-01 09:25:38 +00:00

Author

Owner

J45 backup crash-durability finding resolved (2026-11-01 14:59 IST). The remote snapshot helper now fsyncs each regular file and nested directory bottom-up before COMPLETE, then fsyncs the temporary backup directory before its atomic rename. The previously failing durability regression and the two rollback-unlock regressions all pass on compute01: Ran 3 tests in 0.280s / OK. Fresh bootstrap, recurring update, retention, warm/cold handover, owner unlock, and health rollback are now documented in the ssh, deploy, and Nix package runbooks. I am running the full focused deploy suite and then scripts/verify fast via build-slot.

J45 backup crash-durability finding resolved (2026-11-01 14:59 IST). The remote snapshot helper now fsyncs each regular file and nested directory bottom-up before `COMPLETE`, then fsyncs the temporary backup directory before its atomic rename. The previously failing durability regression and the two rollback-unlock regressions all pass on compute01: `Ran 3 tests in 0.280s` / `OK`. Fresh bootstrap, recurring update, retention, warm/cold handover, owner unlock, and health rollback are now documented in the ssh, deploy, and Nix package runbooks. I am running the full focused deploy suite and then `scripts/verify fast` via build-slot.

kayg commented

2026-11-01 09:26:16 +00:00

Author

Owner

J45 fridge-ingress reachability poll (2026-11-01 14:56:11 IST). HTTP 000 connect 0.000000s total 5.002624s
curl: (28) Connection timed out after 5002 milliseconds

J45 fridge-ingress reachability poll (2026-11-01 14:56:11 IST). HTTP 000 connect 0.000000s total 5.002624s curl: (28) Connection timed out after 5002 milliseconds

kayg commented

2026-11-01 09:31:22 +00:00

Author

Owner

J45 fridge-ingress reachability poll (2026-11-01 15:01:16 IST). HTTP 000 connect 0.000000s total 5.002303s
curl: (28) Connection timed out after 5002 milliseconds

J45 fridge-ingress reachability poll (2026-11-01 15:01:16 IST). HTTP 000 connect 0.000000s total 5.002303s curl: (28) Connection timed out after 5002 milliseconds

kayg commented

2026-11-01 09:36:28 +00:00

Author

Owner

J45 fridge-ingress reachability poll (2026-11-01 15:06:22 IST). HTTP 000 connect 0.000000s total 5.001730s
curl: (28) Connection timed out after 5002 milliseconds

J45 fridge-ingress reachability poll (2026-11-01 15:06:22 IST). HTTP 000 connect 0.000000s total 5.001730s curl: (28) Connection timed out after 5002 milliseconds

kayg commented

2026-11-01 09:36:47 +00:00

Author

Owner

J45 update-path review findings (2026-11-01, about 15:06 IST). The read-only review identified four safety gaps I am fixing before the gate: (1) concurrent update invocations can race shared pointers and snapshots because the remote helper has no exclusive lock; (2) a fresh instance has a real /var/lib/infraternal/setup-token.sock Unix socket (runtime.rs), which the backup rejects even though it is ephemeral and should be skipped only after type validation; (3) RESTARTING only acknowledges the sender prepared an in-memory handover, while the successor may still start CA-locked, so HTTP health cannot substantiate ca_handover=warm; the result must say unverified and tell the owner how to check/unlock; (4) release pruning errors after a cold rollback or malformed retained metadata can escape and replace the rollback record with a generic failure that omits the backup and owner action. I am adding regressions for these cases; no production route or monitor01 changes.

J45 update-path review findings (2026-11-01, about 15:06 IST). The read-only review identified four safety gaps I am fixing before the gate: (1) concurrent update invocations can race shared pointers and snapshots because the remote helper has no exclusive lock; (2) a fresh instance has a real `/var/lib/infraternal/setup-token.sock` Unix socket (`runtime.rs`), which the backup rejects even though it is ephemeral and should be skipped only after type validation; (3) `RESTARTING` only acknowledges the sender prepared an in-memory handover, while the successor may still start CA-locked, so HTTP health cannot substantiate `ca_handover=warm`; the result must say unverified and tell the owner how to check/unlock; (4) release pruning errors after a cold rollback or malformed retained metadata can escape and replace the rollback record with a generic failure that omits the backup and owner action. I am adding regressions for these cases; no production route or monitor01 changes.

kayg commented

2026-11-01 09:39:54 +00:00

Author

Owner

J45 review regressions reproduced on compute01 (2026-11-01, about 15:11 IST): the setup-token socket backup raised state contains a link or special file; concurrent invocation reached apply; stage and restore did not fsync their trees; releases symlink staged outside the managed tree; RESTARTING was classified as warm; malformed retained JSON and cleanup OSError escaped after deployment/rollback. The focused run reports Ran 8 tests with 5 failures and 3 errors, all matching these gaps. I cancelled my queued gate before it acquired a build slot because the reviewed head needs these fixes; I will rerun it after the regressions pass.

J45 review regressions reproduced on compute01 (2026-11-01, about 15:11 IST): the setup-token socket backup raised `state contains a link or special file`; concurrent invocation reached `apply`; stage and restore did not fsync their trees; `releases` symlink staged outside the managed tree; `RESTARTING` was classified as warm; malformed retained JSON and cleanup OSError escaped after deployment/rollback. The focused run reports `Ran 8 tests` with 5 failures and 3 errors, all matching these gaps. I cancelled my queued gate before it acquired a build slot because the reviewed head needs these fixes; I will rerun it after the regressions pass.

kayg commented

2026-11-01 09:41:33 +00:00

Author

Owner

J45 fridge-ingress reachability poll (2026-11-01 15:11:28 IST). curl: (28) Connection timed out after 5002 milliseconds
HTTP 000 connect 0.000000s total 5.002712s

J45 fridge-ingress reachability poll (2026-11-01 15:11:28 IST). curl: (28) Connection timed out after 5002 milliseconds HTTP 000 connect 0.000000s total 5.002712s

kayg commented

2026-11-01 09:46:39 +00:00

Author

Owner

J45 fridge-ingress reachability poll (2026-11-01 15:16:33 IST). curl: (28) Connection timed out after 5002 milliseconds
HTTP 000 connect 0.000000s total 5.002549s

J45 fridge-ingress reachability poll (2026-11-01 15:16:33 IST). curl: (28) Connection timed out after 5002 milliseconds HTTP 000 connect 0.000000s total 5.002549s

kayg commented

2026-11-01 09:47:46 +00:00

Author

Owner

J45 continuation (2026-11-01 IST): resuming prod-on-ssh from checkpoint 62e152bf5, based on origin/dev f4b61cd6c3f52078cad7573c5367d7a6e75f84f0. The state-preserving Nix update path and review regressions are in progress; fresh bootstrap remains separate. Current review finding: several integration tests mock os.geteuid() to simulate root, which also makes production ownership checks reject test-owned temporary directories. I’m adding a dedicated root preflight helper and switching those tests to mock only that boundary, while retaining the real ownership checks. Next: focused deploy suite, scripts/verify fast via compute01 build-slot, branch commit/push. No public route or monitor01 changes; fridge-ingress → ssh:7011 still times out. No secrets in this update.

J45 continuation (2026-11-01 IST): resuming `prod-on-ssh` from checkpoint `62e152bf5`, based on `origin/dev` `f4b61cd6c3f52078cad7573c5367d7a6e75f84f0`. The state-preserving Nix update path and review regressions are in progress; fresh bootstrap remains separate. Current review finding: several integration tests mock `os.geteuid()` to simulate root, which also makes production ownership checks reject test-owned temporary directories. I’m adding a dedicated root preflight helper and switching those tests to mock only that boundary, while retaining the real ownership checks. Next: focused deploy suite, `scripts/verify fast` via compute01 build-slot, branch commit/push. No public route or monitor01 changes; fridge-ingress → ssh:7011 still times out. No secrets in this update.

kayg commented

2026-11-01 09:51:45 +00:00

Author

Owner

J45 fridge-ingress reachability poll (2026-11-01 15:21:39 IST). curl: (28) Connection timed out after 5002 milliseconds
HTTP 000 connect 0.000000s total 5.002291s

J45 fridge-ingress reachability poll (2026-11-01 15:21:39 IST). curl: (28) Connection timed out after 5002 milliseconds HTTP 000 connect 0.000000s total 5.002291s

kayg commented

2026-11-01 09:52:30 +00:00

Author

Owner

J45 update-path finding and fix (2026-11-01 IST): isolated _require_root() now owns the privilege check, so integration tests no longer mock os.geteuid() across the directory ownership checks. The first full run exposed fixture roots created as 0775 under compute01's umask 0002; the updater correctly rejected them as group-writable. I set those fixtures to 0700, matching the deployment-root contract. Fresh compute01 run: Ran 42 tests in 2.640s / OK. Next I’m reviewing the full updater/docs diff, then running scripts/verify fast through ~/infraternal-ops/build-slot. Public ingress remains held; fridge-ingress to ssh:7011 was still timing out at the last poll.

J45 update-path finding and fix (2026-11-01 IST): isolated `_require_root()` now owns the privilege check, so integration tests no longer mock `os.geteuid()` across the directory ownership checks. The first full run exposed fixture roots created as 0775 under compute01's umask 0002; the updater correctly rejected them as group-writable. I set those fixtures to 0700, matching the deployment-root contract. Fresh compute01 run: `Ran 42 tests in 2.640s` / `OK`. Next I’m reviewing the full updater/docs diff, then running `scripts/verify fast` through `~/infraternal-ops/build-slot`. Public ingress remains held; fridge-ingress to ssh:7011 was still timing out at the last poll.

kayg commented

2026-11-01 09:54:12 +00:00

Author

Owner

J45 owner direction change (2026-11-01 IST): switching ingress verification from the blocked NetBird route to the confirmed private LAN path. The five-minute NetBird probe was stopped after its 15:21 IST timeout. I’m checking listener configuration and the current ssh firewall before applying 10.69.0.89:7011 with a source-only allow for fridge-ingress 10.69.0.99; then I’ll back up fridge-ingress Traefik config and route only ssh.kayg.org to the private backend. I will keep the 10.69.69.89 listener intact and leave monitor01 untouched until ssh.kayg.org passes end-to-end TLS, sign-in, and WebSocket checks. No secrets in this update.

J45 owner direction change (2026-11-01 IST): switching ingress verification from the blocked NetBird route to the confirmed private LAN path. The five-minute NetBird probe was stopped after its 15:21 IST timeout. I’m checking listener configuration and the current ssh firewall before applying `10.69.0.89:7011` with a source-only allow for fridge-ingress `10.69.0.99`; then I’ll back up fridge-ingress Traefik config and route only `ssh.kayg.org` to the private backend. I will keep the 10.69.69.89 listener intact and leave monitor01 untouched until ssh.kayg.org passes end-to-end TLS, sign-in, and WebSocket checks. No secrets in this update.

kayg commented

2026-11-01 09:57:35 +00:00

Author

Owner

J45 private-ingress findings (2026-11-01 IST): the server currently binds only 10.69.69.89:7011, and its CLI accepts one SocketAddr; it has no TLS certificate/key options in the active unit, so Traefik will use plaintext HTTP to the backend while terminating public TLS. ssh.kayg.org currently resolves to the requested origin A address 49.205.172.130; fridge-ingress is 10.69.0.99/24 on eth0. The private backend probe currently times out because ssh is not bound to 10.69.0.89. Live host checks show firewalld is absent (LoadState=not-found, no firewall-cmd), so I will provision it through Nix per the package rule and stage a private-LAN drop policy with only the requested rich allow for TCP 7011 from 10.69.0.99; the NetBird listener path will remain available. No host config has changed yet.

J45 private-ingress findings (2026-11-01 IST): the server currently binds only `10.69.69.89:7011`, and its CLI accepts one `SocketAddr`; it has no TLS certificate/key options in the active unit, so Traefik will use plaintext HTTP to the backend while terminating public TLS. `ssh.kayg.org` currently resolves to the requested origin A address `49.205.172.130`; fridge-ingress is `10.69.0.99/24` on eth0. The private backend probe currently times out because ssh is not bound to `10.69.0.89`. Live host checks show firewalld is absent (`LoadState=not-found`, no `firewall-cmd`), so I will provision it through Nix per the package rule and stage a private-LAN drop policy with only the requested rich allow for TCP 7011 from `10.69.0.99`; the NetBird listener path will remain available. No host config has changed yet.

kayg commented

2026-11-01 10:03:52 +00:00

Author

Owner

J45 Cloudflare finding (2026-11-01 IST): using the token file configured for Traefik in-process (token value not printed), the Cloudflare API confirms ssh.kayg.org already has A 49.205.172.130 with proxied=false; no DNS write is needed. The zone has 9 proxied and 66 DNS-only records, and the new host record matches the existing DNS-only mode. The configured Traefik resolver is cf with Cloudflare DNS-01. The service has no backend TLS flags, so the route will terminate public TLS at Traefik and use HTTP to the private backend.

J45 Cloudflare finding (2026-11-01 IST): using the token file configured for Traefik in-process (token value not printed), the Cloudflare API confirms `ssh.kayg.org` already has A `49.205.172.130` with `proxied=false`; no DNS write is needed. The zone has 9 proxied and 66 DNS-only records, and the new host record matches the existing DNS-only mode. The configured Traefik resolver is `cf` with Cloudflare DNS-01. The service has no backend TLS flags, so the route will terminate public TLS at Traefik and use HTTP to the private backend.

kayg commented

2026-11-01 10:04:32 +00:00

Author

Owner

J45 backup review finding (2026-11-01 IST): the live restart control socket is specifically /var/lib/infraternal/restart/control.sock (crates/infraternal/src/restart.rs), but the backup helper currently skips every entry whose first path component is restart. That could silently omit a regular persisted file placed there. I’m adding a regression to preserve regular restart-directory files while excluding only the verified control socket; no service or host data has been changed.

J45 backup review finding (2026-11-01 IST): the live restart control socket is specifically `/var/lib/infraternal/restart/control.sock` (`crates/infraternal/src/restart.rs`), but the backup helper currently skips every entry whose first path component is `restart`. That could silently omit a regular persisted file placed there. I’m adding a regression to preserve regular restart-directory files while excluding only the verified control socket; no service or host data has been changed.

kayg commented

2026-11-01 10:07:23 +00:00

Author

Owner

J45 update-path review fixes (2026-11-01 15:37 IST): the new restart-directory regression first failed because backup_state omitted restart/persisted-state while the live restart/control.sock was present. The snapshot now preserves regular files below /var/lib/infraternal/restart and excludes only the verified control.sock; setup-token socket validation remains strict. The root guard is isolated from ownership checks, and private-mode fixtures match the protected deployment directory contract. Fresh compute01 deployment suite: Ran 43 tests in 2.830s / OK. The Nix-built firewalld package request is queued at build-slot position 10; no ingress, unit, or firewall config has been changed yet.

J45 update-path review fixes (2026-11-01 15:37 IST): the new restart-directory regression first failed because `backup_state` omitted `restart/persisted-state` while the live `restart/control.sock` was present. The snapshot now preserves regular files below `/var/lib/infraternal/restart` and excludes only the verified `control.sock`; setup-token socket validation remains strict. The root guard is isolated from ownership checks, and private-mode fixtures match the protected deployment directory contract. Fresh compute01 deployment suite: `Ran 43 tests in 2.830s` / `OK`. The Nix-built firewalld package request is queued at build-slot position 10; no ingress, unit, or firewall config has been changed yet.

kayg commented

2026-11-01 10:59:34 +00:00

Author

Owner

J45 continuation (2026-11-01 15:45 IST): branch prod-on-ssh is at 62e152bf5, based on the previously selected origin/dev f4b61cd6c. The current fetched origin/dev is 7bf03044f; I will integrate it before the production build. The focused safe-update deployment suite passes on compute01: Ran 43 tests in 3.046s / OK.

Private-LAN findings: the CLI accepts one HTTP socket today; the production unit must retain 10.69.69.89:7011 and add 10.69.0.89:7011, with the trusted proxy narrowed to fridge-ingress 10.69.0.99. firewalld is absent on ssh. The saved nft policy is source/port scoped, but the active table contains an extra unconditional wt0 accept; I am correcting and verifying the active rules before opening the LAN listener. NetBird polling stopped at 15:21 IST. No Traefik route or monitor01 state has changed.

Next: add and test the explicit secondary listener and LAN policy, then provision the Nix-built firewalld package and prove fridge-ingress can reach the private backend. No secrets included.

J45 continuation (2026-11-01 15:45 IST): branch `prod-on-ssh` is at `62e152bf5`, based on the previously selected `origin/dev` `f4b61cd6c`. The current fetched `origin/dev` is `7bf03044f`; I will integrate it before the production build. The focused safe-update deployment suite passes on compute01: `Ran 43 tests in 3.046s` / `OK`. Private-LAN findings: the CLI accepts one HTTP socket today; the production unit must retain `10.69.69.89:7011` and add `10.69.0.89:7011`, with the trusted proxy narrowed to fridge-ingress `10.69.0.99`. `firewalld` is absent on ssh. The saved nft policy is source/port scoped, but the active table contains an extra unconditional `wt0 accept`; I am correcting and verifying the active rules before opening the LAN listener. NetBird polling stopped at 15:21 IST. No Traefik route or monitor01 state has changed. Next: add and test the explicit secondary listener and LAN policy, then provision the Nix-built firewalld package and prove fridge-ingress can reach the private backend. No secrets included.

kayg commented

2026-11-01 11:10:57 +00:00

Author

Owner

J45 continuation (2026-11-01 16:41 IST): continuing on branch prod-on-ssh, head 62e152bf5. The current fetched origin/dev recorded on the branch is 7bf03044f; I will fetch again before the final production build and integrate the latest head. The prior fresh install on ssh remains locally healthy, but it still has only the NetBird HTTP listener; no LAN bind or LAN firewall change has been applied.

The explicit-listener test is queued on compute01 behind its four occupied build slots (queue position 14 at 16:39 IST); no compile has started. The new firewall regression is red because the renderer still emits the prior NetBird rule. I am finishing those fixes and the Nix-managed firewalld package before touching the listener or firewall. NetBird polling remains stopped. The public route, DNS, and all monitor01 state remain untouched. No secrets included.

J45 continuation (2026-11-01 16:41 IST): continuing on branch `prod-on-ssh`, head `62e152bf5`. The current fetched `origin/dev` recorded on the branch is `7bf03044f`; I will fetch again before the final production build and integrate the latest head. The prior fresh install on `ssh` remains locally healthy, but it still has only the NetBird HTTP listener; no LAN bind or LAN firewall change has been applied. The explicit-listener test is queued on compute01 behind its four occupied build slots (queue position 14 at 16:39 IST); no compile has started. The new firewall regression is red because the renderer still emits the prior NetBird rule. I am finishing those fixes and the Nix-managed firewalld package before touching the listener or firewall. NetBird polling remains stopped. The public route, DNS, and all monitor01 state remain untouched. No secrets included.

kayg commented

2026-11-01 11:20:33 +00:00

Author

Owner

J45 private-LAN firewall finding (2026-11-01 16:50 IST), on branch prod-on-ssh, merged with origin/dev 7bf03044f at e598e5ec8. The active ssh nft input chain has policy drop, but it also contains an unconditional iifname "wt0" accept before the explicit per-port rules; the saved /etc/nftables file does not contain that rule. This is active-table drift and makes those later NetBird source/port restrictions ineffective. firewalld.service is inactive and no firewall-cmd is installed.

I will remove only the unconditional rule by reloading the managed inet infraternal_host table from its saved file, then verify management SSH still works and only explicit ports/sources remain. I will add the requested firewalld rich rule for source 10.69.0.99/32, destination 10.69.0.89/32, TCP 7011; the Nix firewall table will enforce the same source and destination. No monitor01 or Traefik change has occurred.

J45 private-LAN firewall finding (2026-11-01 16:50 IST), on branch `prod-on-ssh`, merged with `origin/dev` `7bf03044f` at `e598e5ec8`. The active ssh nft input chain has policy drop, but it also contains an unconditional `iifname "wt0" accept` before the explicit per-port rules; the saved /etc/nftables file does not contain that rule. This is active-table drift and makes those later NetBird source/port restrictions ineffective. `firewalld.service` is inactive and no `firewall-cmd` is installed. I will remove only the unconditional rule by reloading the managed `inet infraternal_host` table from its saved file, then verify management SSH still works and only explicit ports/sources remain. I will add the requested firewalld rich rule for source `10.69.0.99/32`, destination `10.69.0.89/32`, TCP `7011`; the Nix firewall table will enforce the same source and destination. No monitor01 or Traefik change has occurred.

kayg commented

2026-11-01 11:24:51 +00:00

Author

Owner

J45 private-ingress contract regression (2026-11-01 16:55 IST). Three focused deployment checks ran on compute01 against pushed test head a111f5f98 and failed as expected: Ran 3 tests, FAILED (failures=2, errors=1). Evidence: the firewalld rich-rule renderer is absent; deploy help has no --firewalld-package; and rewrite_unit cannot add the private listener/trusted proxy to the candidate while preserving the previous flags for rollback. These are the intended red cases before implementation. The queued copy of this quick test was stopped before it acquired a slot; the same focused tests ran directly as a light unit check. No production service, firewall, route, or monitor01 state was changed.

J45 private-ingress contract regression (2026-11-01 16:55 IST). Three focused deployment checks ran on compute01 against pushed test head `a111f5f98` and failed as expected: `Ran 3 tests`, `FAILED (failures=2, errors=1)`. Evidence: the firewalld rich-rule renderer is absent; deploy help has no `--firewalld-package`; and `rewrite_unit` cannot add the private listener/trusted proxy to the candidate while preserving the previous flags for rollback. These are the intended red cases before implementation. The queued copy of this quick test was stopped before it acquired a slot; the same focused tests ran directly as a light unit check. No production service, firewall, route, or monitor01 state was changed.

kayg commented

2026-11-01 11:33:32 +00:00

Author

Owner

Continuation on prod-on-ssh, current pushed head a111f5f98, based on origin/dev 7bf03044f. I am reviewing the in-progress state-preserving updater and adding the approved private-LAN ingress path (10.69.0.99 → 10.69.0.89:7011). No production host or Traefik changes have been made in this continuation yet; next I am validating firewalld scoping and whether listener changes can use the in-memory signer handover safely.

Continuation on `prod-on-ssh`, current pushed head `a111f5f98`, based on `origin/dev` `7bf03044f`. I am reviewing the in-progress state-preserving updater and adding the approved private-LAN ingress path (`10.69.0.99` → `10.69.0.89:7011`). No production host or Traefik changes have been made in this continuation yet; next I am validating firewalld scoping and whether listener changes can use the in-memory signer handover safely.

kayg commented

2026-11-01 11:49:27 +00:00

Author

Owner

Implementation progress on prod-on-ssh based on origin/dev 7bf03044f:

The server unit now adds 10.69.0.89:7011 while retaining the NetBird listener; recurring updates replace the candidate proxy/listener flags and retain the old flags for rollback.
A listener/proxy argv change uses a cold systemd restart because in-memory exec preserves the old argv. Ordinary eligible updates with unchanged argv still use signer handover.
The Nix firewalld+nftables output uses separate default-drop zones for eth0 and wt0; no broad LAN 7011 allow is present. Exact private rule:
rule family="ipv4" source address="10.69.0.99/32" destination address="10.69.0.89/32" port port="7011" protocol="tcp" accept
deploy_test.py: Ran 47 tests / OK (the fixture emitted its expected release-failure and capacity-deferred messages). No host configuration or public route has changed; the NetBird polling remains stopped.

Next: run scripts/verify fast on compute01, then push the branch and apply the approved host update. No setup code or other secrets are in this comment.

Implementation progress on `prod-on-ssh` based on `origin/dev` `7bf03044f`: - The server unit now adds `10.69.0.89:7011` while retaining the NetBird listener; recurring updates replace the candidate proxy/listener flags and retain the old flags for rollback. - A listener/proxy argv change uses a cold systemd restart because in-memory exec preserves the old argv. Ordinary eligible updates with unchanged argv still use signer handover. - The Nix firewalld+nftables output uses separate default-drop zones for `eth0` and `wt0`; no broad LAN 7011 allow is present. Exact private rule: `rule family="ipv4" source address="10.69.0.99/32" destination address="10.69.0.89/32" port port="7011" protocol="tcp" accept` - `deploy_test.py`: `Ran 47 tests` / `OK` (the fixture emitted its expected release-failure and capacity-deferred messages). No host configuration or public route has changed; the NetBird polling remains stopped. Next: run `scripts/verify fast` on compute01, then push the branch and apply the approved host update. No setup code or other secrets are in this comment.

kayg commented

2026-11-01 11:53:48 +00:00

Author

Owner

Finding with live evidence:

ssh.kayg.org resolves to 49.205.172.130 from both the control host and fridge-ingress, matching the DNS-only record; Traefik is active on fridge-ingress, and there is no ssh hostname route in /etc/traefik/dynamic yet.
The current ssh service answers /healthz with HTTP 200 on 10.69.69.89:7011; the unit still lacks --listen-additional 10.69.0.89:7011.
The active owned nft table contains iifname "wt0" ... accept with no source restriction. The install will back up the existing owned table and replace it with scoped wt0 rules plus the exact eth0 rich rule already recorded above.
No public route or monitor01 change has been made. The required fast gate is still waiting at compute01 build-slot position 17.

No secrets were read or posted.

Finding with live evidence: - `ssh.kayg.org` resolves to `49.205.172.130` from both the control host and fridge-ingress, matching the DNS-only record; Traefik is active on fridge-ingress, and there is no ssh hostname route in `/etc/traefik/dynamic` yet. - The current ssh service answers `/healthz` with HTTP 200 on `10.69.69.89:7011`; the unit still lacks `--listen-additional 10.69.0.89:7011`. - The active owned nft table contains `iifname "wt0" ... accept` with no source restriction. The install will back up the existing owned table and replace it with scoped `wt0` rules plus the exact `eth0` rich rule already recorded above. - No public route or monitor01 change has been made. The required fast gate is still waiting at compute01 build-slot position 17. No secrets were read or posted.

kayg commented

2026-11-01 11:59:13 +00:00

Author

Owner

Follow-up finding: the firewall unit references an immutable Nix store output, so its closure now receives a persistent Nix GC root before the unit is activated. The regression asserts this. Fresh deploy_test.py result: Ran 47 tests / OK (same expected fixture messages). scripts/verify fast remains queued on compute01 at position 17; no production or Traefik change has been applied.

Follow-up finding: the firewall unit references an immutable Nix store output, so its closure now receives a persistent Nix GC root before the unit is activated. The regression asserts this. Fresh `deploy_test.py` result: `Ran 47 tests` / `OK` (same expected fixture messages). `scripts/verify fast` remains queued on compute01 at position 17; no production or Traefik change has been applied.

kayg commented

2026-11-01 12:03:36 +00:00

Author

Owner

J45 continuation (2026-11-01 IST): I checked the live ssh unit and corrected one code-path assumption from the handoff. The Rust CLI already accepts repeated --listen-additional arguments, and the HTTP runtime binds the primary and additional addresses (crates/infraternal/src/main.rs, crates/infraternal-server/src/runtime.rs); no Rust change is needed.

Current ssh evidence: infraternal-server.service is active on 10.69.69.89:7011, /healthz returns 200, and SELinux is Enforcing. It still has the old single-listener argv and trusted proxy 10.69.69.99, so it is not yet reachable on the private address. Firewalld is not currently installed; the deployment will install its Nix-built output and the narrowly scoped policy.

The x86 scripts/verify fast job is still queued through compute01 build-slot (log /tmp/j45-fast.log, queue position 24 at the latest sample); no result is claimed. NetBird polling remains stopped. No Traefik, DNS, or monitor01 change has occurred. Next: finish review of the queued branch changes, let the gate complete, then apply and verify the private listener/firewall before creating the ssh.kayg.org Traefik route.

J45 continuation (2026-11-01 IST): I checked the live ssh unit and corrected one code-path assumption from the handoff. The Rust CLI already accepts repeated `--listen-additional` arguments, and the HTTP runtime binds the primary and additional addresses (`crates/infraternal/src/main.rs`, `crates/infraternal-server/src/runtime.rs`); no Rust change is needed. Current ssh evidence: `infraternal-server.service` is active on `10.69.69.89:7011`, `/healthz` returns 200, and SELinux is Enforcing. It still has the old single-listener argv and trusted proxy `10.69.69.99`, so it is not yet reachable on the private address. Firewalld is not currently installed; the deployment will install its Nix-built output and the narrowly scoped policy. The x86 `scripts/verify fast` job is still queued through compute01 `build-slot` (log `/tmp/j45-fast.log`, queue position 24 at the latest sample); no result is claimed. NetBird polling remains stopped. No Traefik, DNS, or monitor01 change has occurred. Next: finish review of the queued branch changes, let the gate complete, then apply and verify the private listener/firewall before creating the `ssh.kayg.org` Traefik route.

kayg commented

2026-11-01 12:13:13 +00:00

Author

Owner

J45 firewall review finding (2026-11-01 IST): read-only inspection on ssh confirms the active owned nft table has an unconditional iifname "wt0" accept ahead of its source/port rules. The proposed Nix-managed firewalld policy binds eth0 and wt0 to separate default-drop zones and the requested TCP 7011 rich rule, but its parallel nft policy still permits UDP 51820 on eth0 from private-LAN sources. Firewalld excludes 10.69.0.0/24 for that exception; I am aligning nftables with the same exclusion so the private LAN can reach only the requested TCP 7011 rule. No production firewall, service, Traefik, DNS, or monitor01 changes have been made. The fast gate was queued at position 35 and will be restarted after this fix.

J45 firewall review finding (2026-11-01 IST): read-only inspection on ssh confirms the active owned nft table has an unconditional `iifname "wt0" accept` ahead of its source/port rules. The proposed Nix-managed firewalld policy binds `eth0` and `wt0` to separate default-drop zones and the requested TCP 7011 rich rule, but its parallel nft policy still permits UDP 51820 on eth0 from private-LAN sources. Firewalld excludes `10.69.0.0/24` for that exception; I am aligning nftables with the same exclusion so the private LAN can reach only the requested TCP 7011 rule. No production firewall, service, Traefik, DNS, or monitor01 changes have been made. The fast gate was queued at position 35 and will be restarted after this fix.

kayg commented

2026-11-01 12:14:00 +00:00

Author

Owner

Correction to the preceding firewall review note: I misread the generated replacement. render_nix_firewall() already excludes 10.69.0.0/24 from its UDP 51820 exception. That broad UDP rule and the unconditional wt0 accept are present only in the current active table; the installer replaces that owned table with the already-scoped policy, and firewalld independently excludes the private subnet. No code edit is needed. The fast gate was stopped while still queued and will be restarted on the unchanged branch.

Correction to the preceding firewall review note: I misread the generated replacement. `render_nix_firewall()` already excludes `10.69.0.0/24` from its UDP 51820 exception. That broad UDP rule and the unconditional `wt0 accept` are present only in the current active table; the installer replaces that owned table with the already-scoped policy, and firewalld independently excludes the private subnet. No code edit is needed. The fast gate was stopped while still queued and will be restarted on the unchanged branch.

kayg commented

2026-11-01 12:14:22 +00:00

Author

Owner

J45 code checkpoint (2026-11-01 IST): branch prod-on-ssh is pushed at abb3c24a0, based on newest fetched origin/dev 5418f9357. The updater now checks /healthz and / on every listener from the candidate unit; preflight and rollback check the listener set in the previous unit. Focused listener test passed; deployment suite reports Ran 48 tests / OK (expected fixture messages only).

The fast gate was restarted on this exact head via compute01 build-slot (PID 3403435, log ~/j45-fast.log). It is still waiting for a slot; no gate result is claimed. ssh remains active on the NetBird listener with SELinux Enforcing; no private bind/firewall, Traefik route, DNS, or monitor01 change has been made. Next: finish the fast gate, then update ssh and prove fridge-ingress reaches the private listener before adding the public route.

J45 code checkpoint (2026-11-01 IST): branch `prod-on-ssh` is pushed at `abb3c24a0`, based on newest fetched `origin/dev` `5418f9357`. The updater now checks `/healthz` and `/` on every listener from the candidate unit; preflight and rollback check the listener set in the previous unit. Focused listener test passed; deployment suite reports `Ran 48 tests` / `OK` (expected fixture messages only). The fast gate was restarted on this exact head via compute01 build-slot (PID 3403435, log `~/j45-fast.log`). It is still waiting for a slot; no gate result is claimed. `ssh` remains active on the NetBird listener with SELinux Enforcing; no private bind/firewall, Traefik route, DNS, or monitor01 change has been made. Next: finish the fast gate, then update `ssh` and prove fridge-ingress reaches the private listener before adding the public route.

kayg commented

2026-11-01 12:23:30 +00:00

Author

Owner

J45 WebSocket verification finding (2026-11-01 IST): the app's /v1/ws endpoint requires the authenticated __Host-infraternal owner session cookie and exact Origin matching https://ssh.kayg.org (crates/infraternal-server/src/websocket.rs). I will verify the public route forwards a WebSocket handshake to the application without bypassing auth; an authenticated 101 requires kayg to complete first-run passkey enrollment. The old public route, DNS record, and monitor01 service will remain until that check passes. The fast gate is queued on compute01; no production route or firewall change has occurred.

J45 WebSocket verification finding (2026-11-01 IST): the app's `/v1/ws` endpoint requires the authenticated `__Host-infraternal` owner session cookie and exact `Origin` matching `https://ssh.kayg.org` (`crates/infraternal-server/src/websocket.rs`). I will verify the public route forwards a WebSocket handshake to the application without bypassing auth; an authenticated 101 requires kayg to complete first-run passkey enrollment. The old public route, DNS record, and monitor01 service will remain until that check passes. The fast gate is queued on compute01; no production route or firewall change has occurred.

kayg commented

2026-11-01 13:42:26 +00:00

Author

Owner

J45 live-state check (2026-11-01 IST, resumed after provider stream closure):

On ssh (10.69.69.89), infraternal-server.service is active and SELinux is Enforcing, but ss shows only 10.69.69.89:7011. A local request to http://10.69.0.89:7011/healthz fails to connect.
firewalld.service is inactive/not installed. The active owned nft table still has an unconditional iifname "wt0" accept and its old rules; it has not been replaced.
From fridge-ingress, 10.69.0.89:7011/healthz times out. From the control host, strict curl to https://ssh.kayg.org fails certificate verification with a self-signed certificate. No new Traefik route or production firewall was installed.
I found three of my own scripts/verify fast waiters still queued, all before acquiring a slot. I stopped only those three; I will fetch the now-newer origin/dev (including 37bbda455), merge it into prod-on-ssh, and start one gate for the updated head.

No monitor01, Traefik, DNS, ssh service, or firewall configuration was changed during these checks.

J45 live-state check (2026-11-01 IST, resumed after provider stream closure): - On `ssh` (`10.69.69.89`), `infraternal-server.service` is active and SELinux is Enforcing, but `ss` shows only `10.69.69.89:7011`. A local request to `http://10.69.0.89:7011/healthz` fails to connect. - `firewalld.service` is inactive/not installed. The active owned nft table still has an unconditional `iifname "wt0" accept` and its old rules; it has not been replaced. - From fridge-ingress, `10.69.0.89:7011/healthz` times out. From the control host, strict curl to `https://ssh.kayg.org` fails certificate verification with a self-signed certificate. No new Traefik route or production firewall was installed. - I found three of my own `scripts/verify fast` waiters still queued, all before acquiring a slot. I stopped only those three; I will fetch the now-newer `origin/dev` (including `37bbda455`), merge it into `prod-on-ssh`, and start one gate for the updated head. No monitor01, Traefik, DNS, ssh service, or firewall configuration was changed during these checks.

kayg commented

2026-11-01 13:44:01 +00:00

Author

Owner

J45 current-dev checkpoint (2026-11-01 IST): fetched origin/dev at 37bbda45554f6e603a6856fc23319e17d8a1b14f and confirmed the requested commit is its head. prod-on-ssh now includes that base at merge head 86ffe3c93; the current focused deployment suite reports Ran 48 tests / OK. One scripts/verify fast is running through compute01 build-slot (PID 237802, log ~/j45-fast-86ffe3c93.log, queue position 29 at the first sample). The three stale duplicate waiters were stopped before rebase; no gate result is claimed yet.

Live production is unchanged: ssh is still mesh-only on TCP 7011, firewalld is inactive, and the owned nft table still contains the prior unconditional wt0 accept. Fridge-ingress cannot reach 10.69.0.89:7011; strict public curl currently fails on the self-signed Traefik certificate. Next after the gate: build the freshly fetched dev Nix package and firewalld output on compute01, update ssh, prove the private backend, then configure and verify the public route. No monitor01 or DNS mutation has occurred.

J45 current-dev checkpoint (2026-11-01 IST): fetched `origin/dev` at `37bbda45554f6e603a6856fc23319e17d8a1b14f` and confirmed the requested commit is its head. `prod-on-ssh` now includes that base at merge head `86ffe3c93`; the current focused deployment suite reports `Ran 48 tests` / `OK`. One `scripts/verify fast` is running through compute01 build-slot (PID 237802, log `~/j45-fast-86ffe3c93.log`, queue position 29 at the first sample). The three stale duplicate waiters were stopped before rebase; no gate result is claimed yet. Live production is unchanged: ssh is still mesh-only on TCP 7011, firewalld is inactive, and the owned nft table still contains the prior unconditional wt0 accept. Fridge-ingress cannot reach `10.69.0.89:7011`; strict public curl currently fails on the self-signed Traefik certificate. Next after the gate: build the freshly fetched dev Nix package and firewalld output on compute01, update ssh, prove the private backend, then configure and verify the public route. No monitor01 or DNS mutation has occurred.

kayg commented

2026-11-01 13:52:45 +00:00

Author

Owner

J45 resume (2026-11-01 IST). The inherited verify fast on 86ffe3c93 completed but failed the deployment review check. I reproduced the cause: the updated deploy CLI rejects --web-only outside staging, while the fixture omitted --staging and --staging-pid. I updated the fixture to exercise the supported staging path; the focused deployment review suite on compute01 reports 3 pass, 0 fail. Change committed and pushed on prod-on-ssh as ca2bc0f7e.

I am resubmitting one detached scripts/verify fast through compute01 build-slot; its log will be ~/j45-fast-ca2bc0f7e.log. Current live checks remain: ssh service active but bound only to 10.69.69.89:7011; firewall-cmd is absent; SELinux is Enforcing. From the control host, strict curl to https://ssh.kayg.org rejects the current self-signed certificate, and curl -k receives Traefik 404. No ssh listener/firewall, Traefik, DNS, or monitor01 changes have been applied.

J45 resume (2026-11-01 IST). The inherited `verify fast` on `86ffe3c93` completed but failed the deployment review check. I reproduced the cause: the updated deploy CLI rejects `--web-only` outside staging, while the fixture omitted `--staging` and `--staging-pid`. I updated the fixture to exercise the supported staging path; the focused deployment review suite on compute01 reports `3 pass, 0 fail`. Change committed and pushed on `prod-on-ssh` as `ca2bc0f7e`. I am resubmitting one detached `scripts/verify fast` through compute01 `build-slot`; its log will be `~/j45-fast-ca2bc0f7e.log`. Current live checks remain: ssh service active but bound only to `10.69.69.89:7011`; `firewall-cmd` is absent; SELinux is Enforcing. From the control host, strict curl to `https://ssh.kayg.org` rejects the current self-signed certificate, and `curl -k` receives Traefik 404. No ssh listener/firewall, Traefik, DNS, or monitor01 changes have been applied.

kayg commented

2026-11-01 14:00:59 +00:00

Author

Owner

J45 current-dev listener compatibility finding (2026-11-01 IST). The newly fetched origin/dev head is 37bbda45554f6e603a6856fc23319e17d8a1b14f. Its server CLI has one SocketAddr --listen; --listen-additional appears only in the unmerged prod-on-ssh changes. Passing that extra flag to the exact current-dev package would fail startup.

To preserve the requested package provenance, I will use the current CLI's IPv4 wildcard listener only after both the Nix-managed default-drop nft filter and Nix firewalld are active. The service unit will require and order after both filters. The only private-LAN TCP 7011 allow will remain the rich rule from source 10.69.0.99/32 to destination 10.69.0.89/32; the mesh interface retains its separate scoped policy. Candidate health checks will probe 10.69.69.89:7011 and 10.69.0.89:7011 individually. No listener/firewall change has been applied yet; the public route and all monitor01 state remain untouched.

J45 current-dev listener compatibility finding (2026-11-01 IST). The newly fetched `origin/dev` head is `37bbda45554f6e603a6856fc23319e17d8a1b14f`. Its server CLI has one `SocketAddr` `--listen`; `--listen-additional` appears only in the unmerged `prod-on-ssh` changes. Passing that extra flag to the exact current-dev package would fail startup. To preserve the requested package provenance, I will use the current CLI's IPv4 wildcard listener only after both the Nix-managed default-drop nft filter and Nix firewalld are active. The service unit will require and order after both filters. The only private-LAN TCP 7011 allow will remain the rich rule from source `10.69.0.99/32` to destination `10.69.0.89/32`; the mesh interface retains its separate scoped policy. Candidate health checks will probe `10.69.69.89:7011` and `10.69.0.89:7011` individually. No listener/firewall change has been applied yet; the public route and all monitor01 state remain untouched.

kayg commented

2026-11-01 14:11:51 +00:00

Author

Owner

J45 resume (2026-11-01 IST): the live compute01 queue shows PID 306960 running scripts/verify fast on prod-on-ssh at ca2bc0f7e (log ~/j45-fast-ca2bc0f7e.log). The earlier PID 237812 is gone, but the replacement run is active, so I am not creating a duplicate. The exact origin/dev package build is queued as PID 381511 (log ~/j45-package-37bb.log, queue position 3 at inspection). A fresh local fetch and git ls-remote origin refs/heads/dev both report origin/dev = 37bbda45554f6e603a6856fc23319e17d8a1b14f; the previously cited f4b61cd6c was not visible and I will fetch again on compute01 immediately before building. No production host, route, DNS, or monitor01 state changed. For the private-LAN route, the only needed input rule is TCP 7011 from fridge-ingress 10.69.0.99/32 to ssh 10.69.0.89/32; no NetBird allow or polling is needed.

J45 resume (2026-11-01 IST): the live compute01 queue shows PID 306960 running `scripts/verify fast` on `prod-on-ssh` at `ca2bc0f7e` (log `~/j45-fast-ca2bc0f7e.log`). The earlier PID 237812 is gone, but the replacement run is active, so I am not creating a duplicate. The exact origin/dev package build is queued as PID 381511 (log `~/j45-package-37bb.log`, queue position 3 at inspection). A fresh local fetch and `git ls-remote origin refs/heads/dev` both report origin/dev = `37bbda45554f6e603a6856fc23319e17d8a1b14f`; the previously cited `f4b61cd6c` was not visible and I will fetch again on compute01 immediately before building. No production host, route, DNS, or monitor01 state changed. For the private-LAN route, the only needed input rule is TCP 7011 from fridge-ingress `10.69.0.99/32` to ssh `10.69.0.89/32`; no NetBird allow or polling is needed.

kayg commented

2026-11-01 14:16:09 +00:00

Author

Owner

J45 finding (2026-11-01 IST): a new compute01 regression test failed because the updater preflight checks only infraternal-server.service before taking the state snapshot; it did not require either host filter to be active. Exact focused output: Ran 1 test / FAILED (failures=1); expected active checks for infraternal-firewall.service, firewalld.service, then the application service. I have added both firewall checks and tied wildcard HTTP units to those units with Requires= and BindsTo=. No production changes have been made.

J45 finding (2026-11-01 IST): a new compute01 regression test failed because the updater preflight checks only `infraternal-server.service` before taking the state snapshot; it did not require either host filter to be active. Exact focused output: `Ran 1 test` / `FAILED (failures=1)`; expected active checks for `infraternal-firewall.service`, `firewalld.service`, then the application service. I have added both firewall checks and tied wildcard HTTP units to those units with `Requires=` and `BindsTo=`. No production changes have been made.

kayg commented

2026-11-01 14:19:36 +00:00

Author

Owner

J45 live-state check (2026-11-01 IST, read-only): on ssh, infraternal-server.service is active/running, SELinux is Enforcing, and TCP 7011 is bound only at 10.69.69.89:7011. firewalld is inactive and its CLI is absent; the host nft CLI is also absent. Strict control-host curl to https://ssh.kayg.org/healthz still fails certificate verification with a self-signed certificate. No route, DNS, service, or firewall changes were made. I will install and validate the Nix firewall policy through the state-preserving updater before switching the listener.

J45 live-state check (2026-11-01 IST, read-only): on ssh, infraternal-server.service is active/running, SELinux is Enforcing, and TCP 7011 is bound only at `10.69.69.89:7011`. firewalld is inactive and its CLI is absent; the host `nft` CLI is also absent. Strict control-host curl to `https://ssh.kayg.org/healthz` still fails certificate verification with a self-signed certificate. No route, DNS, service, or firewall changes were made. I will install and validate the Nix firewall policy through the state-preserving updater before switching the listener.

kayg commented

2026-11-01 14:22:47 +00:00

Author

Owner

J45 code update (2026-11-01 IST): the wildcard listener, explicit dual-address health checks, and firewall-active preflight are implemented. The fresh bootstrap now refuses to run without the Nix firewalld/nftables output; this keeps its wildcard listener behind both required filters. The deployment suite on compute01 is green: Ran 51 tests in 2.788s / OK. The earlier test fixture error was fixed by letting the mocked subprocess observe the SSH command. I am committing/pushing this slice, then will run scripts/verify fast on the new head. Compute01 still shows the older ca2 gate and origin/dev Nix build active; no production route or service change has happened.

J45 code update (2026-11-01 IST): the wildcard listener, explicit dual-address health checks, and firewall-active preflight are implemented. The fresh bootstrap now refuses to run without the Nix firewalld/nftables output; this keeps its wildcard listener behind both required filters. The deployment suite on compute01 is green: `Ran 51 tests in 2.788s` / `OK`. The earlier test fixture error was fixed by letting the mocked subprocess observe the SSH command. I am committing/pushing this slice, then will run `scripts/verify fast` on the new head. Compute01 still shows the older ca2 gate and origin/dev Nix build active; no production route or service change has happened.

kayg commented

2026-11-01 14:24:24 +00:00

Author

Owner

J45 implementation checkpoint (2026-11-01 IST): pushed prod-on-ssh commit b4dbc086ef818183795d1bfd6fa77c082459dc7b. The focused deployment suite passed: Ran 51 tests in 2.788s / OK. Compute01 freshly fetched origin/dev in the package worktree and both its origin/dev ref and detached build HEAD are 37bbda45554f6e603a6856fc23319e17d8a1b14f; f4b61cd6c is still not visible from the configured origin. The prior ca2bc0f7e fast gate and exact-dev Nix package build remain active; I will run the final gate on b4dbc after that worktree is free. Production remains unchanged.

J45 implementation checkpoint (2026-11-01 IST): pushed `prod-on-ssh` commit `b4dbc086ef818183795d1bfd6fa77c082459dc7b`. The focused deployment suite passed: `Ran 51 tests in 2.788s` / `OK`. Compute01 freshly fetched origin/dev in the package worktree and both its `origin/dev` ref and detached build HEAD are `37bbda45554f6e603a6856fc23319e17d8a1b14f`; `f4b61cd6c` is still not visible from the configured origin. The prior `ca2bc0f7e` fast gate and exact-dev Nix package build remain active; I will run the final gate on b4dbc after that worktree is free. Production remains unchanged.

kayg commented

2026-11-01 14:27:40 +00:00

Author

Owner

J45 edge inventory (2026-11-01 IST, read-only): fridge-ingress Traefik is active, watches /etc/traefik/dynamic, and has the existing cf Cloudflare DNS-01 resolver. No dynamic router currently matches ssh.kayg.org. Cloudflare already has ssh.kayg.org A 49.205.172.130, DNS-only; infraternal.kayg.org A 37.157.255.66 is also DNS-only. The new record therefore already matches the requested address/proxy mode. No DNS or Traefik configuration changed.

J45 edge inventory (2026-11-01 IST, read-only): fridge-ingress Traefik is active, watches `/etc/traefik/dynamic`, and has the existing `cf` Cloudflare DNS-01 resolver. No dynamic router currently matches `ssh.kayg.org`. Cloudflare already has `ssh.kayg.org A 49.205.172.130`, DNS-only; `infraternal.kayg.org A 37.157.255.66` is also DNS-only. The new record therefore already matches the requested address/proxy mode. No DNS or Traefik configuration changed.

kayg commented

2026-11-01 14:29:27 +00:00

Author

Owner

J45 live check (2026-11-01 IST): ssh already has Determinate Nix 2.35.3 (nix 3.24.1), and the running infraternal executable is a Nix store path. The service remains on its single NetBird HTTP bind; SELinux is Enforcing. No host changes yet.

kayg commented

2026-11-01 14:38:56 +00:00

Author

Owner

J45 review findings (2026-11-01 IST), source inspection on pushed head b4dbc086ef818183795d1bfd6fa77c082459dc7b:

update_nix_package runs the firewalld installer before the remote update helper takes its /var/lib/infraternal snapshot. The installer restarts an active infraternal-firewall.service, and the app unit binds to that unit, so reapplying the firewall output can interrupt the server outside the update/rollback path.
The updater's already-current check compares source/version and runtime argv but does not verify the unit's After=, Requires=, and BindsTo= firewall dependencies. It can skip adding required firewall bindings when the package and argv already match.
Fresh bootstrap may select or install /root/.nix-profile/bin/nft independently of the validated firewalld package, then use that different binary for the infraternal firewall unit.

I am adding regression tests before fixes. Live compute01 status currently shows an older verify fast active on this worktree (PID 306960, log ~/j45-fast-ca2bc0f7e.log) and three queued jobs; I am not creating a duplicate concurrent build. No production route or monitor01 state has changed. No secrets included.

J45 review findings (2026-11-01 IST), source inspection on pushed head `b4dbc086ef818183795d1bfd6fa77c082459dc7b`: 1. `update_nix_package` runs the firewalld installer before the remote update helper takes its `/var/lib/infraternal` snapshot. The installer restarts an active `infraternal-firewall.service`, and the app unit binds to that unit, so reapplying the firewall output can interrupt the server outside the update/rollback path. 2. The updater's `already-current` check compares source/version and runtime argv but does not verify the unit's `After=`, `Requires=`, and `BindsTo=` firewall dependencies. It can skip adding required firewall bindings when the package and argv already match. 3. Fresh bootstrap may select or install `/root/.nix-profile/bin/nft` independently of the validated firewalld package, then use that different binary for the infraternal firewall unit. I am adding regression tests before fixes. Live compute01 status currently shows an older `verify fast` active on this worktree (PID 306960, log `~/j45-fast-ca2bc0f7e.log`) and three queued jobs; I am not creating a duplicate concurrent build. No production route or monitor01 state has changed. No secrets included.

kayg commented

2026-11-01 14:45:36 +00:00

Author

Owner

J45 review regressions reproduced on compute01 (2026-11-01 IST), from an isolated copy of the pushed source so the active gate's worktree was untouched. The focused run reports Ran 3 tests in 0.003s / FAILED (failures=3), each for the intended gap: missing firewall dependency comparison for the no-op path; fresh bootstrap does not consume the supplied firewall output's nft; and the firewalld installer has no pre-write guard against restarting the filter while its bound server is active. I am implementing these fixes now. No production configuration changed; no secrets included.

J45 review regressions reproduced on compute01 (2026-11-01 IST), from an isolated copy of the pushed source so the active gate's worktree was untouched. The focused run reports `Ran 3 tests in 0.003s` / `FAILED (failures=3)`, each for the intended gap: missing firewall dependency comparison for the no-op path; fresh bootstrap does not consume the supplied firewall output's `nft`; and the firewalld installer has no pre-write guard against restarting the filter while its bound server is active. I am implementing these fixes now. No production configuration changed; no secrets included.

kayg commented

2026-11-01 14:57:25 +00:00

Author

Owner

J45 continuation started (2026-11-01 IST): branch prod-on-ssh is at b4dbc086ef818183795d1bfd6fa77c082459dc7b, based on the fetched origin/dev head 37bbda45554f6e603a6856fc23319e17d8a1b14f. Three uncommitted review fixes in the Nix deploy/update helper address firewall unit dependency checks, using the validated Nix firewalld closure for nft, and preventing a live bound-server firewall restart; focused regression tests still need a rerun before commit. Live ssh evidence: service active on 10.69.69.89:7011 only, /healthz 200, SELinux Enforcing, firewall-cmd absent. Compute01 currently has 2 busy slots, 3 queued jobs and 8 GiB free, below build-slot’s 25 GiB floor, so no new test/build is running. Public ssh.kayg.org route and monitor01 remain untouched. New owner gate: before adding the public route, POST setup/registration requests without a setup code directly to the deployed server and record the refusal status/body; only kayg may register. No setup code or other secrets were read or posted.

J45 continuation started (2026-11-01 IST): branch `prod-on-ssh` is at `b4dbc086ef818183795d1bfd6fa77c082459dc7b`, based on the fetched `origin/dev` head `37bbda45554f6e603a6856fc23319e17d8a1b14f`. Three uncommitted review fixes in the Nix deploy/update helper address firewall unit dependency checks, using the validated Nix firewalld closure for nft, and preventing a live bound-server firewall restart; focused regression tests still need a rerun before commit. Live `ssh` evidence: service active on `10.69.69.89:7011` only, `/healthz` 200, SELinux Enforcing, `firewall-cmd` absent. Compute01 currently has 2 busy slots, 3 queued jobs and 8 GiB free, below build-slot’s 25 GiB floor, so no new test/build is running. Public `ssh.kayg.org` route and monitor01 remain untouched. New owner gate: before adding the public route, POST setup/registration requests without a setup code directly to the deployed server and record the refusal status/body; only kayg may register. No setup code or other secrets were read or posted.

kayg commented

2026-11-01 14:59:43 +00:00

Author

Owner

Owner reprioritisation noted (2026-11-01 IST): current-dev fresh install and LAN/firewall/public route are the go-live path; updater review fixes are deferred to a separate reviewed branch and will not gate the install. I will keep the first-owner setup guard as a pre-route safety check and repeat it through the public route before sending kayg the URL and setup code. The active ssh instance appears to be an earlier fresh bootstrap; before replacing it I am checking that it contains zero users and no state that needs preserving. No route, firewall, or monitor01 change has occurred.

kayg commented

2026-11-01 15:04:54 +00:00

Author

Owner

J45 live-state finding (2026-11-01 IST): the deployed package on ssh is from f4b61cd6c3f52078cad7573c5367d7a6e75f84f0, not current dev. The SQLite user table contains zero users and zero owners; /var/lib/infraternal and its setup socket are root-only (directory 0700, socket 0600), so there is no owner data to migrate. The active firewall is infraternal-firewall.service (nftables), not firewalld; its on-disk policy differs from the loaded table. The active nft dump currently has an unconditional iifname "wt0" accept and an old mesh 7011 rule. I will stop the empty service before replacing the firewall, then verify the loaded nft policy has default drop and only 10.69.0.99/32 → 10.69.0.89/32 TCP 7011 on the LAN path. No public route or monitor01 changes have occurred.

J45 live-state finding (2026-11-01 IST): the deployed package on ssh is from `f4b61cd6c3f52078cad7573c5367d7a6e75f84f0`, not current dev. The SQLite user table contains zero users and zero owners; `/var/lib/infraternal` and its setup socket are root-only (directory 0700, socket 0600), so there is no owner data to migrate. The active firewall is `infraternal-firewall.service` (nftables), not firewalld; its on-disk policy differs from the loaded table. The active nft dump currently has an unconditional `iifname "wt0" accept` and an old mesh 7011 rule. I will stop the empty service before replacing the firewall, then verify the loaded nft policy has default drop and only `10.69.0.99/32 → 10.69.0.89/32 TCP 7011` on the LAN path. No public route or monitor01 changes have occurred.

kayg referenced this issue

2026-11-10 15:23:56 +00:00

J478: housekeeping 2026-11-10, 60 old origin branches archived #487

kayg referenced this issue

2026-11-10 15:26:24 +00:00

J478: housekeeping 2026-11-10, 60 old origin branches archived #487

kayg referenced this issue

2026-11-10 15:28:09 +00:00

J478: housekeeping 2026-11-10, 60 old origin branches archived #487

kayg referenced this issue

2026-11-10 15:28:09 +00:00

J478: housekeeping 2026-11-10, 60 old origin branches archived #487