kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 191 Pull requests 61 Projects Releases Packages Wiki Activity Actions 20

Relay deploys over NetBird and move staging to compute01 (#686) #688

Closed
kayg wants to merge 1 commit from issue/686-netbird-deploy into dev
pull from: issue/686-netbird-deploy
merge into: kayg:dev
Conversation 46 Commits 1 Files changed 10 +103 -57
kayg commented 2026-10-26 14:59:20 +00:00
Owner
Copy link

Deploys now run from the orchestration host: build-slot.sh builds on compute01, and image transfer plus target operations use configurable NetBird commands relayed from that host. Staging defaults move to compute01; production remains unchanged until an operator runs TARGET=cloud. Rollback uses the same transport. Snapshot, health-wait and automatic rollback behavior is preserved.

Validation: bash -n and shellcheck on compute01 passed. Staging dry-run and live TARGET=next relay deploy passed: origin/dev image a779a46f42 built on compute01, image relayed through orchestration, data snapshotted, then healthy startup. Preview public HTTP status is 200. Public next routing is separately blocked because its EndpointSlice is Argo-managed, contrary to the job plan; no GitOps policy/source was changed. No production deploy was run.

Related #686. Do not merge as part of this migration job.

CI runs 485 (unchanged dev smoke) and 486 (this PR) execute on migrated runners. Server passed; web stops at existing bundle budget failures (308 > 260 KB total JS, 22.7 > 22.1 KB initial CSS, 255 > 240 KB initial JS); Smoke run 485 finished: server success, Rust success, web failure at the bundle gate. Duplicate run 486 was cancelled after the same web failure to give the smoke run capacity. This PR changes deployment text/scripts only and does not change budgets.

Deploys now run from the orchestration host: build-slot.sh builds on compute01, and image transfer plus target operations use configurable NetBird commands relayed from that host. Staging defaults move to compute01; production remains unchanged until an operator runs TARGET=cloud. Rollback uses the same transport. Snapshot, health-wait and automatic rollback behavior is preserved. Validation: bash -n and shellcheck on compute01 passed. Staging dry-run and live TARGET=next relay deploy passed: origin/dev image a779a46f4299 built on compute01, image relayed through orchestration, data snapshotted, then healthy startup. Preview public HTTP status is 200. Public next routing is separately blocked because its EndpointSlice is Argo-managed, contrary to the job plan; no GitOps policy/source was changed. No production deploy was run. Related #686. Do not merge as part of this migration job. CI runs 485 (unchanged dev smoke) and 486 (this PR) execute on migrated runners. Server passed; web stops at existing bundle budget failures (308 > 260 KB total JS, 22.7 > 22.1 KB initial CSS, 255 > 240 KB initial JS); Smoke run 485 finished: server success, Rust success, web failure at the bundle gate. Duplicate run 486 was cancelled after the same web failure to give the smoke run capacity. This PR changes deployment text/scripts only and does not change budgets.
deploy: relay NetBird builds and target operations from orchestration host
Some checks failed
CI / web (pull_request) Failing after 52s
Details
CI / server (pull_request) Successful in 1m15s
Details
CI / rust (pull_request) Has been cancelled
Details
bc25169a3e
kayg commented 2026-10-28 16:49:54 +00:00
Author
Owner
Copy link

Independent review started for PR #688. Local review is detached at base 3d0c6e33d5; PR branch issue/686-netbird-deploy, head bc25169a3e. I will inspect every commit and changed deploy script, run local checks and throwaway-target probes, and record ranked findings. No push, merge, deploy or production contact.

Independent review started for PR #688. Local review is detached at base 3d0c6e33d588335c745a660163bf8bf5cdc8412d; PR branch issue/686-netbird-deploy, head bc25169a3e5c4076d3e4e1dda7c91fab1e3cf98f. I will inspect every commit and changed deploy script, run local checks and throwaway-target probes, and record ranked findings. No push, merge, deploy or production contact.
kayg commented 2026-10-28 16:51:38 +00:00
Author
Owner
Copy link

Independent review finding: P1 — A full disk or switch error leaves the existing service stopped

deploy/next/remote.sh:97-101 stops the service before snapshot and retagging. The snapshot copy (:59), LAST write (:60), and new-image tag (:99) are outside the health failure branch; set -e exits without restarting the old service. There is no EXIT/error recovery trap. This is inherited and retained by this PR.

Evidence: python3 /tmp/review-688-evidence/probe.py runs the unchanged script with local command mocks, isolated temporary userdata, and injected failures. disk-full: copy returns 1 with cp: simulated No space left on device; exit 1, old service remains stopped, no health request. tag-fails: new tag operation returns 1; exit 1, old service remains stopped. Add transactional recovery covering every operation after stop; avoid unnecessary userdata copies for code deploys.

Independent review finding: P1 — A full disk or switch error leaves the existing service stopped `deploy/next/remote.sh:97-101` stops the service before `snapshot` and retagging. The snapshot copy (`:59`), LAST write (`:60`), and new-image tag (`:99`) are outside the health failure branch; `set -e` exits without restarting the old service. There is no EXIT/error recovery trap. This is inherited and retained by this PR. Evidence: `python3 /tmp/review-688-evidence/probe.py` runs the unchanged script with local command mocks, isolated temporary userdata, and injected failures. `disk-full`: copy returns 1 with `cp: simulated No space left on device`; exit 1, old service remains stopped, no health request. `tag-fails`: new tag operation returns 1; exit 1, old service remains stopped. Add transactional recovery covering every operation after stop; avoid unnecessary userdata copies for code deploys.
kayg commented 2026-10-28 16:51:38 +00:00
Author
Owner
Copy link

Independent review finding: P1 — Automatic rollback deletes and replaces live userdata

deploy/next/remote.sh:112 always restores the snapshot on failed health. restore_last_snapshot (:67-70) explicitly runs rm -rf "$DATA" and replaces the data directory. Snapshot retention also deletes with rm -rf (:63). This violates this review's requirement that deploys never delete or manipulate Home/state beyond app operations; daily backups already exist. A failed health response can overwrite writes made after new startup and cannot safely reverse external/multi-root changes. This is inherited, not introduced by the relay.

Evidence: the health-fails local probe executes automatic rollback and records rm -rf <temporary-root>/data.restoring and rm -rf <temporary-root>/data. Keep production code rollback separate from data restoration; require independently planned reversible migrations rather than implicit data replacement.

Independent review finding: P1 — Automatic rollback deletes and replaces live userdata `deploy/next/remote.sh:112` always restores the snapshot on failed health. `restore_last_snapshot` (`:67-70`) explicitly runs `rm -rf "$DATA"` and replaces the data directory. Snapshot retention also deletes with `rm -rf` (`:63`). This violates this review's requirement that deploys never delete or manipulate Home/state beyond app operations; daily backups already exist. A failed health response can overwrite writes made after new startup and cannot safely reverse external/multi-root changes. This is inherited, not introduced by the relay. Evidence: the `health-fails` local probe executes automatic rollback and records `rm -rf <temporary-root>/data.restoring` and `rm -rf <temporary-root>/data`. Keep production code rollback separate from data restoration; require independently planned reversible migrations rather than implicit data replacement.
kayg commented 2026-10-28 16:51:39 +00:00
Author
Owner
Copy link

Independent review finding: P1 — The artifact is not bound to the exact dev revision

deploy/next/relay-deploy.sh:23-30 compares local HEAD with the build checkout, not an authoritative dev SHA. :21-31 skips even that comparison for an explicit tag, while :36,47 saves and loads a mutable tag with no image digest/revision inspection. deploy/Containerfile has no source revision label. A matching feature checkout, stale checkout, or unrelated image with a plausible SHA tag is accepted for cloud. The build and later save are also separate remote calls, so another build can overwrite the tag between them.

Evidence: DRY_RUN=1 TARGET=next NEXT_HOST=throwaway.invalid REMOTE_CMD=/bin/false deploy/next/deploy.sh plans a build of PR HEAD bc25169a3e5c4076d3e4e1dda7c91fab1e3cf98f, despite the API dev SHA being 3d0c6e33d588335c745a660163bf8bf5cdc8412d. The same command with review688 has no BUILD/check step at all. Resolve and record the full dev SHA, build immutable content from that revision, and verify image identity through transfer and running-container health.

Independent review finding: P1 — The artifact is not bound to the exact dev revision `deploy/next/relay-deploy.sh:23-30` compares local HEAD with the build checkout, not an authoritative dev SHA. `:21-31` skips even that comparison for an explicit tag, while `:36,47` saves and loads a mutable tag with no image digest/revision inspection. `deploy/Containerfile` has no source revision label. A matching feature checkout, stale checkout, or unrelated image with a plausible SHA tag is accepted for cloud. The build and later save are also separate remote calls, so another build can overwrite the tag between them. Evidence: `DRY_RUN=1 TARGET=next NEXT_HOST=throwaway.invalid REMOTE_CMD=/bin/false deploy/next/deploy.sh` plans a build of PR HEAD `bc25169a3e5c4076d3e4e1dda7c91fab1e3cf98f`, despite the API dev SHA being `3d0c6e33d588335c745a660163bf8bf5cdc8412d`. The same command with `review688` has no BUILD/check step at all. Resolve and record the full dev SHA, build immutable content from that revision, and verify image identity through transfer and running-container health.
kayg commented 2026-10-28 16:51:41 +00:00
Author
Owner
Copy link

Independent review finding: P1 — Health can report success after service start failed

deploy/next/remote.sh:83-85 does not check the return of sysd start. Because start_and_check is invoked in an if (:101), Bash suppresses errexit inside the function; a successful curl masks a failed systemctl start. healthy (:74-80) checks any HTTP success at a fixed address, not service state or deployed revision. There is no pre-deploy health check. This is inherited.

Evidence: start-fails local probe makes systemctl start return 1 and HTTP return 0. The script exits 0, prints healthy: new is live, and the simulated service remains stopped. This models a stale or unrelated responder. Check the start status explicitly and verify service/container identity as well as HTTP health, before and after switching.

Independent review finding: P1 — Health can report success after service start failed `deploy/next/remote.sh:83-85` does not check the return of `sysd start`. Because `start_and_check` is invoked in an `if` (`:101`), Bash suppresses errexit inside the function; a successful curl masks a failed systemctl start. `healthy` (`:74-80`) checks any HTTP success at a fixed address, not service state or deployed revision. There is no pre-deploy health check. This is inherited. Evidence: `start-fails` local probe makes systemctl start return 1 and HTTP return 0. The script exits 0, prints `healthy: new is live`, and the simulated service remains stopped. This models a stale or unrelated responder. Check the start status explicitly and verify service/container identity as well as HTTP health, before and after switching.
kayg commented 2026-10-28 16:51:41 +00:00
Author
Owner
Copy link

Independent review finding: P1 — Predictable root-written unit staging file is unsafe

deploy/next/relay-deploy.sh:37,48 uses root's tee /tmp/$NAME.container; remote.sh:42 installs that shared path as a root-owned operation without checking ownership, symlinks or contents. Any local user can precreate the predictable path as a symlink before transfer or replace a writable staged unit before installation. This gives a local user a root-file overwrite opportunity or changes the container unit. The predictable file existed before; the new relay retains it and adds no isolation or file-mode controls.

Evidence: the dry-run prints UNIT throwaway.invalid: tee /tmp/calternal-next.container. tee follows symlinks and does not create exclusive private storage. Use a private, root-owned mode-0700 transaction directory created with mktemp and an exclusive mode-0600 file, carried through the same locked transaction. No real target or shared /tmp unit was used in this review.

Independent review finding: P1 — Predictable root-written unit staging file is unsafe `deploy/next/relay-deploy.sh:37,48` uses root's `tee /tmp/$NAME.container`; `remote.sh:42` installs that shared path as a root-owned operation without checking ownership, symlinks or contents. Any local user can precreate the predictable path as a symlink before transfer or replace a writable staged unit before installation. This gives a local user a root-file overwrite opportunity or changes the container unit. The predictable file existed before; the new relay retains it and adds no isolation or file-mode controls. Evidence: the dry-run prints `UNIT throwaway.invalid: tee /tmp/calternal-next.container`. `tee` follows symlinks and does not create exclusive private storage. Use a private, root-owned mode-0700 transaction directory created with mktemp and an exclusive mode-0600 file, carried through the same locked transaction. No real target or shared /tmp unit was used in this review.
kayg commented 2026-10-28 16:51:42 +00:00
Author
Owner
Copy link

Independent review finding: P2 — Re-running deploy erases the usable rollback revision

deploy/next/remote.sh:94-95 unconditionally overwrites rollback with current, even if the requested artifact is already current. A successful retry converts rollback from the previous working build to the same build currently deployed. Snapshot names also have only second precision (:58), and cp nests data into an existing snapshot on a same-second retry. There is no deployment lock across relay calls (relay-deploy.sh:46-49), shared unit staging, current/rollback tags or LAST, so concurrent operations can interleave these mutations. This is inherited switch behavior exposed by the relay.

Evidence: rerun local probe runs deploy twice with current=old and requested=new. Both exit 0; final tags are current=new, rollback=new. Its deliberately fixed timestamp also reproduces the snapshot collision. Compare immutable current/requested identities and return without changing rollback on retry; serialize the complete upload/switch transaction and use unique snapshot names if snapshots remain elsewhere.

Independent review finding: P2 — Re-running deploy erases the usable rollback revision `deploy/next/remote.sh:94-95` unconditionally overwrites rollback with current, even if the requested artifact is already current. A successful retry converts rollback from the previous working build to the same build currently deployed. Snapshot names also have only second precision (`:58`), and `cp` nests data into an existing snapshot on a same-second retry. There is no deployment lock across relay calls (`relay-deploy.sh:46-49`), shared unit staging, current/rollback tags or LAST, so concurrent operations can interleave these mutations. This is inherited switch behavior exposed by the relay. Evidence: `rerun` local probe runs deploy twice with current=old and requested=new. Both exit 0; final tags are current=new, rollback=new. Its deliberately fixed timestamp also reproduces the snapshot collision. Compare immutable current/requested identities and return without changing rollback on retry; serialize the complete upload/switch transaction and use unique snapshot names if snapshots remain elsewhere.
kayg commented 2026-10-28 16:51:44 +00:00
Author
Owner
Copy link

Independent review finding: P2 — Failure logging does not guarantee secret redaction

deploy/next/remote.sh:107 exports the last 20 service journal lines and excludes only lines containing the word token. Secrets named passwords, keys, authorization headers or sensitive app content are not excluded. The units load a secrets env file (deploy/cloud/calternal.container:20-21, next equivalent), but the deploy path neither provisions nor validates its mode. This is inherited. No actual secret was read or printed in this review.

Evidence: the health-fails probe uses only a synthetic journal line, synthetic credential value: REVIEW688_FAKE_VALUE; the script forwards it verbatim. Keep raw service journals on the target; return bounded non-sensitive deployment diagnostics. Require restrictive modes for secret files in the setup contract.

Independent review finding: P2 — Failure logging does not guarantee secret redaction `deploy/next/remote.sh:107` exports the last 20 service journal lines and excludes only lines containing the word `token`. Secrets named passwords, keys, authorization headers or sensitive app content are not excluded. The units load a secrets env file (`deploy/cloud/calternal.container:20-21`, next equivalent), but the deploy path neither provisions nor validates its mode. This is inherited. No actual secret was read or printed in this review. Evidence: the `health-fails` probe uses only a synthetic journal line, `synthetic credential value: REVIEW688_FAKE_VALUE`; the script forwards it verbatim. Keep raw service journals on the target; return bounded non-sensitive deployment diagnostics. Require restrictive modes for secret files in the setup contract.
kayg commented 2026-10-28 16:53:23 +00:00
Author
Owner
Copy link

Independent review finished. Verdict: fix first.

PR #688 independent review

Verdict: fix first. Reviewed the description and the sole commit, bc25169a3e5c4076d3e4e1dda7c91fab1e3cf98f, including all ten changed files. Forgejo reported dev base 3d0c6e33d588335c745a660163bf8bf5cdc8412d. Findings below distinguish new relay problems from inherited switch-script problems: both matter before using this production path. No production connection, deployment, push, merge or commit was performed.

P1 — A full disk or switch error leaves the existing service stopped

deploy/next/remote.sh:97-101 stops the service before snapshot and retagging. The snapshot copy (:59), LAST write (:60), and new-image tag (:99) are outside the health failure branch; set -e exits without restarting the old service. There is no EXIT/error recovery trap. This is inherited and retained by this PR.

Evidence: python3 /tmp/review-688-evidence/probe.py runs the unchanged script with local command mocks, isolated temporary userdata, and injected failures. disk-full: copy returns 1 with cp: simulated No space left on device; exit 1, old service remains stopped, no health request. tag-fails: new tag operation returns 1; exit 1, old service remains stopped. Add transactional recovery covering every operation after stop; avoid unnecessary userdata copies for code deploys.

P1 — Automatic rollback deletes and replaces live userdata

deploy/next/remote.sh:112 always restores the snapshot on failed health. restore_last_snapshot (:67-70) explicitly runs rm -rf "$DATA" and replaces the data directory. Snapshot retention also deletes with rm -rf (:63). This violates this review's requirement that deploys never delete or manipulate Home/state beyond app operations; daily backups already exist. A failed health response can overwrite writes made after new startup and cannot safely reverse external/multi-root changes. This is inherited, not introduced by the relay.

Evidence: the health-fails local probe executes automatic rollback and records rm -rf <temporary-root>/data.restoring and rm -rf <temporary-root>/data. Keep production code rollback separate from data restoration; require independently planned reversible migrations rather than implicit data replacement.

P1 — The artifact is not bound to the exact dev revision

deploy/next/relay-deploy.sh:23-30 compares local HEAD with the build checkout, not an authoritative dev SHA. :21-31 skips even that comparison for an explicit tag, while :36,47 saves and loads a mutable tag with no image digest/revision inspection. deploy/Containerfile has no source revision label. A matching feature checkout, stale checkout, or unrelated image with a plausible SHA tag is accepted for cloud. The build and later save are also separate remote calls, so another build can overwrite the tag between them.

Evidence: DRY_RUN=1 TARGET=next NEXT_HOST=throwaway.invalid REMOTE_CMD=/bin/false deploy/next/deploy.sh plans a build of PR HEAD bc25169a3e5c4076d3e4e1dda7c91fab1e3cf98f, despite the API dev SHA being 3d0c6e33d588335c745a660163bf8bf5cdc8412d. The same command with review688 has no BUILD/check step at all. Resolve and record the full dev SHA, build immutable content from that revision, and verify image identity through transfer and running-container health.

P1 — Health can report success after service start failed

deploy/next/remote.sh:83-85 does not check the return of sysd start. Because start_and_check is invoked in an if (:101), Bash suppresses errexit inside the function; a successful curl masks a failed systemctl start. healthy (:74-80) checks any HTTP success at a fixed address, not service state or deployed revision. There is no pre-deploy health check. This is inherited.

Evidence: start-fails local probe makes systemctl start return 1 and HTTP return 0. The script exits 0, prints healthy: new is live, and the simulated service remains stopped. This models a stale or unrelated responder. Check the start status explicitly and verify service/container identity as well as HTTP health, before and after switching.

P1 — Predictable root-written unit staging file is unsafe

deploy/next/relay-deploy.sh:37,48 uses root's tee /tmp/$NAME.container; remote.sh:42 installs that shared path as a root-owned operation without checking ownership, symlinks or contents. Any local user can precreate the predictable path as a symlink before transfer or replace a writable staged unit before installation. This gives a local user a root-file overwrite opportunity or changes the container unit. The predictable file existed before; the new relay retains it and adds no isolation or file-mode controls.

Evidence: the dry-run prints UNIT throwaway.invalid: tee /tmp/calternal-next.container. tee follows symlinks and does not create exclusive private storage. Use a private, root-owned mode-0700 transaction directory created with mktemp and an exclusive mode-0600 file, carried through the same locked transaction. No real target or shared /tmp unit was used in this review.

P2 — Re-running deploy erases the usable rollback revision

deploy/next/remote.sh:94-95 unconditionally overwrites rollback with current, even if the requested artifact is already current. A successful retry converts rollback from the previous working build to the same build currently deployed. Snapshot names also have only second precision (:58), and cp nests data into an existing snapshot on a same-second retry. There is no deployment lock across relay calls (relay-deploy.sh:46-49), shared unit staging, current/rollback tags or LAST, so concurrent operations can interleave these mutations. This is inherited switch behavior exposed by the relay.

Evidence: rerun local probe runs deploy twice with current=old and requested=new. Both exit 0; final tags are current=new, rollback=new. Its deliberately fixed timestamp also reproduces the snapshot collision. Compare immutable current/requested identities and return without changing rollback on retry; serialize the complete upload/switch transaction and use unique snapshot names if snapshots remain elsewhere.

P2 — Failure logging does not guarantee secret redaction

deploy/next/remote.sh:107 exports the last 20 service journal lines and excludes only lines containing the word token. Secrets named passwords, keys, authorization headers or sensitive app content are not excluded. The units load a secrets env file (deploy/cloud/calternal.container:20-21), but the deploy path neither provisions nor validates its mode. This is inherited. No actual secret was read or printed in this review.

Evidence: the health-fails probe uses only a synthetic journal line, synthetic credential value: REVIEW688_FAKE_VALUE; the script forwards it verbatim. Keep raw service journals on the target; return bounded non-sensitive deployment diagnostics. Require restrictive modes for secret files in the setup contract.

Coverage and limits

  • Artifact identity: unsafe as above. No immutable dev-to-running-image provenance.
  • Switch: Podman retagging changes the current pointer, but installing/reloading the unit, stopping, copying userdata, retagging and starting are separate operations without a transaction or recovery trap. No pre-switch health check. Clean-stop logic waits for the named container, but suppresses systemctl-stop errors and even suppresses failure to stop before data restoration (remote.sh:108).
  • Data: no new migration code in this PR; normal app startup is used (deploy/entrypoint.sh:3-8). The entrypoint requires explicit creation opt-in for new storage. This review does not certify arbitrary dev revisions' migrations. Userdata snapshot and deletion are unnecessary here and violate the requested deploy contract. Full-disk failure was injected, not produced by filling a real disk.
  • Secrets: no literal credentials or credential arguments were found in the diff. Target secret-file permissions are not established by these scripts and were not inspected remotely. Raw journal forwarding remains unsafe as above. The synthetic probe value is not a secret.
  • Network/access: both relay connections default to netbird ssh (transport.sh:5-6); cloud defaults to a NetBird hostname (deploy/cloud/target.env:5). REMOTE_CMD can bypass that policy. No firewall or sshd changes are included. Staging adds a loopback binding at 127.0.0.1:8080 alongside its changed LAN binding (calternal-next.container:12-13); cloud's LAN-only port is unchanged. Target SSH is unrestricted root by default, not a constrained deployment account; image runtime is rootless. No ACL/sudo provisioning is supplied, so least privilege is not demonstrated.
  • Operability: deploy and rollback dry-run work with REMOTE_CMD=/bin/false and a throwaway hostname. Manual code rollback is documented (README.md:57-62, cloud selection :88-94), but retry corrupts its previous-image pointer. Success logs identify a caller-selected tag, not full source SHA/image digest. No deploy-specific tests were found in the tree or added by the sole PR commit.
  • Validation commands: bash /tmp/review-688-evidence/gate.sh, python3 /tmp/review-688-evidence/probe.py, and python3 /tmp/review-688-evidence/relay-probe.py. The probe source, event traces and run JSON remain outside the repo under /tmp/review-688-evidence, /tmp/review688-probe-65j0p1bz, and the relay directories named in their output. Mocks cover command ordering and status handling, not real Podman, systemd, disk exhaustion, NetBird ACLs or public routing. No build or production access was needed.

Final gate

Reviewed head: bc25169a3e5c4076d3e4e1dda7c91fab1e3cf98f.

Verbatim gate output (passing reproduction probes mean the defects were reproduced, not that production deployment is safe):

bash -n: PASS
shellcheck: PASS
commit diff whitespace: PASS
deploy and rollback dry-run without transport: PASS
remote probes: PASS (six scenarios; failure modes reproduced)
relay success: PASS (exit=0, deploy_called=True)
relay save-fails: PASS (exit=1, deploy_called=False)
relay unit-fails: PASS (exit=1, deploy_called=False)
invalid tag rejected: PASS
predictable unit symlink overwrite reproduced: PASS (private fixture only)
Evidence directory: /tmp/review688-relay-j66ferky
scripts own tests: NONE FOUND
production safety verdict: FIX FIRST

The only workspace addition is this uncommitted review report. No product changes, commits, pushes, merges, deployments, issue closure or production connections.

Independent review finished. Verdict: **fix first**. # PR #688 independent review Verdict: **fix first**. Reviewed the description and the sole commit, `bc25169a3e5c4076d3e4e1dda7c91fab1e3cf98f`, including all ten changed files. Forgejo reported dev base `3d0c6e33d588335c745a660163bf8bf5cdc8412d`. Findings below distinguish new relay problems from inherited switch-script problems: both matter before using this production path. No production connection, deployment, push, merge or commit was performed. ## P1 — A full disk or switch error leaves the existing service stopped `deploy/next/remote.sh:97-101` stops the service before `snapshot` and retagging. The snapshot copy (`:59`), LAST write (`:60`), and new-image tag (`:99`) are outside the health failure branch; `set -e` exits without restarting the old service. There is no EXIT/error recovery trap. This is inherited and retained by this PR. Evidence: `python3 /tmp/review-688-evidence/probe.py` runs the unchanged script with local command mocks, isolated temporary userdata, and injected failures. `disk-full`: copy returns 1 with `cp: simulated No space left on device`; exit 1, old service remains stopped, no health request. `tag-fails`: new tag operation returns 1; exit 1, old service remains stopped. Add transactional recovery covering every operation after stop; avoid unnecessary userdata copies for code deploys. ## P1 — Automatic rollback deletes and replaces live userdata `deploy/next/remote.sh:112` always restores the snapshot on failed health. `restore_last_snapshot` (`:67-70`) explicitly runs `rm -rf "$DATA"` and replaces the data directory. Snapshot retention also deletes with `rm -rf` (`:63`). This violates this review's requirement that deploys never delete or manipulate Home/state beyond app operations; daily backups already exist. A failed health response can overwrite writes made after new startup and cannot safely reverse external/multi-root changes. This is inherited, not introduced by the relay. Evidence: the `health-fails` local probe executes automatic rollback and records `rm -rf <temporary-root>/data.restoring` and `rm -rf <temporary-root>/data`. Keep production code rollback separate from data restoration; require independently planned reversible migrations rather than implicit data replacement. ## P1 — The artifact is not bound to the exact dev revision `deploy/next/relay-deploy.sh:23-30` compares local HEAD with the build checkout, not an authoritative dev SHA. `:21-31` skips even that comparison for an explicit tag, while `:36,47` saves and loads a mutable tag with no image digest/revision inspection. `deploy/Containerfile` has no source revision label. A matching feature checkout, stale checkout, or unrelated image with a plausible SHA tag is accepted for cloud. The build and later save are also separate remote calls, so another build can overwrite the tag between them. Evidence: `DRY_RUN=1 TARGET=next NEXT_HOST=throwaway.invalid REMOTE_CMD=/bin/false deploy/next/deploy.sh` plans a build of PR HEAD `bc25169a3e5c4076d3e4e1dda7c91fab1e3cf98f`, despite the API dev SHA being `3d0c6e33d588335c745a660163bf8bf5cdc8412d`. The same command with `review688` has no BUILD/check step at all. Resolve and record the full dev SHA, build immutable content from that revision, and verify image identity through transfer and running-container health. ## P1 — Health can report success after service start failed `deploy/next/remote.sh:83-85` does not check the return of `sysd start`. Because `start_and_check` is invoked in an `if` (`:101`), Bash suppresses errexit inside the function; a successful curl masks a failed systemctl start. `healthy` (`:74-80`) checks any HTTP success at a fixed address, not service state or deployed revision. There is no pre-deploy health check. This is inherited. Evidence: `start-fails` local probe makes systemctl start return 1 and HTTP return 0. The script exits 0, prints `healthy: new is live`, and the simulated service remains stopped. This models a stale or unrelated responder. Check the start status explicitly and verify service/container identity as well as HTTP health, before and after switching. ## P1 — Predictable root-written unit staging file is unsafe `deploy/next/relay-deploy.sh:37,48` uses root's `tee /tmp/$NAME.container`; `remote.sh:42` installs that shared path as a root-owned operation without checking ownership, symlinks or contents. Any local user can precreate the predictable path as a symlink before transfer or replace a writable staged unit before installation. This gives a local user a root-file overwrite opportunity or changes the container unit. The predictable file existed before; the new relay retains it and adds no isolation or file-mode controls. Evidence: the dry-run prints `UNIT throwaway.invalid: tee /tmp/calternal-next.container`. `tee` follows symlinks and does not create exclusive private storage. Use a private, root-owned mode-0700 transaction directory created with mktemp and an exclusive mode-0600 file, carried through the same locked transaction. No real target or shared /tmp unit was used in this review. ## P2 — Re-running deploy erases the usable rollback revision `deploy/next/remote.sh:94-95` unconditionally overwrites rollback with current, even if the requested artifact is already current. A successful retry converts rollback from the previous working build to the same build currently deployed. Snapshot names also have only second precision (`:58`), and `cp` nests data into an existing snapshot on a same-second retry. There is no deployment lock across relay calls (`relay-deploy.sh:46-49`), shared unit staging, current/rollback tags or LAST, so concurrent operations can interleave these mutations. This is inherited switch behavior exposed by the relay. Evidence: `rerun` local probe runs deploy twice with current=old and requested=new. Both exit 0; final tags are current=new, rollback=new. Its deliberately fixed timestamp also reproduces the snapshot collision. Compare immutable current/requested identities and return without changing rollback on retry; serialize the complete upload/switch transaction and use unique snapshot names if snapshots remain elsewhere. ## P2 — Failure logging does not guarantee secret redaction `deploy/next/remote.sh:107` exports the last 20 service journal lines and excludes only lines containing the word `token`. Secrets named passwords, keys, authorization headers or sensitive app content are not excluded. The units load a secrets env file (`deploy/cloud/calternal.container:20-21`), but the deploy path neither provisions nor validates its mode. This is inherited. No actual secret was read or printed in this review. Evidence: the `health-fails` probe uses only a synthetic journal line, `synthetic credential value: REVIEW688_FAKE_VALUE`; the script forwards it verbatim. Keep raw service journals on the target; return bounded non-sensitive deployment diagnostics. Require restrictive modes for secret files in the setup contract. ## Coverage and limits - Artifact identity: unsafe as above. No immutable dev-to-running-image provenance. - Switch: Podman retagging changes the current pointer, but installing/reloading the unit, stopping, copying userdata, retagging and starting are separate operations without a transaction or recovery trap. No pre-switch health check. Clean-stop logic waits for the named container, but suppresses systemctl-stop errors and even suppresses failure to stop before data restoration (`remote.sh:108`). - Data: no new migration code in this PR; normal app startup is used (`deploy/entrypoint.sh:3-8`). The entrypoint requires explicit creation opt-in for new storage. This review does not certify arbitrary dev revisions' migrations. Userdata snapshot and deletion are unnecessary here and violate the requested deploy contract. Full-disk failure was injected, not produced by filling a real disk. - Secrets: no literal credentials or credential arguments were found in the diff. Target secret-file permissions are not established by these scripts and were not inspected remotely. Raw journal forwarding remains unsafe as above. The synthetic probe value is not a secret. - Network/access: both relay connections default to `netbird ssh` (`transport.sh:5-6`); cloud defaults to a NetBird hostname (`deploy/cloud/target.env:5`). REMOTE_CMD can bypass that policy. No firewall or sshd changes are included. Staging adds a loopback binding at `127.0.0.1:8080` alongside its changed LAN binding (`calternal-next.container:12-13`); cloud's LAN-only port is unchanged. Target SSH is unrestricted root by default, not a constrained deployment account; image runtime is rootless. No ACL/sudo provisioning is supplied, so least privilege is not demonstrated. - Operability: deploy and rollback dry-run work with REMOTE_CMD=/bin/false and a throwaway hostname. Manual code rollback is documented (`README.md:57-62`, cloud selection `:88-94`), but retry corrupts its previous-image pointer. Success logs identify a caller-selected tag, not full source SHA/image digest. No deploy-specific tests were found in the tree or added by the sole PR commit. - Validation commands: `bash /tmp/review-688-evidence/gate.sh`, `python3 /tmp/review-688-evidence/probe.py`, and `python3 /tmp/review-688-evidence/relay-probe.py`. The probe source, event traces and run JSON remain outside the repo under `/tmp/review-688-evidence`, `/tmp/review688-probe-65j0p1bz`, and the relay directories named in their output. Mocks cover command ordering and status handling, not real Podman, systemd, disk exhaustion, NetBird ACLs or public routing. No build or production access was needed. ## Final gate Reviewed head: `bc25169a3e5c4076d3e4e1dda7c91fab1e3cf98f`. Verbatim gate output (passing reproduction probes mean the defects were reproduced, not that production deployment is safe): ``` bash -n: PASS shellcheck: PASS commit diff whitespace: PASS deploy and rollback dry-run without transport: PASS remote probes: PASS (six scenarios; failure modes reproduced) relay success: PASS (exit=0, deploy_called=True) relay save-fails: PASS (exit=1, deploy_called=False) relay unit-fails: PASS (exit=1, deploy_called=False) invalid tag rejected: PASS predictable unit symlink overwrite reproduced: PASS (private fixture only) Evidence directory: /tmp/review688-relay-j66ferky scripts own tests: NONE FOUND production safety verdict: FIX FIRST ``` The only workspace addition is this uncommitted review report. No product changes, commits, pushes, merges, deployments, issue closure or production connections.
kayg commented 2026-10-28 16:55:25 +00:00
Author
Owner
Copy link

Starting deploy-fix-688 on branch fix/deploy-688. Base after merging current origin/dev: e4d8ebf309. Read all seven P1/P2 findings and comment 66964. I will replace userdata handling with code-only recovery, pin origin/dev artifacts, lock upload and switch together, and commit mock regression tests. No production contact, merge or deploy.

Starting deploy-fix-688 on branch fix/deploy-688. Base after merging current origin/dev: e4d8ebf309ef7b0904f13668c1a66a3feaa5c11c. Read all seven P1/P2 findings and comment 66964. I will replace userdata handling with code-only recovery, pin origin/dev artifacts, lock upload and switch together, and commit mock regression tests. No production contact, merge or deploy.
kayg commented 2026-10-28 17:02:35 +00:00
Author
Owner
Copy link

Finding 1, stopped service after failure: remote.sh now prepares uploads, checksum checks, trial tags and unit files before stop, and arms EXIT/HUP/INT/TERM recovery around the switch. Local committed-probe candidates passed disk-full, tag-fails, switch-tag-fails, reload-fails, unit-move-fails and signal scenarios: each returned failure with the old service running. No production connection.

Finding 1, stopped service after failure: remote.sh now prepares uploads, checksum checks, trial tags and unit files before stop, and arms EXIT/HUP/INT/TERM recovery around the switch. Local committed-probe candidates passed disk-full, tag-fails, switch-tag-fails, reload-fails, unit-move-fails and signal scenarios: each returned failure with the old service running. No production connection.
kayg commented 2026-10-28 17:02:36 +00:00
Author
Owner
Copy link

Finding 2, userdata replacement: removed all snapshots, LAST markers, retention deletion and restore code. Deploy and rollback only change image tags and the quadlet. The health-fails probe passed with the old image restarted, and command traces contain no Home/state or snapshot operations. The relay probe rejects rollback --data.

Finding 2, userdata replacement: removed all snapshots, LAST markers, retention deletion and restore code. Deploy and rollback only change image tags and the quadlet. The health-fails probe passed with the old image restarted, and command traces contain no Home/state or snapshot operations. The relay probe rejects rollback --data.
kayg commented 2026-10-28 17:02:37 +00:00
Author
Owner
Copy link

Finding 3, dev provenance: relay resolves full fetched origin/dev; builder archives that exact tree, labels the image and compiles the revision into /healthz, then saves by immutable image ID with a transfer checksum. Target validates the checksum, ID and OCI revision. Local label-mismatch, id-mismatch and checksum-mismatch probes passed; next/cloud reject mutable tags and non-dev SHAs before transport. Podman documents bare image IDs, so checks normalize the prefix: https://docs.podman.io/en/latest/markdown/podman-image-inspect.1.html .

Finding 3, dev provenance: relay resolves full fetched origin/dev; builder archives that exact tree, labels the image and compiles the revision into /healthz, then saves by immutable image ID with a transfer checksum. Target validates the checksum, ID and OCI revision. Local label-mismatch, id-mismatch and checksum-mismatch probes passed; next/cloud reject mutable tags and non-dev SHAs before transport. Podman documents bare image IDs, so checks normalize the prefix: https://docs.podman.io/en/latest/markdown/podman-image-inspect.1.html .
kayg commented 2026-10-28 17:02:37 +00:00
Author
Owner
Copy link

Finding 4, health: start_and_check explicitly checks systemctl start, active state, running container ID and the compiled health revision. The same checks run before deploy. start-fails, inactive-systemd, wrong-container-image, stale-response and pre-unhealthy probes passed. Safest first-use choice: an existing image without revision metadata fails closed and needs supervised bootstrap; documented in README. The fixes must land on dev first.

Finding 4, health: start_and_check explicitly checks systemctl start, active state, running container ID and the compiled health revision. The same checks run before deploy. start-fails, inactive-systemd, wrong-container-image, stale-response and pre-unhealthy probes passed. Safest first-use choice: an existing image without revision metadata fails closed and needs supervised bootstrap; documented in README. The fixes must land on dev first.
kayg commented 2026-10-28 17:02:38 +00:00
Author
Owner
Copy link

Finding 5, unit staging: one target flock now covers precheck, the entire upload and switch. Staging is a private mktemp directory on the unit filesystem, with 0700 directory and 0600 files; no shared /tmp unit path. The load mock checks actual staging modes. Unit installation and atomic rename failure probes passed; neither leaves the old service down.

Finding 5, unit staging: one target flock now covers precheck, the entire upload and switch. Staging is a private mktemp directory on the unit filesystem, with 0700 directory and 0600 files; no shared /tmp unit path. The load mock checks actual staging modes. Unit installation and atomic rename failure probes passed; neither leaves the old service down.
kayg commented 2026-10-28 17:02:39 +00:00
Author
Owner
Copy link

Finding 6, retries and concurrency: immutable current/requested IDs are compared before any service or rollback-pointer change. The rerun probe passed with rollback still pointing at the old image and no second stop/start/tag. All target operations use the same per-service flock. A further concurrent-upload probe is being added to verify serialization directly.

Finding 6, retries and concurrency: immutable current/requested IDs are compared before any service or rollback-pointer change. The rerun probe passed with rollback still pointing at the old image and no second stop/start/tag. All target operations use the same per-service flock. A further concurrent-upload probe is being added to verify serialization directly.
kayg commented 2026-10-28 17:02:40 +00:00
Author
Owner
Copy link

Finding 7, logs and secrets: removed journal forwarding entirely; target diagnostics use fixed text and validated SHA/image IDs. Secret files must be regular, not symlinks, owned by calternal and mode 0400 or 0600. open-secret, missing-secret and symlink-secret probes passed without a stop. Synthetic credential text is asserted absent from output across all probes. Required modes and local-only diagnostics are documented.

Finding 7, logs and secrets: removed journal forwarding entirely; target diagnostics use fixed text and validated SHA/image IDs. Secret files must be regular, not symlinks, owned by calternal and mode 0400 or 0600. open-secret, missing-secret and symlink-secret probes passed without a stop. Synthetic credential text is asserted absent from output across all probes. Required modes and local-only diagnostics are documented.
kayg commented 2026-10-28 17:07:03 +00:00
Author
Owner
Copy link

Finished deploy-fix-688. Replacement PR: #733 (supersedes #688; neither PR was merged or closed).

Branch: fix/deploy-688. Head: 63e1ce5a7d. Base includes current origin/dev at 3d0c6e33d5. Pushed only this new branch. No production contact, deployment or userdata operations.

All seven findings now have fixes and committed regression coverage. Additional probes verify real flock serialization, exact Git archive contexts despite dirty/untracked files, immutable-ID saving, and old-image startup when recovery retagging also fails. Evidence and logs were not committed.

Safest first-use choice: a current image without OCI revision metadata and a matching compiled health SHA is refused. Such a baseline needs supervised bootstrap after these fixes land on dev. Tests use local mocks; real Podman, systemd and NetBird first-use checks remain supervised work, not a deployment performed by this job.

Verbatim output from bash deploy/tests/gate.sh at the head above:

bash -n: PASS
shellcheck: PASS
diff whitespace: PASS
remote normal: PASS
remote disk-full: PASS
remote tag-fails: PASS
remote switch-tag-fails: PASS
remote start-fails: PASS
remote health-fails: PASS
remote stale-response: PASS
remote inactive-systemd: PASS
remote wrong-container-image: PASS
remote stop-fails: PASS
remote reload-fails: PASS
remote rollback-tag-fails: PASS
remote unit-install-fails: PASS
remote unit-move-fails: PASS
remote recovery-tag-fails: PASS
remote all-current-tags-fail: PASS
remote load-fails: PASS
remote checksum-mismatch: PASS
remote label-mismatch: PASS
remote id-mismatch: PASS
remote pre-unhealthy: PASS
remote open-secret: PASS
remote missing-secret: PASS
remote symlink-secret: PASS
remote signal: PASS
remote rerun: PASS
remote manual-rollback: PASS
remote concurrent upload/switch serialized: PASS
relay success: PASS
relay save-fails: PASS
relay target-fails: PASS
non-dev revision and mutable tags rejected for next/cloud: PASS
data rollback options rejected: PASS
build normal: PASS
build dev-moved: PASS
build build-fails: PASS
build bad-id: PASS
build wrong-label: PASS
build save-fails: PASS
deploy and code rollback dry-run without transport (next/cloud): PASS
deploy regression gates: PASS (local mocks; no production contact)

The targeted Rust HTTP health test also passed through build-slot with CALTERNAL_REVISION=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb:

running 1 test
test health_answers_without_signing_in ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 5 filtered out; finished in 0.01s

cargo fmt --all -- --check passed. The worktree is clean.

Finished deploy-fix-688. Replacement PR: https://git.kayg.org/kayg/calternal/pulls/733 (supersedes #688; neither PR was merged or closed). Branch: fix/deploy-688. Head: 63e1ce5a7d563eec8b9542b6e88405ae1d84cd62. Base includes current origin/dev at 3d0c6e33d588335c745a660163bf8bf5cdc8412d. Pushed only this new branch. No production contact, deployment or userdata operations. All seven findings now have fixes and committed regression coverage. Additional probes verify real flock serialization, exact Git archive contexts despite dirty/untracked files, immutable-ID saving, and old-image startup when recovery retagging also fails. Evidence and logs were not committed. Safest first-use choice: a current image without OCI revision metadata and a matching compiled health SHA is refused. Such a baseline needs supervised bootstrap after these fixes land on dev. Tests use local mocks; real Podman, systemd and NetBird first-use checks remain supervised work, not a deployment performed by this job. Verbatim output from bash deploy/tests/gate.sh at the head above: ```text bash -n: PASS shellcheck: PASS diff whitespace: PASS remote normal: PASS remote disk-full: PASS remote tag-fails: PASS remote switch-tag-fails: PASS remote start-fails: PASS remote health-fails: PASS remote stale-response: PASS remote inactive-systemd: PASS remote wrong-container-image: PASS remote stop-fails: PASS remote reload-fails: PASS remote rollback-tag-fails: PASS remote unit-install-fails: PASS remote unit-move-fails: PASS remote recovery-tag-fails: PASS remote all-current-tags-fail: PASS remote load-fails: PASS remote checksum-mismatch: PASS remote label-mismatch: PASS remote id-mismatch: PASS remote pre-unhealthy: PASS remote open-secret: PASS remote missing-secret: PASS remote symlink-secret: PASS remote signal: PASS remote rerun: PASS remote manual-rollback: PASS remote concurrent upload/switch serialized: PASS relay success: PASS relay save-fails: PASS relay target-fails: PASS non-dev revision and mutable tags rejected for next/cloud: PASS data rollback options rejected: PASS build normal: PASS build dev-moved: PASS build build-fails: PASS build bad-id: PASS build wrong-label: PASS build save-fails: PASS deploy and code rollback dry-run without transport (next/cloud): PASS deploy regression gates: PASS (local mocks; no production contact) ``` The targeted Rust HTTP health test also passed through build-slot with CALTERNAL_REVISION=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb: ```text running 1 test test health_answers_without_signing_in ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 5 filtered out; finished in 0.01s ``` cargo fmt --all -- --check passed. The worktree is clean.
kayg commented 2026-10-28 17:25:06 +00:00
Author
Owner
Copy link

Starting second deploy fix round in /home/agent/wt/deploy-fix, branch fix/deploy-688, base 63e1ce5a7d. I will reproduce the recovery restart and EnvironmentFile mode-check findings with mocks, add regression tests, document supervised bootstrap, and run bash -n, shellcheck, deploy tests and dry-run. No production contact, merge or deploy.

Starting second deploy fix round in /home/agent/wt/deploy-fix, branch fix/deploy-688, base 63e1ce5a7d563eec8b9542b6e88405ae1d84cd62. I will reproduce the recovery restart and EnvironmentFile mode-check findings with mocks, add regression tests, document supervised bootstrap, and run bash -n, shellcheck, deploy tests and dry-run. No production contact, merge or deploy.
kayg commented 2026-10-28 17:26:27 +00:00
Author
Owner
Copy link

Findings reproduced against 63e1ce5a with committed-test candidates and isolated local mocks. P1: persistent daemon-reload failure, persistent unit rename failure, stop returning an error after stopping, and post-stop Podman error all exit with the old service stopped and zero restart calls. P2: spaced EnvironmentFile directives in either the installed or replacement unit allow a synthetic mode-0644 file through and deploy succeeds. The revised systemd mock keeps the loaded generated definition separate from the quadlet. Evidence is /tmp/deploy-733-red.log plus targeted repro output on 8by64-compute01; no production contact. Fix in progress: unconditional best-effort recovery start with independent preparation steps and retained recovery units on failure; whitespace-aware, fail-closed secret path validation.

Findings reproduced against 63e1ce5a with committed-test candidates and isolated local mocks. P1: persistent daemon-reload failure, persistent unit rename failure, stop returning an error after stopping, and post-stop Podman error all exit with the old service stopped and zero restart calls. P2: spaced EnvironmentFile directives in either the installed or replacement unit allow a synthetic mode-0644 file through and deploy succeeds. The revised systemd mock keeps the loaded generated definition separate from the quadlet. Evidence is /tmp/deploy-733-red.log plus targeted repro output on 8by64-compute01; no production contact. Fix in progress: unconditional best-effort recovery start with independent preparation steps and retained recovery units on failure; whitespace-aware, fail-closed secret path validation.
kayg commented 2026-10-28 17:31:01 +00:00
Author
Owner
Copy link

Second deploy fix round finished. Branch fix/deploy-688, head a6a50fe526, pushed successfully. PR #733 reports that exact head. Base was 63e1ce5a7d.

Finding Fix Test and evidence
P1: a persistent recovery stop, unit rename or reload failure skipped the restart and left the old service stopped. Recovery preparation steps are independent and always reach a restart attempt. Unit copies preserve ownership and mode. A failed restart or health check returns failure, reports it, and retains both prepared units in the private staging directory. Before the fix, the four reviewer recovery probes reproduced a stopped old service with no restart. The committed tests now require a restart and the healthy old immutable image for persistent-daemon-reload, persistent-unit-rename, stop-returns-error-after-stopping and post-stop-podman-error. recovery-start-fails requires two start attempts, a clear failure message, private retained recovery files and an unchanged rollback pointer. The mock now tracks the generated systemd definition independently of the quadlet.
P2: valid spaced EnvironmentFile directives bypassed the mode check. Parse logical lines and whitespace around the directive. Check every listed path with metadata only, including repeated directives, whole-path quotes, continuations and comments, optional-file prefixes, and %h / %% expansion. Check installed and replacement units. Unsupported specifiers, C-style escapes, wildcards, relative paths and malformed quoting fail closed. Optional files must still exist with safe metadata. Before the fix, spaced installed and replacement directives both accepted a synthetic mode-0644 file. Committed spaced-open-secret and spaced-new-open-secret now refuse before any stop. Multiple, repeated, optional, Service-section, quoted and continued open-secret vectors also refuse. Corresponding safe vectors, including %h and literal percent paths, pass. Unsupported escapes and specifiers refuse.
Bootstrap: an unlabelled image needs a supervised first installation. Preserve the refusal and add an eight-step supervised bootstrap procedure covering the exact dev artifact, target lock, saved old image/unit, secret metadata, existing storage, verified switch and recovery. unlabelled-baseline requires failure with the bootstrap message, no stop or start, and the old image still running.

Safest syntax choice: unsupported path forms are explicitly refused rather than guessed. The README documents the accepted subset and conservative handling of optional files and reset directives. Parsing references were read from the primary Podman Quadlet and systemd syntax documentation through Firecrawl.

Gates: timeout -k 10 360 bash deploy/tests/gate.sh exited 0. The additional bash -n deploy/*.sh deploy/next/*.sh deploy/tests/gate.sh and git diff --check also exited 0. The working tree is clean. All tests use local mocks; next/cloud dry-runs use an invalid destination and /bin/false transport. No production contact, merge or deployment. No evidence or artifacts were committed.

Gate output verbatim:

bash -n: PASS
shellcheck: PASS
diff whitespace: PASS
remote normal: PASS
remote disk-full: PASS
remote tag-fails: PASS
remote switch-tag-fails: PASS
remote start-fails: PASS
remote health-fails: PASS
remote stale-response: PASS
remote inactive-systemd: PASS
remote wrong-container-image: PASS
remote stop-fails: PASS
remote reload-fails: PASS
remote rollback-tag-fails: PASS
remote unit-install-fails: PASS
remote unit-move-fails: PASS
remote recovery-tag-fails: PASS
remote all-current-tags-fail: PASS
remote load-fails: PASS
remote checksum-mismatch: PASS
remote label-mismatch: PASS
remote id-mismatch: PASS
remote pre-unhealthy: PASS
remote open-secret: PASS
remote missing-secret: PASS
remote symlink-secret: PASS
remote unlabelled-baseline: PASS
remote persistent-daemon-reload: PASS
remote persistent-unit-rename: PASS
remote stop-returns-error-after-stopping: PASS
remote post-stop-podman-error: PASS
remote recovery-start-fails: PASS
remote spaced-open-secret: PASS
remote spaced-new-open-secret: PASS
remote multiple-open-secret: PASS
remote optional-open-secret: PASS
remote service-open-secret: PASS
remote continued-open-secret: PASS
remote quoted-open-secret: PASS
remote unsupported-specifier: PASS
remote unsupported-escape: PASS
remote repeated-open-secret: PASS
remote spaced-safe-secret: PASS
remote multiple-safe-secret: PASS
remote optional-safe-secret: PASS
remote service-safe-secret: PASS
remote continued-safe-secret: PASS
remote quoted-safe-secret: PASS
remote home-safe-secret: PASS
remote percent-safe-secret: PASS
remote signal: PASS
remote rerun: PASS
remote manual-rollback: PASS
remote concurrent upload/switch serialized: PASS
relay success: PASS
relay save-fails: PASS
relay target-fails: PASS
non-dev revision and mutable tags rejected for next/cloud: PASS
data rollback options rejected: PASS
build normal: PASS
build dev-moved: PASS
build build-fails: PASS
build bad-id: PASS
build wrong-label: PASS
build save-fails: PASS
deploy and code rollback dry-run without transport (next/cloud): PASS
deploy regression gates: PASS (local mocks; no production contact)
Second deploy fix round finished. Branch fix/deploy-688, head a6a50fe52667b9e631085369d9cdf52270a92d84, pushed successfully. PR #733 reports that exact head. Base was 63e1ce5a7d563eec8b9542b6e88405ae1d84cd62. | Finding | Fix | Test and evidence | | --- | --- | --- | | P1: a persistent recovery stop, unit rename or reload failure skipped the restart and left the old service stopped. | Recovery preparation steps are independent and always reach a restart attempt. Unit copies preserve ownership and mode. A failed restart or health check returns failure, reports it, and retains both prepared units in the private staging directory. | Before the fix, the four reviewer recovery probes reproduced a stopped old service with no restart. The committed tests now require a restart and the healthy old immutable image for persistent-daemon-reload, persistent-unit-rename, stop-returns-error-after-stopping and post-stop-podman-error. recovery-start-fails requires two start attempts, a clear failure message, private retained recovery files and an unchanged rollback pointer. The mock now tracks the generated systemd definition independently of the quadlet. | | P2: valid spaced EnvironmentFile directives bypassed the mode check. | Parse logical lines and whitespace around the directive. Check every listed path with metadata only, including repeated directives, whole-path quotes, continuations and comments, optional-file prefixes, and %h / %% expansion. Check installed and replacement units. Unsupported specifiers, C-style escapes, wildcards, relative paths and malformed quoting fail closed. Optional files must still exist with safe metadata. | Before the fix, spaced installed and replacement directives both accepted a synthetic mode-0644 file. Committed spaced-open-secret and spaced-new-open-secret now refuse before any stop. Multiple, repeated, optional, Service-section, quoted and continued open-secret vectors also refuse. Corresponding safe vectors, including %h and literal percent paths, pass. Unsupported escapes and specifiers refuse. | | Bootstrap: an unlabelled image needs a supervised first installation. | Preserve the refusal and add an eight-step supervised bootstrap procedure covering the exact dev artifact, target lock, saved old image/unit, secret metadata, existing storage, verified switch and recovery. | unlabelled-baseline requires failure with the bootstrap message, no stop or start, and the old image still running. | Safest syntax choice: unsupported path forms are explicitly refused rather than guessed. The README documents the accepted subset and conservative handling of optional files and reset directives. Parsing references were read from the primary Podman Quadlet and systemd syntax documentation through Firecrawl. Gates: `timeout -k 10 360 bash deploy/tests/gate.sh` exited 0. The additional `bash -n deploy/*.sh deploy/next/*.sh deploy/tests/gate.sh` and `git diff --check` also exited 0. The working tree is clean. All tests use local mocks; next/cloud dry-runs use an invalid destination and /bin/false transport. No production contact, merge or deployment. No evidence or artifacts were committed. Gate output verbatim: ```text bash -n: PASS shellcheck: PASS diff whitespace: PASS remote normal: PASS remote disk-full: PASS remote tag-fails: PASS remote switch-tag-fails: PASS remote start-fails: PASS remote health-fails: PASS remote stale-response: PASS remote inactive-systemd: PASS remote wrong-container-image: PASS remote stop-fails: PASS remote reload-fails: PASS remote rollback-tag-fails: PASS remote unit-install-fails: PASS remote unit-move-fails: PASS remote recovery-tag-fails: PASS remote all-current-tags-fail: PASS remote load-fails: PASS remote checksum-mismatch: PASS remote label-mismatch: PASS remote id-mismatch: PASS remote pre-unhealthy: PASS remote open-secret: PASS remote missing-secret: PASS remote symlink-secret: PASS remote unlabelled-baseline: PASS remote persistent-daemon-reload: PASS remote persistent-unit-rename: PASS remote stop-returns-error-after-stopping: PASS remote post-stop-podman-error: PASS remote recovery-start-fails: PASS remote spaced-open-secret: PASS remote spaced-new-open-secret: PASS remote multiple-open-secret: PASS remote optional-open-secret: PASS remote service-open-secret: PASS remote continued-open-secret: PASS remote quoted-open-secret: PASS remote unsupported-specifier: PASS remote unsupported-escape: PASS remote repeated-open-secret: PASS remote spaced-safe-secret: PASS remote multiple-safe-secret: PASS remote optional-safe-secret: PASS remote service-safe-secret: PASS remote continued-safe-secret: PASS remote quoted-safe-secret: PASS remote home-safe-secret: PASS remote percent-safe-secret: PASS remote signal: PASS remote rerun: PASS remote manual-rollback: PASS remote concurrent upload/switch serialized: PASS relay success: PASS relay save-fails: PASS relay target-fails: PASS non-dev revision and mutable tags rejected for next/cloud: PASS data rollback options rejected: PASS build normal: PASS build dev-moved: PASS build build-fails: PASS build bad-id: PASS build wrong-label: PASS build save-fails: PASS deploy and code rollback dry-run without transport (next/cloud): PASS deploy regression gates: PASS (local mocks; no production contact) ```
kayg commented 2026-10-28 17:44:12 +00:00
Author
Owner
Copy link

Starting deploy-fix-3 on branch fix/deploy-688, base a6a50fe526. Reading the third review and probe scripts, then replacing the switch with an explicit state machine and exhaustive failure injection. Local tests and dry-run only; no production access, merge or deploy. The job explicitly requires pushing fix/deploy-688 and commenting on PR #733; I will follow that specific branch instruction.

Starting deploy-fix-3 on branch fix/deploy-688, base a6a50fe52667b9e631085369d9cdf52270a92d84. Reading the third review and probe scripts, then replacing the switch with an explicit state machine and exhaustive failure injection. Local tests and dry-run only; no production access, merge or deploy. The job explicitly requires pushing fix/deploy-688 and commenting on PR #733; I will follow that specific branch instruction.
kayg commented 2026-10-28 17:44:35 +00:00
Author
Owner
Copy link

Confirmed third-review root causes at a6a50fe5: recovery copies restore.container after stopping and retagging, so ENOSPC can leave the loaded candidate definition; retry checks only digest, so a different digest with the same revision overwrites rollback. The reviewer host SSH timed out; retrying and reading the complete probe attached in comment 67040 as fallback. Fix will pin immutable image identities before stop and model all switch transitions explicitly.

Confirmed third-review root causes at a6a50fe5: recovery copies restore.container after stopping and retagging, so ENOSPC can leave the loaded candidate definition; retry checks only digest, so a different digest with the same revision overwrites rollback. The reviewer host SSH timed out; retrying and reading the complete probe attached in comment 67040 as fallback. Fix will pin immutable image identities before stop and model all switch transitions explicitly.
kayg commented 2026-10-28 17:49:32 +00:00
Author
Owner
Copy link

Reproduced both review findings with the independent source from comment 67040 and corrected safety assertions: disk-full-after-start exited 1 with candidate digest 2222 running; same-revision exited 0 after one stop and replaced the prior rollback. Added both as regression tests. The state machine now prepares digest-pinned main/recovery units and an independent loaded fallback before stopping; recovery consumes the prepared unit by rename and never starts an uncertain candidate definition. Same-SHA deploys preserve service and rollback even with another digest. Both new regression cases now pass. Before/after command-boundary failure injection is running, including recovery paths and userdata sentinels. Bootstrap choice: keep the existing refusal for unlabelled baselines and test that refusal, rather than automate the supervised maintenance procedure. Reviewer-host access is blocked: hostname SSH timed out then failed DNS; a bounded retry to its last resolved non-production IP also timed out. The complete attached probe source and findings in comments 67031/67032/67040/67041 were read instead. No production access.

Reproduced both review findings with the independent source from comment 67040 and corrected safety assertions: disk-full-after-start exited 1 with candidate digest 2222 running; same-revision exited 0 after one stop and replaced the prior rollback. Added both as regression tests. The state machine now prepares digest-pinned main/recovery units and an independent loaded fallback before stopping; recovery consumes the prepared unit by rename and never starts an uncertain candidate definition. Same-SHA deploys preserve service and rollback even with another digest. Both new regression cases now pass. Before/after command-boundary failure injection is running, including recovery paths and userdata sentinels. Bootstrap choice: keep the existing refusal for unlabelled baselines and test that refusal, rather than automate the supervised maintenance procedure. Reviewer-host access is blocked: hostname SSH timed out then failed DNS; a bounded retry to its last resolved non-production IP also timed out. The complete attached probe source and findings in comments 67031/67032/67040/67041 were read instead. No production access.
kayg commented 2026-10-28 17:54:42 +00:00
Author
Owner
Copy link

Additional state-machine checks: daemon-reload alone does not prove that a generator replaced the loaded service definition. Recovery now checks the loaded ExecStart image before starting the primary, and uses the preloaded old-digest fallback if it is stale. Added passing stale-recovery-definition (refuse before stop), stale-primary-recovery (old digest through fallback), and post-start-unit-rename (old digest through fallback) probes. The real installed Quadlet generator also compiles both pinned services, with separate CID files and a Conflicts relation. Retained recovery material is refused and preserved; the retained-recovery probe passes. A tag write that takes effect before returning an error is treated as the commit point only after candidate health was verified; exhaustive before/after command injection covers this boundary. The final gate is running from a fixed source version; earlier exploratory runs are not being reported as a final pass.

Additional state-machine checks: daemon-reload alone does not prove that a generator replaced the loaded service definition. Recovery now checks the loaded ExecStart image before starting the primary, and uses the preloaded old-digest fallback if it is stale. Added passing stale-recovery-definition (refuse before stop), stale-primary-recovery (old digest through fallback), and post-start-unit-rename (old digest through fallback) probes. The real installed Quadlet generator also compiles both pinned services, with separate CID files and a Conflicts relation. Retained recovery material is refused and preserved; the retained-recovery probe passes. A tag write that takes effect before returning an error is treated as the commit point only after candidate health was verified; exhaustive before/after command injection covers this boundary. The final gate is running from a fixed source version; earlier exploratory runs are not being reported as a final pass.
kayg commented 2026-10-28 18:07:54 +00:00
Author
Owner
Copy link

Gate finding: bash -n accepts one script followed by its positional arguments, so the previous wildcard invocation did not parse every deploy script. The gate now loops over each file; the explicit all-files syntax check passed, and the final suite has started with that corrected gate. The exhaustive test oracle independently records command return status and requires a successful candidate revision validation before accepting a committed image; userdata snapshots compare modes, ownership and timestamps as well as bytes. No switch implementation changes were made during this final test strengthening.

Gate finding: bash -n accepts one script followed by its positional arguments, so the previous wildcard invocation did not parse every deploy script. The gate now loops over each file; the explicit all-files syntax check passed, and the final suite has started with that corrected gate. The exhaustive test oracle independently records command return status and requires a successful candidate revision validation before accepting a committed image; userdata snapshots compare modes, ownership and timestamps as well as bytes. No switch implementation changes were made during this final test strengthening.
kayg commented 2026-10-28 18:26:23 +00:00
Author
Owner
Copy link

deploy-fix-3 finished. Branch: fix/deploy-688. Base: a6a50fe526. Pushed head: 8dc52337a5.

Review this commit. Forgejo reports PR #733 at this head.

Finding Fix Test
P1 full-disk recovery can restart the candidate Prepare the final old-digest unit before stop and rename it directly. Preload an independent old-digest recovery service. Check the loaded primary ExecStart before restarting it; otherwise use the fallback. No recovery copy or mutable current tag selects the image. disk-full-after-start; stale-primary-recovery; post-start-unit-rename; real Quadlet generation; before/after failure sweeps
P2 a rebuild of the current SHA erases rollback A healthy current SHA is a deploy no-op even when the artifact digest differs. Rollback still selects the exact digest and SHA pair. same-revision different digest; rerun; same-sha and same-image exhaustive paths
Switch errors depend on where mutable state was read One small switch.sh module owns the states, transitions, recovery and rollback invariant. The healthy candidate's rollback-tag write is the commit point; an error after the write is resolved by inspecting that pointer. 982 before/after external-command failures across 12 paths; an independent successful revision-check record is required before accepting a committed candidate
Recovery itself can fail Use the preloaded digest-pinned service, make two bounded start/health attempts, retain private repair files on persistent failure, and refuse to delete an existing recovery unit. recovery-start-fails; retained-recovery; stale-definition and rename-failure sweeps for deploy and rollback
Bootstrap must not bypass provenance or touch data Keep unlabelled and missing-unit baselines fail-closed. The supervised bootstrap remains separate; no storage provisioning was added. both bootstrap refusal sweeps; unlabelled-baseline; all property cases compare userdata bytes, modes, ownership and timestamps and reject data-operation arguments
The old bash -n wildcard checked only the first script Parse every deploy script individually. all-files bash syntax, ShellCheck, staged whitespace check

Validation: /usr/local/bin/build-slot.sh bash deploy/tests/gate.sh exited 0. The exhaustive fixtures use at most three independent workers inside that one slot. Final all-files syntax, ShellCheck and staged whitespace checks also passed. Gate output verbatim:

bash -n: PASS
shellcheck: PASS
diff whitespace: PASS
real Quadlet generation (main and immutable recovery): PASS
remote normal: PASS
remote stale-recovery-definition: PASS
remote stale-primary-recovery: PASS
remote post-start-unit-rename: PASS
remote retained-recovery: PASS
remote same-revision different digest: PASS
remote disk-full-after-start: PASS
remote disk-full: PASS
remote tag-fails: PASS
remote switch-tag-fails: PASS
remote start-fails: PASS
remote health-fails: PASS
remote stale-response: PASS
remote inactive-systemd: PASS
remote wrong-container-image: PASS
remote stop-fails: PASS
remote reload-fails: PASS
remote rollback-tag-fails: PASS
remote unit-install-fails: PASS
remote unit-move-fails: PASS
remote recovery-tag-fails: PASS
remote all-current-tags-fail: PASS
remote load-fails: PASS
remote checksum-mismatch: PASS
remote label-mismatch: PASS
remote id-mismatch: PASS
remote pre-unhealthy: PASS
remote open-secret: PASS
remote missing-secret: PASS
remote symlink-secret: PASS
remote unlabelled-baseline: PASS
remote persistent-daemon-reload: PASS
remote persistent-unit-rename: PASS
remote stop-returns-error-after-stopping: PASS
remote post-stop-podman-error: PASS
remote recovery-start-fails: PASS
remote spaced-open-secret: PASS
remote spaced-new-open-secret: PASS
remote multiple-open-secret: PASS
remote optional-open-secret: PASS
remote service-open-secret: PASS
remote continued-open-secret: PASS
remote quoted-open-secret: PASS
remote unsupported-specifier: PASS
remote unsupported-escape: PASS
remote repeated-open-secret: PASS
remote spaced-safe-secret: PASS
remote multiple-safe-secret: PASS
remote optional-safe-secret: PASS
remote service-safe-secret: PASS
remote continued-safe-secret: PASS
remote quoted-safe-secret: PASS
remote home-safe-secret: PASS
remote percent-safe-secret: PASS
remote signal: PASS
remote rerun: PASS
remote manual-rollback: PASS
remote concurrent upload/switch serialized: PASS
injection deploy/normal/before: 40/52 PASS
injection deploy/normal/after: 40/52 PASS
state machine deploy/normal: 52 steps, before/after failures PASS
state machine same-sha/normal: 28 steps, before/after failures PASS
state machine same-image/normal: 28 steps, before/after failures PASS
state machine rollback/normal: 39 steps, before/after failures PASS
state machine bootstrap/normal: 8 steps, before/after failures PASS
state machine bootstrap-missing-unit/normal: 5 steps, before/after failures PASS
injection deploy/health-fails/before: 40/63 PASS
injection deploy/health-fails/after: 40/63 PASS
state machine deploy/health-fails: 63 steps, before/after failures PASS
injection rollback/health-fails/before: 40/50 PASS
injection rollback/health-fails/after: 40/50 PASS
state machine rollback/health-fails: 50 steps, before/after failures PASS
injection deploy/stale-primary-recovery/before: 40/62 PASS
injection deploy/stale-primary-recovery/after: 40/62 PASS
state machine deploy/stale-primary-recovery: 62 steps, before/after failures PASS
injection rollback/stale-primary-recovery/before: 40/49 PASS
injection rollback/stale-primary-recovery/after: 40/49 PASS
state machine rollback/stale-primary-recovery: 49 steps, before/after failures PASS
injection deploy/post-start-unit-rename/before: 40/60 PASS
injection deploy/post-start-unit-rename/after: 40/60 PASS
state machine deploy/post-start-unit-rename: 60 steps, before/after failures PASS
injection rollback/post-start-unit-rename/before: 40/47 PASS
injection rollback/post-start-unit-rename/after: 40/47 PASS
state machine rollback/post-start-unit-rename: 47 steps, before/after failures PASS
exhaustive failure injection: PASS (982 cases; userdata unchanged)
relay success: PASS
relay save-fails: PASS
relay target-fails: PASS
non-dev revision and mutable tags rejected for next/cloud: PASS
data rollback options rejected: PASS
build normal: PASS
build dev-moved: PASS
build build-fails: PASS
build bad-id: PASS
build wrong-label: PASS
build save-fails: PASS
deploy and code rollback dry-run without transport (next/cloud): PASS
deploy regression gates: PASS (local mocks; no production contact)

Limits and choices: the automated bootstrap tests cover refusal, not the separate supervised maintenance procedure. Real Quadlet generation and the local systemd ExecStart field format were checked; the command mocks do not certify live target Podman/systemd/NetBird behavior. Persistent infrastructure failure retains repair material and reports failure; power loss and SIGKILL still require local repair.

The reviewer host could not be read: bounded NetBird SSH attempts timed out or failed DNS, including a final retry. I read comments 67031, 67032, 67040 and 67041, including the complete independent probe source, and reproduced both findings locally before fixing them.

Only fix/deploy-688 was pushed. No production connection, merge or deploy. No issue was closed. The commit contains product scripts, tests and lasting operational documentation only; evidence and gate output stay in issue comments.

deploy-fix-3 finished. Branch: fix/deploy-688. Base: a6a50fe52667b9e631085369d9cdf52270a92d84. Pushed head: 8dc52337a566158e86b0234bcfb082331d76d1cd. [Review this commit](https://git.kayg.org/kayg/calternal/commit/8dc52337a566158e86b0234bcfb082331d76d1cd). Forgejo reports PR #733 at this head. | Finding | Fix | Test | | --- | --- | --- | | P1 full-disk recovery can restart the candidate | Prepare the final old-digest unit before stop and rename it directly. Preload an independent old-digest recovery service. Check the loaded primary ExecStart before restarting it; otherwise use the fallback. No recovery copy or mutable current tag selects the image. | disk-full-after-start; stale-primary-recovery; post-start-unit-rename; real Quadlet generation; before/after failure sweeps | | P2 a rebuild of the current SHA erases rollback | A healthy current SHA is a deploy no-op even when the artifact digest differs. Rollback still selects the exact digest and SHA pair. | same-revision different digest; rerun; same-sha and same-image exhaustive paths | | Switch errors depend on where mutable state was read | One small switch.sh module owns the states, transitions, recovery and rollback invariant. The healthy candidate's rollback-tag write is the commit point; an error after the write is resolved by inspecting that pointer. | 982 before/after external-command failures across 12 paths; an independent successful revision-check record is required before accepting a committed candidate | | Recovery itself can fail | Use the preloaded digest-pinned service, make two bounded start/health attempts, retain private repair files on persistent failure, and refuse to delete an existing recovery unit. | recovery-start-fails; retained-recovery; stale-definition and rename-failure sweeps for deploy and rollback | | Bootstrap must not bypass provenance or touch data | Keep unlabelled and missing-unit baselines fail-closed. The supervised bootstrap remains separate; no storage provisioning was added. | both bootstrap refusal sweeps; unlabelled-baseline; all property cases compare userdata bytes, modes, ownership and timestamps and reject data-operation arguments | | The old bash -n wildcard checked only the first script | Parse every deploy script individually. | all-files bash syntax, ShellCheck, staged whitespace check | Validation: /usr/local/bin/build-slot.sh bash deploy/tests/gate.sh exited 0. The exhaustive fixtures use at most three independent workers inside that one slot. Final all-files syntax, ShellCheck and staged whitespace checks also passed. Gate output verbatim: ```text bash -n: PASS shellcheck: PASS diff whitespace: PASS real Quadlet generation (main and immutable recovery): PASS remote normal: PASS remote stale-recovery-definition: PASS remote stale-primary-recovery: PASS remote post-start-unit-rename: PASS remote retained-recovery: PASS remote same-revision different digest: PASS remote disk-full-after-start: PASS remote disk-full: PASS remote tag-fails: PASS remote switch-tag-fails: PASS remote start-fails: PASS remote health-fails: PASS remote stale-response: PASS remote inactive-systemd: PASS remote wrong-container-image: PASS remote stop-fails: PASS remote reload-fails: PASS remote rollback-tag-fails: PASS remote unit-install-fails: PASS remote unit-move-fails: PASS remote recovery-tag-fails: PASS remote all-current-tags-fail: PASS remote load-fails: PASS remote checksum-mismatch: PASS remote label-mismatch: PASS remote id-mismatch: PASS remote pre-unhealthy: PASS remote open-secret: PASS remote missing-secret: PASS remote symlink-secret: PASS remote unlabelled-baseline: PASS remote persistent-daemon-reload: PASS remote persistent-unit-rename: PASS remote stop-returns-error-after-stopping: PASS remote post-stop-podman-error: PASS remote recovery-start-fails: PASS remote spaced-open-secret: PASS remote spaced-new-open-secret: PASS remote multiple-open-secret: PASS remote optional-open-secret: PASS remote service-open-secret: PASS remote continued-open-secret: PASS remote quoted-open-secret: PASS remote unsupported-specifier: PASS remote unsupported-escape: PASS remote repeated-open-secret: PASS remote spaced-safe-secret: PASS remote multiple-safe-secret: PASS remote optional-safe-secret: PASS remote service-safe-secret: PASS remote continued-safe-secret: PASS remote quoted-safe-secret: PASS remote home-safe-secret: PASS remote percent-safe-secret: PASS remote signal: PASS remote rerun: PASS remote manual-rollback: PASS remote concurrent upload/switch serialized: PASS injection deploy/normal/before: 40/52 PASS injection deploy/normal/after: 40/52 PASS state machine deploy/normal: 52 steps, before/after failures PASS state machine same-sha/normal: 28 steps, before/after failures PASS state machine same-image/normal: 28 steps, before/after failures PASS state machine rollback/normal: 39 steps, before/after failures PASS state machine bootstrap/normal: 8 steps, before/after failures PASS state machine bootstrap-missing-unit/normal: 5 steps, before/after failures PASS injection deploy/health-fails/before: 40/63 PASS injection deploy/health-fails/after: 40/63 PASS state machine deploy/health-fails: 63 steps, before/after failures PASS injection rollback/health-fails/before: 40/50 PASS injection rollback/health-fails/after: 40/50 PASS state machine rollback/health-fails: 50 steps, before/after failures PASS injection deploy/stale-primary-recovery/before: 40/62 PASS injection deploy/stale-primary-recovery/after: 40/62 PASS state machine deploy/stale-primary-recovery: 62 steps, before/after failures PASS injection rollback/stale-primary-recovery/before: 40/49 PASS injection rollback/stale-primary-recovery/after: 40/49 PASS state machine rollback/stale-primary-recovery: 49 steps, before/after failures PASS injection deploy/post-start-unit-rename/before: 40/60 PASS injection deploy/post-start-unit-rename/after: 40/60 PASS state machine deploy/post-start-unit-rename: 60 steps, before/after failures PASS injection rollback/post-start-unit-rename/before: 40/47 PASS injection rollback/post-start-unit-rename/after: 40/47 PASS state machine rollback/post-start-unit-rename: 47 steps, before/after failures PASS exhaustive failure injection: PASS (982 cases; userdata unchanged) relay success: PASS relay save-fails: PASS relay target-fails: PASS non-dev revision and mutable tags rejected for next/cloud: PASS data rollback options rejected: PASS build normal: PASS build dev-moved: PASS build build-fails: PASS build bad-id: PASS build wrong-label: PASS build save-fails: PASS deploy and code rollback dry-run without transport (next/cloud): PASS deploy regression gates: PASS (local mocks; no production contact) ``` Limits and choices: the automated bootstrap tests cover refusal, not the separate supervised maintenance procedure. Real Quadlet generation and the local systemd ExecStart field format were checked; the command mocks do not certify live target Podman/systemd/NetBird behavior. Persistent infrastructure failure retains repair material and reports failure; power loss and SIGKILL still require local repair. The reviewer host could not be read: bounded NetBird SSH attempts timed out or failed DNS, including a final retry. I read comments 67031, 67032, 67040 and 67041, including the complete independent probe source, and reproduced both findings locally before fixing them. Only fix/deploy-688 was pushed. No production connection, merge or deploy. No issue was closed. The commit contains product scripts, tests and lasting operational documentation only; evidence and gate output stay in issue comments.
kayg commented 2026-10-28 23:42:18 +00:00
Author
Owner
Copy link

Starting deploy-fix-6 on staging 8by64-compute01. Base d2ade1fb93, fetched fix/deploy-688-round5. Working branch fix/deploy-688-round6: the job's hard rule requires a new branch and PR because this forge ignores pushes to an open PR branch. I will link the replacement here and keep #748 open.

Scope: scripted bootstrap with all image/unit writes and free-space checks before stop; missing first predecessor; worker umask; FIFO producer race. Gates: bash -n, shellcheck, all deploy tests, real staging bootstrap/full-disk/deploy/same-SHA/rollback rehearsal. No production contact, merge, or production deploy. Evidence stays untracked and in issue comments.

Starting deploy-fix-6 on staging 8by64-compute01. Base d2ade1fb931b4ac74fb698015fcc95c3e7cf69c5, fetched fix/deploy-688-round5. Working branch fix/deploy-688-round6: the job's hard rule requires a new branch and PR because this forge ignores pushes to an open PR branch. I will link the replacement here and keep #748 open. Scope: scripted bootstrap with all image/unit writes and free-space checks before stop; missing first predecessor; worker umask; FIFO producer race. Gates: bash -n, shellcheck, all deploy tests, real staging bootstrap/full-disk/deploy/same-SHA/rollback rehearsal. No production contact, merge, or production deploy. Evidence stays untracked and in issue comments.
kayg commented 2026-10-28 23:43:59 +00:00
Author
Owner
Copy link

Finding → fix → test, round 6 in progress:

  1. P1 bootstrap: README steps 6–8 stop before tagging; the review's persistent ENOSPC model strands recovery. Replacing the manual procedure with deploy.sh bootstrap, routed through the existing detached lock, free-space admission, runtime reserves, scratch generator, prepared unit/pointer files and recovery. The running legacy digest is explicitly recorded before stop; only that digest permits status-only health, without fabricating a revision. New images still require the dev SHA. README becomes a dry-run and one command.

  2. P2 first update: switch_prepare required :rollback even when a healthy labelled baseline had no predecessor. The image record now represents an absent predecessor as -; commit still records the old baseline as the real predecessor. Regression paths exercise the first update and its failures.

  3. P2 worker permissions: the parent umask does not cross systemd-run. Worker now sets umask 077 itself. Real user-systemd tests under UMask=0002 pass and verify receipt 0600 and enclosing directory 0700.

  4. P2 FIFO race: shell > recreates the path after an early receiver exit. Producer now opens without O_CREAT, refuses symlinks and verifies the opened descriptor is a FIFO. A handshake forces receiver completion before producer open; mocks and real user systemd both pass with no retained artifact from a 262144-byte payload. Existing disconnect, abandoned upload, partial receipt, failure result and lock tests pass as well.

Evidence: .job-evidence/round6/transport.log (not committed). Full fault injection and real app rehearsals remain to run.

Finding → fix → test, round 6 in progress: 1. P1 bootstrap: README steps 6–8 stop before tagging; the review's persistent ENOSPC model strands recovery. Replacing the manual procedure with `deploy.sh bootstrap`, routed through the existing detached lock, free-space admission, runtime reserves, scratch generator, prepared unit/pointer files and recovery. The running legacy digest is explicitly recorded before stop; only that digest permits status-only health, without fabricating a revision. New images still require the dev SHA. README becomes a dry-run and one command. 2. P2 first update: switch_prepare required `:rollback` even when a healthy labelled baseline had no predecessor. The image record now represents an absent predecessor as `-`; commit still records the old baseline as the real predecessor. Regression paths exercise the first update and its failures. 3. P2 worker permissions: the parent umask does not cross systemd-run. Worker now sets umask 077 itself. Real user-systemd tests under UMask=0002 pass and verify receipt 0600 and enclosing directory 0700. 4. P2 FIFO race: shell `>` recreates the path after an early receiver exit. Producer now opens without O_CREAT, refuses symlinks and verifies the opened descriptor is a FIFO. A handshake forces receiver completion before producer open; mocks and real user systemd both pass with no retained artifact from a 262144-byte payload. Existing disconnect, abandoned upload, partial receipt, failure result and lock tests pass as well. Evidence: .job-evidence/round6/transport.log (not committed). Full fault injection and real app rehearsals remain to run.
kayg commented 2026-10-28 23:48:29 +00:00
Author
Owner
Copy link

Real rehearsal progress on staging only: installed next normal deploy, same-SHA redeploy (same MainPID), and rollback pass. Disposable real Podman/systemd bootstrap from an unlabelled image passes; failed candidate health recovers the legacy image and original unit bytes. Both full-disk bootstrap admission and exhaustion after admission retain the exact old MainPID at zero available blocks.

The extra legacy rollback run correctly refused free-space admission in the 1 GiB fixture: legacy image size 115693123 bytes requires about 523 MiB headroom while about 461 MiB remained. This is a fixture sizing failure, not a service outage or bypass. The old candidate stayed healthy; cleanup completed. First-attempt logs are retained. Repeating the entire rehearsal with a dedicated 2 GiB filesystem, including explicit proof that the after-admission exhaustion hook completed before proceeding.

Real rehearsal progress on staging only: installed next normal deploy, same-SHA redeploy (same MainPID), and rollback pass. Disposable real Podman/systemd bootstrap from an unlabelled image passes; failed candidate health recovers the legacy image and original unit bytes. Both full-disk bootstrap admission and exhaustion after admission retain the exact old MainPID at zero available blocks. The extra legacy rollback run correctly refused free-space admission in the 1 GiB fixture: legacy image size 115693123 bytes requires about 523 MiB headroom while about 461 MiB remained. This is a fixture sizing failure, not a service outage or bypass. The old candidate stayed healthy; cleanup completed. First-attempt logs are retained. Repeating the entire rehearsal with a dedicated 2 GiB filesystem, including explicit proof that the after-admission exhaustion hook completed before proceeding.
kayg commented 2026-10-28 23:49:43 +00:00
Author
Owner
Copy link

Round-six real staging rehearsal: PASS. All 12 rows below ran on 8by64-compute01 using real rootless Podman, real user systemd, the installed Quadlet generator and the real app. Existing A/B images and the original unlabelled image were reused; this script-only change requires no app rebuild. Normal deploy/rollback used the installed next service. Bootstrap and ENOSPC used a separate disposable real app and a bounded dedicated 2 GiB ext4 image store; no shared filesystem was filled.

Run Result Evidence
real staging normal deploy PASS mount roots: unchanged
real staging same SHA, same MainPID PASS mount roots: unchanged
real staging rollback PASS mount roots: unchanged
bootstrap full filesystem admission PASS old service: same MainPID; available blocks: 0
bootstrap full filesystem after admission PASS old service: same MainPID; available blocks: 0
bootstrap candidate failure recovers unlabelled image PASS unit bytes: original
bootstrap from unlabelled running image PASS exact image and pointer identities verified
rollback to recorded legacy image PASS health: status and exact digest
normal deploy after bootstrap removes legacy exception PASS exact image and pointer identities verified
same SHA after bootstrap PASS old service: same MainPID
normal rollback after bootstrap PASS exact image and pointer identities verified
final staging state PASS mount roots: unchanged

Both full-disk rows verified zero available blocks with the original MainPID continuously alive. The after-admission hook separately recorded zero blocks before image-load/tag/preparation could complete. Failed candidate health restored the exact unlabelled digest and original unit bytes. Explicit legacy rollback verified status and digest; normal deploy subsequently removed the legacy exception. Staging finishes at A with rollback B and byte-identical original installed unit; mount roots match entry. Disposable service/store/data were cleaned. Per-run logs and rehearsal source are retained untracked and will be attached with the final gate output.

Full gate now running through one local build slot, with three injection workers. No production contact, merge, or production deploy.

Round-six real staging rehearsal: PASS. All 12 rows below ran on 8by64-compute01 using real rootless Podman, real user systemd, the installed Quadlet generator and the real app. Existing A/B images and the original unlabelled image were reused; this script-only change requires no app rebuild. Normal deploy/rollback used the installed next service. Bootstrap and ENOSPC used a separate disposable real app and a bounded dedicated 2 GiB ext4 image store; no shared filesystem was filled. | Run | Result | Evidence | | --- | --- | --- | | real staging normal deploy | PASS | mount roots: unchanged | | real staging same SHA, same MainPID | PASS | mount roots: unchanged | | real staging rollback | PASS | mount roots: unchanged | | bootstrap full filesystem admission | PASS | old service: same MainPID; available blocks: 0 | | bootstrap full filesystem after admission | PASS | old service: same MainPID; available blocks: 0 | | bootstrap candidate failure recovers unlabelled image | PASS | unit bytes: original | | bootstrap from unlabelled running image | PASS | exact image and pointer identities verified | | rollback to recorded legacy image | PASS | health: status and exact digest | | normal deploy after bootstrap removes legacy exception | PASS | exact image and pointer identities verified | | same SHA after bootstrap | PASS | old service: same MainPID | | normal rollback after bootstrap | PASS | exact image and pointer identities verified | | final staging state | PASS | mount roots: unchanged | Both full-disk rows verified zero available blocks with the original MainPID continuously alive. The after-admission hook separately recorded zero blocks before image-load/tag/preparation could complete. Failed candidate health restored the exact unlabelled digest and original unit bytes. Explicit legacy rollback verified status and digest; normal deploy subsequently removed the legacy exception. Staging finishes at A with rollback B and byte-identical original installed unit; mount roots match entry. Disposable service/store/data were cleaned. Per-run logs and rehearsal source are retained untracked and will be attached with the final gate output. Full gate now running through one local build slot, with three injection workers. No production contact, merge, or production deploy.
kayg commented 2026-10-29 00:16:13 +00:00
Author
Owner
Copy link

Additional finding during this fix's verification: the first absence-handling implementation conflated an image-inspection error with an absent rollback tag. A targeted injection at the rollback image inspection, followed by candidate health failure, reproduced old-image recovery with the older predecessor erased to -. Evidence: .job-evidence/round6/predecessor-red.log. This did not affect the earlier real staging rehearsal, which had no runtime inspection errors.

I stopped only this job's recorded timeout PID 1452960 after verifying its exact command and owner. The partial sweep is retained with exit 143; it is not a full gate pass. No other process was signalled.

Fix: distinguish Podman's explicit image-exists result 1 (absent) from result 0 (present, inspection must succeed) and runtime errors (abort before switch). This keeps real absence supported while preserving an existing predecessor on errors. Targeted before/after injections and explicit runtime/inspection error tests are being run, then the full real rehearsal and gate will be repeated from the start.

Additional finding during this fix's verification: the first absence-handling implementation conflated an image-inspection error with an absent rollback tag. A targeted injection at the rollback image inspection, followed by candidate health failure, reproduced old-image recovery with the older predecessor erased to `-`. Evidence: .job-evidence/round6/predecessor-red.log. This did not affect the earlier real staging rehearsal, which had no runtime inspection errors. I stopped only this job's recorded timeout PID 1452960 after verifying its exact command and owner. The partial sweep is retained with exit 143; it is not a full gate pass. No other process was signalled. Fix: distinguish Podman's explicit image-exists result 1 (absent) from result 0 (present, inspection must succeed) and runtime errors (abort before switch). This keeps real absence supported while preserving an existing predecessor on errors. Targeted before/after injections and explicit runtime/inspection error tests are being run, then the full real rehearsal and gate will be repeated from the start.
kayg commented 2026-10-29 00:19:12 +00:00
Author
Owner
Copy link

After correcting the predecessor inspection/absence edge case, the entire real staging rehearsal was repeated from the start and all 12 rows passed again. Actual Podman image-exists results were explicitly checked: present image = 0, absent image = 1. Both zero-block bootstrap cases keep the old MainPID unchanged; failed bootstrap health recovers the exact unlabelled image; legacy rollback, normal deploy, same-SHA and normal rollback all pass. Staging again returns to A with rollback B, original installed unit bytes and unchanged mount identities. Disposable resources are removed.

Targeted predecessor existence and inspection failures before/after each command now preserve the older rollback identity through failed health. A fresh full gate is running from the beginning. The prior interrupted partial sweep remains recorded as exit 143 and is not claimed as a pass. Final evidence will include both the reproduced edge case and its passing tests, plus every rehearsal run log.

After correcting the predecessor inspection/absence edge case, the entire real staging rehearsal was repeated from the start and all 12 rows passed again. Actual Podman image-exists results were explicitly checked: present image = 0, absent image = 1. Both zero-block bootstrap cases keep the old MainPID unchanged; failed bootstrap health recovers the exact unlabelled image; legacy rollback, normal deploy, same-SHA and normal rollback all pass. Staging again returns to A with rollback B, original installed unit bytes and unchanged mount identities. Disposable resources are removed. Targeted predecessor existence and inspection failures before/after each command now preserve the older rollback identity through failed health. A fresh full gate is running from the beginning. The prior interrupted partial sweep remains recorded as exit 143 and is not claimed as a pass. Final evidence will include both the reproduced edge case and its passing tests, plus every rehearsal run log.
kayg commented 2026-10-29 05:47:13 +00:00
Author
Owner
Copy link

Resuming after host reboot on fix/deploy-688-round5, base/head 768dbacbb6. The coordinator already pushed the WIP to PR #748; this task explicitly requires updates only to that branch and PR, overriding the earlier generic new-branch instruction. Rechecking staging, runner and fixtures; rerunning the interrupted full gate and real staging rehearsals. No production access or merge. Evidence remains untracked.

Resuming after host reboot on fix/deploy-688-round5, base/head 768dbacbb6b25773af560b5e10f69a9fbd88f1da. The coordinator already pushed the WIP to PR #748; this task explicitly requires updates only to that branch and PR, overriding the earlier generic new-branch instruction. Rechecking staging, runner and fixtures; rerunning the interrupted full gate and real staging rehearsals. No production access or merge. Evidence remains untracked.
kayg commented 2026-10-29 05:47:39 +00:00
Author
Owner
Copy link

Post-reboot evidence on 8by64-compute01: boot ID 3e68dc6c-c253-49bd-a643-7dc507439df6. calternal-next.service active under user systemd; container running image sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb. Health revision 8dc52337a5, status ok. Saved pointer current matches that image; rollback sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108. Both forgejo-runner.service and forgejo-runner-web.service active since 06:43:49 CET with NRestarts=0. No leftover disposable round6/round6-extra stores. A/B archives and bundle fixtures remain present. Root fixture access works. Previous gate stopped without gate.exit at bootstrap-health-fails after-command 80/106; not a completed gate. Additional real regression evidence directory contains fixture symlinks only; those rehearsals have not run. Rerunning all gates and real rehearsals through one build slot, without changing production or filling a shared filesystem.

Post-reboot evidence on 8by64-compute01: boot ID 3e68dc6c-c253-49bd-a643-7dc507439df6. calternal-next.service active under user systemd; container running image sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb. Health revision 8dc52337a566158e86b0234bcfb082331d76d1cd, status ok. Saved pointer current matches that image; rollback sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108. Both forgejo-runner.service and forgejo-runner-web.service active since 06:43:49 CET with NRestarts=0. No leftover disposable round6/round6-extra stores. A/B archives and bundle fixtures remain present. Root fixture access works. Previous gate stopped without gate.exit at bootstrap-health-fails after-command 80/106; not a completed gate. Additional real regression evidence directory contains fixture symlinks only; those rehearsals have not run. Rerunning all gates and real rehearsals through one build slot, without changing production or filling a shared filesystem.
kayg commented 2026-10-29 05:51:25 +00:00
Author
Owner
Copy link

Fresh post-reboot bootstrap rehearsal: all 12 rows PASS, command exit 0. Real next deploy, same-SHA with same MainPID, rollback, unlabelled bootstrap, legacy rollback, failed bootstrap health recovery, normal post-bootstrap deploy/same-SHA/rollback and final restoration passed. Both full-disk bootstrap cases (before admission and after admission before image load) observed zero available blocks and the same original service MainPID. Staging returns to A/rollback B, byte-identical installed unit and unchanged mount roots. Per-run logs are retained untracked and will be attached with final gates. Additional real regression rehearsal is running, then the interrupted exhaustive gate will rerun from its start.

{"case": "real staging normal deploy", "outcome": "PASS", "current": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "rollback": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "revision": "5b0dd8d009", "mount_roots": "unchanged"}
{"case": "real staging same SHA, same MainPID", "outcome": "PASS", "current": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "rollback": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "revision": "5b0dd8d009", "mount_roots": "unchanged"}
{"case": "real staging rollback", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "revision": "8dc52337a5", "mount_roots": "unchanged"}
real Podman image-exists present=0 absent=1: PASS
{"case": "bootstrap full filesystem admission", "outcome": "PASS", "current": "sha256:983a8a530cb4330dadeb8c3f125ea2cfec9b242ae21c571d4fc6e964b5a1e17d", "old_service": "same MainPID", "available_blocks": 0}
{"case": "bootstrap full filesystem after admission", "outcome": "PASS", "current": "sha256:983a8a530cb4330dadeb8c3f125ea2cfec9b242ae21c571d4fc6e964b5a1e17d", "old_service": "same MainPID", "available_blocks": 0}
{"case": "bootstrap candidate failure recovers unlabelled image", "outcome": "PASS", "current": "sha256:983a8a530cb4330dadeb8c3f125ea2cfec9b242ae21c571d4fc6e964b5a1e17d", "rollback": "-", "unit_bytes": "original"}
{"case": "bootstrap from unlabelled running image", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:983a8a530cb4330dadeb8c3f125ea2cfec9b242ae21c571d4fc6e964b5a1e17d", "revision": "8dc52337a5"}
{"case": "rollback to recorded legacy image", "outcome": "PASS", "current": "sha256:983a8a530cb4330dadeb8c3f125ea2cfec9b242ae21c571d4fc6e964b5a1e17d", "rollback": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "health": "status and exact digest"}
{"case": "normal deploy after bootstrap removes legacy exception", "outcome": "PASS", "current": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "rollback": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "revision": "5b0dd8d009"}
{"case": "same SHA after bootstrap", "outcome": "PASS", "current": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "rollback": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "old_service": "same MainPID"}
{"case": "normal rollback after bootstrap", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "revision": "8dc52337a5"}
{"case": "final staging state", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "revision": "8dc52337a5", "mount_roots": "unchanged"}
round six real staging rehearsal: PASS

Fresh post-reboot bootstrap rehearsal: all 12 rows PASS, command exit 0. Real next deploy, same-SHA with same MainPID, rollback, unlabelled bootstrap, legacy rollback, failed bootstrap health recovery, normal post-bootstrap deploy/same-SHA/rollback and final restoration passed. Both full-disk bootstrap cases (before admission and after admission before image load) observed zero available blocks and the same original service MainPID. Staging returns to A/rollback B, byte-identical installed unit and unchanged mount roots. Per-run logs are retained untracked and will be attached with final gates. Additional real regression rehearsal is running, then the interrupted exhaustive gate will rerun from its start. {"case": "real staging normal deploy", "outcome": "PASS", "current": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "rollback": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "revision": "5b0dd8d00916eed389a47b7aee1e4d2186240ad6", "mount_roots": "unchanged"} {"case": "real staging same SHA, same MainPID", "outcome": "PASS", "current": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "rollback": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "revision": "5b0dd8d00916eed389a47b7aee1e4d2186240ad6", "mount_roots": "unchanged"} {"case": "real staging rollback", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "revision": "8dc52337a566158e86b0234bcfb082331d76d1cd", "mount_roots": "unchanged"} real Podman image-exists present=0 absent=1: PASS {"case": "bootstrap full filesystem admission", "outcome": "PASS", "current": "sha256:983a8a530cb4330dadeb8c3f125ea2cfec9b242ae21c571d4fc6e964b5a1e17d", "old_service": "same MainPID", "available_blocks": 0} {"case": "bootstrap full filesystem after admission", "outcome": "PASS", "current": "sha256:983a8a530cb4330dadeb8c3f125ea2cfec9b242ae21c571d4fc6e964b5a1e17d", "old_service": "same MainPID", "available_blocks": 0} {"case": "bootstrap candidate failure recovers unlabelled image", "outcome": "PASS", "current": "sha256:983a8a530cb4330dadeb8c3f125ea2cfec9b242ae21c571d4fc6e964b5a1e17d", "rollback": "-", "unit_bytes": "original"} {"case": "bootstrap from unlabelled running image", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:983a8a530cb4330dadeb8c3f125ea2cfec9b242ae21c571d4fc6e964b5a1e17d", "revision": "8dc52337a566158e86b0234bcfb082331d76d1cd"} {"case": "rollback to recorded legacy image", "outcome": "PASS", "current": "sha256:983a8a530cb4330dadeb8c3f125ea2cfec9b242ae21c571d4fc6e964b5a1e17d", "rollback": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "health": "status and exact digest"} {"case": "normal deploy after bootstrap removes legacy exception", "outcome": "PASS", "current": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "rollback": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "revision": "5b0dd8d00916eed389a47b7aee1e4d2186240ad6"} {"case": "same SHA after bootstrap", "outcome": "PASS", "current": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "rollback": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "old_service": "same MainPID"} {"case": "normal rollback after bootstrap", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "revision": "8dc52337a566158e86b0234bcfb082331d76d1cd"} {"case": "final staging state", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "revision": "8dc52337a566158e86b0234bcfb082331d76d1cd", "mount_roots": "unchanged"} round six real staging rehearsal: PASS
kayg commented 2026-10-29 05:52:32 +00:00
Author
Owner
Copy link

Fresh additional real staging rehearsal: 13 rows PASS, command exit 0. Together with the bootstrap rehearsal, all 25 rows now pass after reboot. Persistent ENOSPC remained at zero blocks at prepared, candidate_started and recovery_health; the filler stayed until recovery was verified. Real invalid Quadlet output loss recovered via the precompiled static unit. Terminating only this job's recorded loopback SSH client after stop did not abort detached failed-health recovery; result 001 was persisted and the original A image/unit with rollback B restored. Fixtures cleaned. Full gate has restarted from the beginning in one local build slot. Evidence below; final archive and verbatim gates follow on completion.

{"case": "real staging deploy", "outcome": "PASS", "current": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "rollback": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "revision": "5b0dd8d009", "mounts_and_data_roots": "unchanged"}
{"case": "real staging same SHA", "outcome": "PASS", "current": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "rollback": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "revision": "5b0dd8d009", "mounts_and_data_roots": "unchanged"}
{"case": "real staging rollback", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "revision": "8dc52337a5", "mounts_and_data_roots": "unchanged"}
{"case": "real full receipt filesystem cleans partial upload", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "revision": "8dc52337a5", "mounts_and_data_roots": "unchanged"}
{"case": "real staging invalid generator input", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "revision": "8dc52337a5", "mounts_and_data_roots": "unchanged"}
{"case": "real staging failed candidate recovery", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "revision": "8dc52337a5", "mounts_and_data_roots": "unchanged"}
{"case": "isolated store paths verified", "outcome": "PASS", "graph_root": "/home/calternal/.deploy-round6-extra/graph", "run_root": "/run/user/30033/calternal-round6-extra-run"}
{"case": "isolated real Podman deploy and rollback", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "store": "dedicated ext4"}
{"case": "persistent full image store through recovery", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "zero_free_blocks_at": ["candidate_started", "prepared", "recovery_health"], "filler": "retained until recovery verified"}
{"case": "free-space margin refuses next deploy", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "old_service": "same MainPID throughout", "available_bytes": 0}
{"case": "final real staging check", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "revision": "8dc52337a5", "mounts_and_data_roots": "unchanged"}
real staging rehearsal: PASS
{"case": "real generator output loss and static fallback", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "revision": "8dc52337a5", "details": "Actual invalid Quadlet removed primary generator output; a precompiled static unit served A. Original unit and primary A restored under the deploy lock."}
{"case": "real SSH disconnect mid-switch", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "revision": "8dc52337a5", "details": "Killed the recorded loopback SSH client after the old service stopped; released candidate into failed health. PID 1 completed A recovery and persisted result 001 with original unit bytes and rollback B."}
real generator-loss and SSH-disconnect rehearsal: PASS

Fresh additional real staging rehearsal: 13 rows PASS, command exit 0. Together with the bootstrap rehearsal, all 25 rows now pass after reboot. Persistent ENOSPC remained at zero blocks at prepared, candidate_started and recovery_health; the filler stayed until recovery was verified. Real invalid Quadlet output loss recovered via the precompiled static unit. Terminating only this job's recorded loopback SSH client after stop did not abort detached failed-health recovery; result 001 was persisted and the original A image/unit with rollback B restored. Fixtures cleaned. Full gate has restarted from the beginning in one local build slot. Evidence below; final archive and verbatim gates follow on completion. {"case": "real staging deploy", "outcome": "PASS", "current": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "rollback": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "revision": "5b0dd8d00916eed389a47b7aee1e4d2186240ad6", "mounts_and_data_roots": "unchanged"} {"case": "real staging same SHA", "outcome": "PASS", "current": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "rollback": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "revision": "5b0dd8d00916eed389a47b7aee1e4d2186240ad6", "mounts_and_data_roots": "unchanged"} {"case": "real staging rollback", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "revision": "8dc52337a566158e86b0234bcfb082331d76d1cd", "mounts_and_data_roots": "unchanged"} {"case": "real full receipt filesystem cleans partial upload", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "revision": "8dc52337a566158e86b0234bcfb082331d76d1cd", "mounts_and_data_roots": "unchanged"} {"case": "real staging invalid generator input", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "revision": "8dc52337a566158e86b0234bcfb082331d76d1cd", "mounts_and_data_roots": "unchanged"} {"case": "real staging failed candidate recovery", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "revision": "8dc52337a566158e86b0234bcfb082331d76d1cd", "mounts_and_data_roots": "unchanged"} {"case": "isolated store paths verified", "outcome": "PASS", "graph_root": "/home/calternal/.deploy-round6-extra/graph", "run_root": "/run/user/30033/calternal-round6-extra-run"} {"case": "isolated real Podman deploy and rollback", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "store": "dedicated ext4"} {"case": "persistent full image store through recovery", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "zero_free_blocks_at": ["candidate_started", "prepared", "recovery_health"], "filler": "retained until recovery verified"} {"case": "free-space margin refuses next deploy", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "old_service": "same MainPID throughout", "available_bytes": 0} {"case": "final real staging check", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "revision": "8dc52337a566158e86b0234bcfb082331d76d1cd", "mounts_and_data_roots": "unchanged"} real staging rehearsal: PASS {"case": "real generator output loss and static fallback", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "revision": "8dc52337a566158e86b0234bcfb082331d76d1cd", "details": "Actual invalid Quadlet removed primary generator output; a precompiled static unit served A. Original unit and primary A restored under the deploy lock."} {"case": "real SSH disconnect mid-switch", "outcome": "PASS", "current": "sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb", "rollback": "sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108", "revision": "8dc52337a566158e86b0234bcfb082331d76d1cd", "details": "Killed the recorded loopback SSH client after the old service stopped; released candidate into failed health. PID 1 completed A recovery and persisted result 001 with original unit bytes and rollback B."} real generator-loss and SSH-disconnect rehearsal: PASS
kayg commented 2026-10-29 11:19:01 +00:00
Author
Owner
Copy link

Resuming round six on fix/deploy-688-round5 at base/head 768dbacbb6. Previous saved real staging and extra regression rehearsals exited 0 after reboot; exhaustive gate has no exit marker and is incomplete. Rechecking services and fixtures, then rerunning gates from the beginning under one build slot. The explicit job instruction to push only this branch and rehearse staging takes precedence over the general new-branch/no-deploy boilerplate. No production access or merge.

Resuming round six on fix/deploy-688-round5 at base/head 768dbacbb6b25773af560b5e10f69a9fbd88f1da. Previous saved real staging and extra regression rehearsals exited 0 after reboot; exhaustive gate has no exit marker and is incomplete. Rechecking services and fixtures, then rerunning gates from the beginning under one build slot. The explicit job instruction to push only this branch and rehearse staging takes precedence over the general new-branch/no-deploy boilerplate. No production access or merge.
kayg commented 2026-10-29 11:19:44 +00:00
Author
Owner
Copy link

Current resumption health check: calternal-next.service is active and its running digest is 60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb. /healthz returns {"revision":"8dc52337a566158e86b0234bcfb082331d76d1cd","status":"ok","version":"0.1.0"}. The private mode-0600 image record agrees and preserves rollback 26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108. Both forgejo-runner.service and forgejo-runner-web.service are active since 06:43:49 CET with NRestarts=0. There are no failed system units or leftover round6 disposable stores.

The saved post-reboot real staging rehearsals have both exit markers 0 and all 25 rows PASS. Their results were already posted at comments 69160 and 69169, and remain valid on the unchanged 768dbacb head and the same boot. No staging rerun is necessary. The interrupted exhaustive gate has no completed result; a fresh full run is now active in one build slot, with output kept separately from the interrupted log. No product edits are needed from this resumption's diff review so far.

Current resumption health check: calternal-next.service is active and its running digest is 60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb. /healthz returns {"revision":"8dc52337a566158e86b0234bcfb082331d76d1cd","status":"ok","version":"0.1.0"}. The private mode-0600 image record agrees and preserves rollback 26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108. Both forgejo-runner.service and forgejo-runner-web.service are active since 06:43:49 CET with NRestarts=0. There are no failed system units or leftover round6 disposable stores. The saved post-reboot real staging rehearsals have both exit markers 0 and all 25 rows PASS. Their results were already posted at comments 69160 and 69169, and remain valid on the unchanged 768dbacb head and the same boot. No staging rerun is necessary. The interrupted exhaustive gate has no completed result; a fresh full run is now active in one build slot, with output kept separately from the interrupted log. No product edits are needed from this resumption's diff review so far.
kayg commented 2026-10-29 11:21:38 +00:00
Author
Owner
Copy link

Evidence audit found the complete pre-reboot log still available on unchanged product/test source, with explicit PASS positions through bootstrap-health-fails after 80/106. Those results were not lost. To follow the instruction to resume completed work, stopped only the current resumption's recorded timeout PID 534890, verified by /proc command and owner. Its exit 143 is retained and is not a passing gate.

The remaining gate uses an untracked resume harness. It reads the original matrix list from the committed test's AST, runs each baseline again, checks retained step counts, and calls the original inject function on every position without an explicit retained PASS. It keeps the original three-worker maximum and one build slot. Syntax, shellcheck, Quadlet, transport, ordinary transaction, bootstrap, relay, artifact and dry-run gates run fresh. Final evidence will state retained and fresh counts, include both original logs and resume source, and will not describe this as an uninterrupted full rerun. All 25 real staging rows already passed on this boot and unchanged head.

Evidence audit found the complete pre-reboot log still available on unchanged product/test source, with explicit PASS positions through bootstrap-health-fails after 80/106. Those results were not lost. To follow the instruction to resume completed work, stopped only the current resumption's recorded timeout PID 534890, verified by /proc command and owner. Its exit 143 is retained and is not a passing gate. The remaining gate uses an untracked resume harness. It reads the original matrix list from the committed test's AST, runs each baseline again, checks retained step counts, and calls the original inject function on every position without an explicit retained PASS. It keeps the original three-worker maximum and one build slot. Syntax, shellcheck, Quadlet, transport, ordinary transaction, bootstrap, relay, artifact and dry-run gates run fresh. Final evidence will state retained and fresh counts, include both original logs and resume source, and will not describe this as an uninterrupted full rerun. All 25 real staging rows already passed on this boot and unchanged head.
kayg commented 2026-10-29 12:00:37 +00:00
Author
Owner
Copy link

Round-six verification checkpoint on unchanged head 768dbacbb6: 2802 injection positions are now fully accounted PASS from retained explicit results plus fresh missing positions. Bootstrap health-failure and persistent-full-disk sweeps are complete, as are failed deploy and failed rollback ordinary and persistent-full-disk sweeps. The original matrix continues through stale-primary recovery and post-start unit-rename recovery. No failed test or new product defect has appeared. This is still a running gate, not a final gate pass. All 25 real staging rehearsal rows remain recorded PASS after this boot. No product file changes, production access or merge.

Round-six verification checkpoint on unchanged head 768dbacbb6b25773af560b5e10f69a9fbd88f1da: 2802 injection positions are now fully accounted PASS from retained explicit results plus fresh missing positions. Bootstrap health-failure and persistent-full-disk sweeps are complete, as are failed deploy and failed rollback ordinary and persistent-full-disk sweeps. The original matrix continues through stale-primary recovery and post-start unit-rename recovery. No failed test or new product defect has appeared. This is still a running gate, not a final gate pass. All 25 real staging rehearsal rows remain recorded PASS after this boot. No product file changes, production access or merge.
kayg commented 2026-10-29 12:44:03 +00:00
Author
Owner
Copy link

Resuming deploy-fix-8 on fix/deploy-688-round5, base/head 768dbacbb6. Work only in /home/agent/wt/deploy-fix. The explicit task-specific instruction to push this branch overrides the older generic new-branch instruction. No production contact, merges, or production deployment. Post-reboot staging calternal-next and both CI runners active with zero restarts. Health reports expected 8dc52337a5; real running image 60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb. Both runners have active jobs; no failed system units. RAM available 58GiB; root free 124GiB; systemd-oomd active. Product/test source is unchanged from WIP. Retained explicit passing injection positions will be combined with fresh remaining positions; all other gates and real rehearsals will run on this boot, one build slot, maximum three workers. Evidence remains untracked.

Resuming deploy-fix-8 on fix/deploy-688-round5, base/head 768dbacbb6b25773af560b5e10f69a9fbd88f1da. Work only in /home/agent/wt/deploy-fix. The explicit task-specific instruction to push this branch overrides the older generic new-branch instruction. No production contact, merges, or production deployment. Post-reboot staging calternal-next and both CI runners active with zero restarts. Health reports expected 8dc52337a566158e86b0234bcfb082331d76d1cd; real running image 60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb. Both runners have active jobs; no failed system units. RAM available 58GiB; root free 124GiB; systemd-oomd active. Product/test source is unchanged from WIP. Retained explicit passing injection positions will be combined with fresh remaining positions; all other gates and real rehearsals will run on this boot, one build slot, maximum three workers. Evidence remains untracked.
kayg commented 2026-10-29 12:56:06 +00:00
Author
Owner
Copy link

Verification checkpoint on unchanged head 768dbacbb6: 3498 injection positions are fully accounted PASS from retained explicit results plus fresh positions. Stale-primary rollback recovery now passes every ordinary before/after failure and persistent-full-disk before-command failure; persistent after-command sweep is running. Fresh syntax, shellcheck, Quadlet, transport, transaction and bootstrap checks passed. No new product defect or code change. This is still a running gate, not a final gate pass. Fresh real staging rehearsals follow in the same single-slot sequence.

Verification checkpoint on unchanged head 768dbacbb6b25773af560b5e10f69a9fbd88f1da: 3498 injection positions are fully accounted PASS from retained explicit results plus fresh positions. Stale-primary rollback recovery now passes every ordinary before/after failure and persistent-full-disk before-command failure; persistent after-command sweep is running. Fresh syntax, shellcheck, Quadlet, transport, transaction and bootstrap checks passed. No new product defect or code change. This is still a running gate, not a final gate pass. Fresh real staging rehearsals follow in the same single-slot sequence.
kayg commented 2026-10-29 13:22:16 +00:00
Author
Owner
Copy link

Regression gate completed on unchanged head 768dbacbb6, exit 0. Exhaustive failure injection: 4,354 cases = 3,314 retained explicit passes + 1,040 fresh passes; userdata unchanged. Fresh syntax, shellcheck, all other deploy probes, dry runs and real user-systemd checks pass. Fresh real staging rehearsal is running next in one build slot. Final completion comment will include verbatim gate output and all real rehearsal rows. No code changes, production access or merge.

Regression gate completed on unchanged head 768dbacbb6b25773af560b5e10f69a9fbd88f1da, exit 0. Exhaustive failure injection: 4,354 cases = 3,314 retained explicit passes + 1,040 fresh passes; userdata unchanged. Fresh syntax, shellcheck, all other deploy probes, dry runs and real user-systemd checks pass. Fresh real staging rehearsal is running next in one build slot. Final completion comment will include verbatim gate output and all real rehearsal rows. No code changes, production access or merge.
kayg commented 2026-10-29 13:25:57 +00:00
Author
Owner
Copy link

Completed deploy-fix-8 on fix/deploy-688-round5, head 768dbacbb6b25773af560b5e10f69a9fbd88f1da. The WIP commit already contained all four fixes; review and verification found no additional product changes necessary. No new or empty commits were made. Pushed only the requested branch. No production contact, merges, or issue closure.

Finding → fix → test:

  • P1 bootstrap: scripted deploy.sh bootstrap replaces manual stop-before-tag steps. Verified image, unit and pointer preparation, free-space admission, static recovery and independent runtime reserves finish before stop. Real unlabelled bootstrap, candidate failure, legacy rollback and full disk before/after admission passed. Both bootstrap full-disk cases observed zero available blocks and preserved the exact old MainPID.
  • P2 first update: an absent predecessor is recorded as -; successful update records the old baseline. Runtime/inspection errors refuse before stop. Regression checks verify first update, failed first update, absent rollback refusal and preservation of older predecessors.
  • P2 worker permissions: receiver applies umask 077 itself. Mock and real user-systemd checks under UMask=0002 verify directory 0700 and receipt 0600.
  • P2 FIFO race: producer opens without O_CREAT, uses O_NOFOLLOW and checks FIFO type. Deterministic early receiver failure passes with mocks and real systemd, retaining none of the 262144 payload bytes.

Post-reboot health was verified before any rehearsal: both staging services (calternal-next and calternal-preview) active, zero restarts; next health and exact image/pointer identities matched expectations. Both CI runners active with zero restarts and running jobs. No failed system or calternal user units. CI slice MemoryMax=32 GiB and agent slice MemoryMax=24 GiB, with oomd enabled. All fresh heavy work used one build slot and no more than three workers. No build was needed.

Verification accounting: unchanged committed product/test source, with SHA-256 manifest attached. Explicit passing injection positions retained from the two interrupted runs were combined with freshly executed missing positions using the original injection function and matrix. Every baseline was rerun and boundary counts checked. This is a completed combined sweep, not an uninterrupted full rerun. Other gates and all 25 real rehearsal rows ran fresh on this boot. Evidence is untracked; no artifacts were committed.

Real staging rehearsal (fresh this boot):

Run Result
real staging normal deploy PASS
real staging same SHA, same MainPID PASS
real staging rollback PASS
bootstrap full filesystem admission PASS
bootstrap full filesystem after admission PASS
bootstrap candidate failure recovers unlabelled image PASS
bootstrap from unlabelled running image PASS
rollback to recorded legacy image PASS
normal deploy after bootstrap removes legacy exception PASS
same SHA after bootstrap PASS
normal rollback after bootstrap PASS
final staging state PASS
real staging deploy PASS
real staging same SHA PASS
real staging rollback PASS
real full receipt filesystem cleans partial upload PASS
real staging invalid generator input PASS
real staging failed candidate recovery PASS
isolated store paths verified PASS
isolated real Podman deploy and rollback PASS
persistent full image store through recovery PASS
free-space margin refuses next deploy PASS
final real staging check PASS
real generator output loss and static fallback PASS
real SSH disconnect mid-switch PASS

Rehearsals restored original staging unit bytes, image pointers and mount identities. Full disk cases used bounded dedicated filesystems; shared filesystems were never filled. Disposable fixtures were cleaned.

Complete evidence, retained output, resume source, source hashes and per-run logs.

Final health output, verbatim:

NRestarts=0
Id=forgejo-runner.service
ActiveState=active
SubState=running

NRestarts=0
Id=forgejo-runner-web.service
ActiveState=active
SubState=running
MemoryHigh=30064771072
MemoryMax=34359738368
ManagedOOMMemoryPressure=kill
Id=ci.slice

MemoryHigh=21474836480
MemoryMax=25769803776
ManagedOOMMemoryPressure=kill
Id=user-1002.slice
active
NRestarts=0
Id=calternal-next.service
ActiveState=active
SubState=running

NRestarts=0
Id=calternal-preview.service
ActiveState=active
SubState=running
calternal-next running=true image=60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb
  UNIT LOAD ACTIVE SUB DESCRIPTION

0 loaded units listed.
  UNIT LOAD ACTIVE SUB DESCRIPTION

0 loaded units listed.
image record: sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108
disposable leftovers: []
{"revision":"8dc52337a566158e86b0234bcfb082331d76d1cd","status":"ok","version":"0.1.0"}

Gate output, verbatim:

bash -n: PASS
shellcheck: PASS
diff whitespace: PASS
real Quadlet generation (main and immutable recovery): PASS
real Quadlet invalid candidate rejected before install: PASS
target receiver fails before producer open; no retained artifact: PASS
target caller lost before FIFO open cleans upload and releases lock: PASS
target partial receipt failure cleans private upload: PASS
target normal: PASS
target failed-transaction: PASS
target closed-output: PASS
target dropped-caller: PASS
target locked-receipt: PASS
remote predecessor-runtime-error: PASS
remote predecessor-inspection-error: PASS
remote malformed-pointer: PASS
remote multiline-pointer: PASS
remote foreign-legacy-pointer: PASS
remote equal-pointers: PASS
remote symlink-pointer: PASS
remote invalid-generator: PASS
remote closed-output: PASS
remote generator-fails: PASS
remote byte-exact-recovery: PASS
remote normal: PASS
remote stale-recovery-definition: PASS
remote stale-primary-recovery: PASS
remote post-start-unit-rename: PASS
remote retained-recovery: PASS
remote same-revision different digest: PASS
remote disk-full: PASS
remote tag-fails: PASS
remote pointer-commit-fails: PASS
remote start-fails: PASS
remote health-fails: PASS
remote stale-response: PASS
remote inactive-systemd: PASS
remote wrong-container-image: PASS
remote stop-fails: PASS
remote reload-fails: PASS
remote unit-install-fails: PASS
remote unit-move-fails: PASS
remote load-fails: PASS
remote checksum-mismatch: PASS
remote label-mismatch: PASS
remote id-mismatch: PASS
remote pre-unhealthy: PASS
remote open-secret: PASS
remote missing-secret: PASS
remote symlink-secret: PASS
remote unlabelled-baseline: PASS
remote persistent-daemon-reload: PASS
remote persistent-unit-rename: PASS
remote stop-returns-error-after-stopping: PASS
remote post-stop-podman-error: PASS
remote recovery-start-fails: PASS
remote spaced-open-secret: PASS
remote spaced-new-open-secret: PASS
remote multiple-open-secret: PASS
remote optional-open-secret: PASS
remote service-open-secret: PASS
remote continued-open-secret: PASS
remote quoted-open-secret: PASS
remote unsupported-specifier: PASS
remote unsupported-escape: PASS
remote repeated-open-secret: PASS
remote spaced-safe-secret: PASS
remote multiple-safe-secret: PASS
remote optional-safe-secret: PASS
remote service-safe-secret: PASS
remote continued-safe-secret: PASS
remote quoted-safe-secret: PASS
remote home-safe-secret: PASS
remote percent-safe-secret: PASS
remote signal: PASS
remote rerun: PASS
remote manual-rollback: PASS
remote concurrent upload/switch serialized: PASS
bootstrap legacy rollback verifies exact digest and status: PASS
failed first update preserves baseline; absent rollback refuses before stop: PASS
predecessor existence/inspection errors preserve older rollback through failed health: PASS
injection deploy/normal/before: retained 92/92 explicit PASS positions from unchanged source
injection deploy/normal/before: 92/92 accounted PASS
injection deploy/normal/after: retained 92/92 explicit PASS positions from unchanged source
injection deploy/normal/after: 92/92 accounted PASS
persistent disk deploy/normal/before: retained 92/92 explicit PASS positions from unchanged source
persistent disk deploy/normal/before: 92/92 accounted PASS
persistent disk deploy/normal/after: retained 92/92 explicit PASS positions from unchanged source
persistent disk deploy/normal/after: 92/92 accounted PASS
injection same-sha/normal/before: retained 34/34 explicit PASS positions from unchanged source
injection same-sha/normal/before: 34/34 accounted PASS
injection same-sha/normal/after: retained 34/34 explicit PASS positions from unchanged source
injection same-sha/normal/after: 34/34 accounted PASS
injection same-image/normal/before: retained 34/34 explicit PASS positions from unchanged source
injection same-image/normal/before: 34/34 accounted PASS
injection same-image/normal/after: retained 34/34 explicit PASS positions from unchanged source
injection same-image/normal/after: 34/34 accounted PASS
injection rollback/normal/before: retained 79/79 explicit PASS positions from unchanged source
injection rollback/normal/before: 79/79 accounted PASS
injection rollback/normal/after: retained 79/79 explicit PASS positions from unchanged source
injection rollback/normal/after: 79/79 accounted PASS
persistent disk rollback/normal/before: retained 79/79 explicit PASS positions from unchanged source
persistent disk rollback/normal/before: 79/79 accounted PASS
persistent disk rollback/normal/after: retained 79/79 explicit PASS positions from unchanged source
persistent disk rollback/normal/after: 79/79 accounted PASS
injection first-update/normal/before: retained 91/91 explicit PASS positions from unchanged source
injection first-update/normal/before: 91/91 accounted PASS
injection first-update/normal/after: retained 91/91 explicit PASS positions from unchanged source
injection first-update/normal/after: 91/91 accounted PASS
persistent disk first-update/normal/before: retained 91/91 explicit PASS positions from unchanged source
persistent disk first-update/normal/before: 91/91 accounted PASS
persistent disk first-update/normal/after: retained 91/91 explicit PASS positions from unchanged source
persistent disk first-update/normal/after: 91/91 accounted PASS
injection bootstrap-upgrade/normal/before: retained 91/91 explicit PASS positions from unchanged source
injection bootstrap-upgrade/normal/before: 91/91 accounted PASS
injection bootstrap-upgrade/normal/after: retained 91/91 explicit PASS positions from unchanged source
injection bootstrap-upgrade/normal/after: 91/91 accounted PASS
persistent disk bootstrap-upgrade/normal/before: retained 91/91 explicit PASS positions from unchanged source
persistent disk bootstrap-upgrade/normal/before: 91/91 accounted PASS
persistent disk bootstrap-upgrade/normal/after: retained 91/91 explicit PASS positions from unchanged source
persistent disk bootstrap-upgrade/normal/after: 91/91 accounted PASS
injection bootstrap-health-fails/normal/before: retained 106/106 explicit PASS positions from unchanged source
injection bootstrap-health-fails/normal/before: 106/106 accounted PASS
injection bootstrap-health-fails/normal/after: retained 106/106 explicit PASS positions from unchanged source
injection bootstrap-health-fails/normal/after: 106/106 accounted PASS
persistent disk bootstrap-health-fails/normal/before: retained 106/106 explicit PASS positions from unchanged source
persistent disk bootstrap-health-fails/normal/before: 106/106 accounted PASS
persistent disk bootstrap-health-fails/normal/after: retained 106/106 explicit PASS positions from unchanged source
persistent disk bootstrap-health-fails/normal/after: 106/106 accounted PASS
injection bootstrap/normal/before: retained 8/8 explicit PASS positions from unchanged source
injection bootstrap/normal/before: 8/8 accounted PASS
injection bootstrap/normal/after: retained 8/8 explicit PASS positions from unchanged source
injection bootstrap/normal/after: 8/8 accounted PASS
injection bootstrap-missing-unit/normal/before: retained 5/5 explicit PASS positions from unchanged source
injection bootstrap-missing-unit/normal/before: 5/5 accounted PASS
injection bootstrap-missing-unit/normal/after: retained 5/5 explicit PASS positions from unchanged source
injection bootstrap-missing-unit/normal/after: 5/5 accounted PASS
injection deploy/health-fails/before: retained 107/107 explicit PASS positions from unchanged source
injection deploy/health-fails/before: 107/107 accounted PASS
injection deploy/health-fails/after: retained 107/107 explicit PASS positions from unchanged source
injection deploy/health-fails/after: 107/107 accounted PASS
persistent disk deploy/health-fails/before: retained 107/107 explicit PASS positions from unchanged source
persistent disk deploy/health-fails/before: 107/107 accounted PASS
persistent disk deploy/health-fails/after: retained 107/107 explicit PASS positions from unchanged source
persistent disk deploy/health-fails/after: 107/107 accounted PASS
injection rollback/health-fails/before: retained 94/94 explicit PASS positions from unchanged source
injection rollback/health-fails/before: 94/94 accounted PASS
injection rollback/health-fails/after: retained 94/94 explicit PASS positions from unchanged source
injection rollback/health-fails/after: 94/94 accounted PASS
persistent disk rollback/health-fails/before: retained 94/94 explicit PASS positions from unchanged source
persistent disk rollback/health-fails/before: 94/94 accounted PASS
persistent disk rollback/health-fails/after: retained 94/94 explicit PASS positions from unchanged source
persistent disk rollback/health-fails/after: 94/94 accounted PASS
injection deploy/stale-primary-recovery/before: retained 105/105 explicit PASS positions from unchanged source
injection deploy/stale-primary-recovery/before: 105/105 accounted PASS
injection deploy/stale-primary-recovery/after: retained 105/105 explicit PASS positions from unchanged source
injection deploy/stale-primary-recovery/after: 105/105 accounted PASS
persistent disk deploy/stale-primary-recovery/before: retained 105/105 explicit PASS positions from unchanged source
persistent disk deploy/stale-primary-recovery/before: 105/105 accounted PASS
persistent disk deploy/stale-primary-recovery/after: retained 105/105 explicit PASS positions from unchanged source
persistent disk deploy/stale-primary-recovery/after: 105/105 accounted PASS
injection rollback/stale-primary-recovery/before: retained 92/92 explicit PASS positions from unchanged source
injection rollback/stale-primary-recovery/before: 92/92 accounted PASS
injection rollback/stale-primary-recovery/after: retained 0/92 explicit PASS positions from unchanged source
injection rollback/stale-primary-recovery/after: resumed 40/92 PASS
injection rollback/stale-primary-recovery/after: resumed 80/92 PASS
injection rollback/stale-primary-recovery/after: 92/92 accounted PASS
persistent disk rollback/stale-primary-recovery/before: retained 0/92 explicit PASS positions from unchanged source
persistent disk rollback/stale-primary-recovery/before: resumed 40/92 PASS
persistent disk rollback/stale-primary-recovery/before: resumed 80/92 PASS
persistent disk rollback/stale-primary-recovery/before: 92/92 accounted PASS
persistent disk rollback/stale-primary-recovery/after: retained 0/92 explicit PASS positions from unchanged source
persistent disk rollback/stale-primary-recovery/after: resumed 40/92 PASS
persistent disk rollback/stale-primary-recovery/after: resumed 80/92 PASS
persistent disk rollback/stale-primary-recovery/after: 92/92 accounted PASS
injection deploy/post-start-unit-rename/before: retained 0/102 explicit PASS positions from unchanged source
injection deploy/post-start-unit-rename/before: resumed 40/102 PASS
injection deploy/post-start-unit-rename/before: resumed 80/102 PASS
injection deploy/post-start-unit-rename/before: 102/102 accounted PASS
injection deploy/post-start-unit-rename/after: retained 0/102 explicit PASS positions from unchanged source
injection deploy/post-start-unit-rename/after: resumed 40/102 PASS
injection deploy/post-start-unit-rename/after: resumed 80/102 PASS
injection deploy/post-start-unit-rename/after: 102/102 accounted PASS
persistent disk deploy/post-start-unit-rename/before: retained 0/102 explicit PASS positions from unchanged source
persistent disk deploy/post-start-unit-rename/before: resumed 40/102 PASS
persistent disk deploy/post-start-unit-rename/before: resumed 80/102 PASS
persistent disk deploy/post-start-unit-rename/before: 102/102 accounted PASS
persistent disk deploy/post-start-unit-rename/after: retained 0/102 explicit PASS positions from unchanged source
persistent disk deploy/post-start-unit-rename/after: resumed 40/102 PASS
persistent disk deploy/post-start-unit-rename/after: resumed 80/102 PASS
persistent disk deploy/post-start-unit-rename/after: 102/102 accounted PASS
injection rollback/post-start-unit-rename/before: retained 0/89 explicit PASS positions from unchanged source
injection rollback/post-start-unit-rename/before: resumed 40/89 PASS
injection rollback/post-start-unit-rename/before: resumed 80/89 PASS
injection rollback/post-start-unit-rename/before: 89/89 accounted PASS
injection rollback/post-start-unit-rename/after: retained 0/89 explicit PASS positions from unchanged source
injection rollback/post-start-unit-rename/after: resumed 40/89 PASS
injection rollback/post-start-unit-rename/after: resumed 80/89 PASS
injection rollback/post-start-unit-rename/after: 89/89 accounted PASS
persistent disk rollback/post-start-unit-rename/before: retained 0/89 explicit PASS positions from unchanged source
persistent disk rollback/post-start-unit-rename/before: resumed 40/89 PASS
persistent disk rollback/post-start-unit-rename/before: resumed 80/89 PASS
persistent disk rollback/post-start-unit-rename/before: 89/89 accounted PASS
persistent disk rollback/post-start-unit-rename/after: retained 0/89 explicit PASS positions from unchanged source
persistent disk rollback/post-start-unit-rename/after: resumed 40/89 PASS
persistent disk rollback/post-start-unit-rename/after: resumed 80/89 PASS
persistent disk rollback/post-start-unit-rename/after: 89/89 accounted PASS
exhaustive failure injection resumed: PASS (4354 cases = 3314 retained + 1040 fresh; userdata unchanged)
relay success: PASS
relay save-fails: PASS
relay target-fails: PASS
relay bootstrap: PASS
non-dev revision and mutable tags rejected for next/cloud: PASS
data rollback options rejected: PASS
build normal: PASS
build dev-moved: PASS
build build-fails: PASS
build bad-id: PASS
build wrong-label: PASS
build save-fails: PASS
build export-wrong-label: PASS
build multiple-images: PASS
build root-owned-iid: PASS
deploy and code rollback dry-run without transport (next/cloud): PASS
deploy regression gates: PASS (completed with retained injection positions; local mocks; no production contact)
target receiver fails before producer open; no retained artifact: PASS (real user systemd)
target caller lost before FIFO open cleans upload and releases lock: PASS (real user systemd)
target normal: PASS (real user systemd)
target failed-transaction: PASS (real user systemd)
target closed-output: PASS (real user systemd)
target dropped-caller: PASS (real user systemd)
target locked-receipt: PASS (real user systemd)

Gate exit: 0. Real bootstrap/staging rehearsal exit: 0. Additional real regression and SSH-disconnect rehearsal exit: 0. Remote branch and PR head both verified as 768dbacbb6b25773af560b5e10f69a9fbd88f1da.

Completed deploy-fix-8 on `fix/deploy-688-round5`, head `768dbacbb6b25773af560b5e10f69a9fbd88f1da`. The WIP commit already contained all four fixes; review and verification found no additional product changes necessary. No new or empty commits were made. Pushed only the requested branch. No production contact, merges, or issue closure. Finding → fix → test: - P1 bootstrap: scripted `deploy.sh bootstrap` replaces manual stop-before-tag steps. Verified image, unit and pointer preparation, free-space admission, static recovery and independent runtime reserves finish before stop. Real unlabelled bootstrap, candidate failure, legacy rollback and full disk before/after admission passed. Both bootstrap full-disk cases observed zero available blocks and preserved the exact old MainPID. - P2 first update: an absent predecessor is recorded as `-`; successful update records the old baseline. Runtime/inspection errors refuse before stop. Regression checks verify first update, failed first update, absent rollback refusal and preservation of older predecessors. - P2 worker permissions: receiver applies umask 077 itself. Mock and real user-systemd checks under UMask=0002 verify directory 0700 and receipt 0600. - P2 FIFO race: producer opens without O_CREAT, uses O_NOFOLLOW and checks FIFO type. Deterministic early receiver failure passes with mocks and real systemd, retaining none of the 262144 payload bytes. Post-reboot health was verified before any rehearsal: both staging services (calternal-next and calternal-preview) active, zero restarts; next health and exact image/pointer identities matched expectations. Both CI runners active with zero restarts and running jobs. No failed system or calternal user units. CI slice MemoryMax=32 GiB and agent slice MemoryMax=24 GiB, with oomd enabled. All fresh heavy work used one build slot and no more than three workers. No build was needed. Verification accounting: unchanged committed product/test source, with SHA-256 manifest attached. Explicit passing injection positions retained from the two interrupted runs were combined with freshly executed missing positions using the original injection function and matrix. Every baseline was rerun and boundary counts checked. This is a completed combined sweep, not an uninterrupted full rerun. Other gates and all 25 real rehearsal rows ran fresh on this boot. Evidence is untracked; no artifacts were committed. Real staging rehearsal (fresh this boot): | Run | Result | | --- | --- | | real staging normal deploy | PASS | | real staging same SHA, same MainPID | PASS | | real staging rollback | PASS | | bootstrap full filesystem admission | PASS | | bootstrap full filesystem after admission | PASS | | bootstrap candidate failure recovers unlabelled image | PASS | | bootstrap from unlabelled running image | PASS | | rollback to recorded legacy image | PASS | | normal deploy after bootstrap removes legacy exception | PASS | | same SHA after bootstrap | PASS | | normal rollback after bootstrap | PASS | | final staging state | PASS | | real staging deploy | PASS | | real staging same SHA | PASS | | real staging rollback | PASS | | real full receipt filesystem cleans partial upload | PASS | | real staging invalid generator input | PASS | | real staging failed candidate recovery | PASS | | isolated store paths verified | PASS | | isolated real Podman deploy and rollback | PASS | | persistent full image store through recovery | PASS | | free-space margin refuses next deploy | PASS | | final real staging check | PASS | | real generator output loss and static fallback | PASS | | real SSH disconnect mid-switch | PASS | Rehearsals restored original staging unit bytes, image pointers and mount identities. Full disk cases used bounded dedicated filesystems; shared filesystems were never filled. Disposable fixtures were cleaned. [Complete evidence, retained output, resume source, source hashes and per-run logs](https://git.kayg.org/attachments/41c05e95-7549-44d8-9b76-889b74a237c4). Final health output, verbatim: ```text NRestarts=0 Id=forgejo-runner.service ActiveState=active SubState=running NRestarts=0 Id=forgejo-runner-web.service ActiveState=active SubState=running MemoryHigh=30064771072 MemoryMax=34359738368 ManagedOOMMemoryPressure=kill Id=ci.slice MemoryHigh=21474836480 MemoryMax=25769803776 ManagedOOMMemoryPressure=kill Id=user-1002.slice active NRestarts=0 Id=calternal-next.service ActiveState=active SubState=running NRestarts=0 Id=calternal-preview.service ActiveState=active SubState=running calternal-next running=true image=60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb UNIT LOAD ACTIVE SUB DESCRIPTION 0 loaded units listed. UNIT LOAD ACTIVE SUB DESCRIPTION 0 loaded units listed. image record: sha256:60f680ae778350c332b127e477c3db3590927d4040007f0d9446c4d5d07c57eb sha256:26ec8a7decf4c161acbabf8c99ebb2ba868075dcd4697fe08df356295aab8108 disposable leftovers: [] {"revision":"8dc52337a566158e86b0234bcfb082331d76d1cd","status":"ok","version":"0.1.0"} ``` Gate output, verbatim: ```text bash -n: PASS shellcheck: PASS diff whitespace: PASS real Quadlet generation (main and immutable recovery): PASS real Quadlet invalid candidate rejected before install: PASS target receiver fails before producer open; no retained artifact: PASS target caller lost before FIFO open cleans upload and releases lock: PASS target partial receipt failure cleans private upload: PASS target normal: PASS target failed-transaction: PASS target closed-output: PASS target dropped-caller: PASS target locked-receipt: PASS remote predecessor-runtime-error: PASS remote predecessor-inspection-error: PASS remote malformed-pointer: PASS remote multiline-pointer: PASS remote foreign-legacy-pointer: PASS remote equal-pointers: PASS remote symlink-pointer: PASS remote invalid-generator: PASS remote closed-output: PASS remote generator-fails: PASS remote byte-exact-recovery: PASS remote normal: PASS remote stale-recovery-definition: PASS remote stale-primary-recovery: PASS remote post-start-unit-rename: PASS remote retained-recovery: PASS remote same-revision different digest: PASS remote disk-full: PASS remote tag-fails: PASS remote pointer-commit-fails: PASS remote start-fails: PASS remote health-fails: PASS remote stale-response: PASS remote inactive-systemd: PASS remote wrong-container-image: PASS remote stop-fails: PASS remote reload-fails: PASS remote unit-install-fails: PASS remote unit-move-fails: PASS remote load-fails: PASS remote checksum-mismatch: PASS remote label-mismatch: PASS remote id-mismatch: PASS remote pre-unhealthy: PASS remote open-secret: PASS remote missing-secret: PASS remote symlink-secret: PASS remote unlabelled-baseline: PASS remote persistent-daemon-reload: PASS remote persistent-unit-rename: PASS remote stop-returns-error-after-stopping: PASS remote post-stop-podman-error: PASS remote recovery-start-fails: PASS remote spaced-open-secret: PASS remote spaced-new-open-secret: PASS remote multiple-open-secret: PASS remote optional-open-secret: PASS remote service-open-secret: PASS remote continued-open-secret: PASS remote quoted-open-secret: PASS remote unsupported-specifier: PASS remote unsupported-escape: PASS remote repeated-open-secret: PASS remote spaced-safe-secret: PASS remote multiple-safe-secret: PASS remote optional-safe-secret: PASS remote service-safe-secret: PASS remote continued-safe-secret: PASS remote quoted-safe-secret: PASS remote home-safe-secret: PASS remote percent-safe-secret: PASS remote signal: PASS remote rerun: PASS remote manual-rollback: PASS remote concurrent upload/switch serialized: PASS bootstrap legacy rollback verifies exact digest and status: PASS failed first update preserves baseline; absent rollback refuses before stop: PASS predecessor existence/inspection errors preserve older rollback through failed health: PASS injection deploy/normal/before: retained 92/92 explicit PASS positions from unchanged source injection deploy/normal/before: 92/92 accounted PASS injection deploy/normal/after: retained 92/92 explicit PASS positions from unchanged source injection deploy/normal/after: 92/92 accounted PASS persistent disk deploy/normal/before: retained 92/92 explicit PASS positions from unchanged source persistent disk deploy/normal/before: 92/92 accounted PASS persistent disk deploy/normal/after: retained 92/92 explicit PASS positions from unchanged source persistent disk deploy/normal/after: 92/92 accounted PASS injection same-sha/normal/before: retained 34/34 explicit PASS positions from unchanged source injection same-sha/normal/before: 34/34 accounted PASS injection same-sha/normal/after: retained 34/34 explicit PASS positions from unchanged source injection same-sha/normal/after: 34/34 accounted PASS injection same-image/normal/before: retained 34/34 explicit PASS positions from unchanged source injection same-image/normal/before: 34/34 accounted PASS injection same-image/normal/after: retained 34/34 explicit PASS positions from unchanged source injection same-image/normal/after: 34/34 accounted PASS injection rollback/normal/before: retained 79/79 explicit PASS positions from unchanged source injection rollback/normal/before: 79/79 accounted PASS injection rollback/normal/after: retained 79/79 explicit PASS positions from unchanged source injection rollback/normal/after: 79/79 accounted PASS persistent disk rollback/normal/before: retained 79/79 explicit PASS positions from unchanged source persistent disk rollback/normal/before: 79/79 accounted PASS persistent disk rollback/normal/after: retained 79/79 explicit PASS positions from unchanged source persistent disk rollback/normal/after: 79/79 accounted PASS injection first-update/normal/before: retained 91/91 explicit PASS positions from unchanged source injection first-update/normal/before: 91/91 accounted PASS injection first-update/normal/after: retained 91/91 explicit PASS positions from unchanged source injection first-update/normal/after: 91/91 accounted PASS persistent disk first-update/normal/before: retained 91/91 explicit PASS positions from unchanged source persistent disk first-update/normal/before: 91/91 accounted PASS persistent disk first-update/normal/after: retained 91/91 explicit PASS positions from unchanged source persistent disk first-update/normal/after: 91/91 accounted PASS injection bootstrap-upgrade/normal/before: retained 91/91 explicit PASS positions from unchanged source injection bootstrap-upgrade/normal/before: 91/91 accounted PASS injection bootstrap-upgrade/normal/after: retained 91/91 explicit PASS positions from unchanged source injection bootstrap-upgrade/normal/after: 91/91 accounted PASS persistent disk bootstrap-upgrade/normal/before: retained 91/91 explicit PASS positions from unchanged source persistent disk bootstrap-upgrade/normal/before: 91/91 accounted PASS persistent disk bootstrap-upgrade/normal/after: retained 91/91 explicit PASS positions from unchanged source persistent disk bootstrap-upgrade/normal/after: 91/91 accounted PASS injection bootstrap-health-fails/normal/before: retained 106/106 explicit PASS positions from unchanged source injection bootstrap-health-fails/normal/before: 106/106 accounted PASS injection bootstrap-health-fails/normal/after: retained 106/106 explicit PASS positions from unchanged source injection bootstrap-health-fails/normal/after: 106/106 accounted PASS persistent disk bootstrap-health-fails/normal/before: retained 106/106 explicit PASS positions from unchanged source persistent disk bootstrap-health-fails/normal/before: 106/106 accounted PASS persistent disk bootstrap-health-fails/normal/after: retained 106/106 explicit PASS positions from unchanged source persistent disk bootstrap-health-fails/normal/after: 106/106 accounted PASS injection bootstrap/normal/before: retained 8/8 explicit PASS positions from unchanged source injection bootstrap/normal/before: 8/8 accounted PASS injection bootstrap/normal/after: retained 8/8 explicit PASS positions from unchanged source injection bootstrap/normal/after: 8/8 accounted PASS injection bootstrap-missing-unit/normal/before: retained 5/5 explicit PASS positions from unchanged source injection bootstrap-missing-unit/normal/before: 5/5 accounted PASS injection bootstrap-missing-unit/normal/after: retained 5/5 explicit PASS positions from unchanged source injection bootstrap-missing-unit/normal/after: 5/5 accounted PASS injection deploy/health-fails/before: retained 107/107 explicit PASS positions from unchanged source injection deploy/health-fails/before: 107/107 accounted PASS injection deploy/health-fails/after: retained 107/107 explicit PASS positions from unchanged source injection deploy/health-fails/after: 107/107 accounted PASS persistent disk deploy/health-fails/before: retained 107/107 explicit PASS positions from unchanged source persistent disk deploy/health-fails/before: 107/107 accounted PASS persistent disk deploy/health-fails/after: retained 107/107 explicit PASS positions from unchanged source persistent disk deploy/health-fails/after: 107/107 accounted PASS injection rollback/health-fails/before: retained 94/94 explicit PASS positions from unchanged source injection rollback/health-fails/before: 94/94 accounted PASS injection rollback/health-fails/after: retained 94/94 explicit PASS positions from unchanged source injection rollback/health-fails/after: 94/94 accounted PASS persistent disk rollback/health-fails/before: retained 94/94 explicit PASS positions from unchanged source persistent disk rollback/health-fails/before: 94/94 accounted PASS persistent disk rollback/health-fails/after: retained 94/94 explicit PASS positions from unchanged source persistent disk rollback/health-fails/after: 94/94 accounted PASS injection deploy/stale-primary-recovery/before: retained 105/105 explicit PASS positions from unchanged source injection deploy/stale-primary-recovery/before: 105/105 accounted PASS injection deploy/stale-primary-recovery/after: retained 105/105 explicit PASS positions from unchanged source injection deploy/stale-primary-recovery/after: 105/105 accounted PASS persistent disk deploy/stale-primary-recovery/before: retained 105/105 explicit PASS positions from unchanged source persistent disk deploy/stale-primary-recovery/before: 105/105 accounted PASS persistent disk deploy/stale-primary-recovery/after: retained 105/105 explicit PASS positions from unchanged source persistent disk deploy/stale-primary-recovery/after: 105/105 accounted PASS injection rollback/stale-primary-recovery/before: retained 92/92 explicit PASS positions from unchanged source injection rollback/stale-primary-recovery/before: 92/92 accounted PASS injection rollback/stale-primary-recovery/after: retained 0/92 explicit PASS positions from unchanged source injection rollback/stale-primary-recovery/after: resumed 40/92 PASS injection rollback/stale-primary-recovery/after: resumed 80/92 PASS injection rollback/stale-primary-recovery/after: 92/92 accounted PASS persistent disk rollback/stale-primary-recovery/before: retained 0/92 explicit PASS positions from unchanged source persistent disk rollback/stale-primary-recovery/before: resumed 40/92 PASS persistent disk rollback/stale-primary-recovery/before: resumed 80/92 PASS persistent disk rollback/stale-primary-recovery/before: 92/92 accounted PASS persistent disk rollback/stale-primary-recovery/after: retained 0/92 explicit PASS positions from unchanged source persistent disk rollback/stale-primary-recovery/after: resumed 40/92 PASS persistent disk rollback/stale-primary-recovery/after: resumed 80/92 PASS persistent disk rollback/stale-primary-recovery/after: 92/92 accounted PASS injection deploy/post-start-unit-rename/before: retained 0/102 explicit PASS positions from unchanged source injection deploy/post-start-unit-rename/before: resumed 40/102 PASS injection deploy/post-start-unit-rename/before: resumed 80/102 PASS injection deploy/post-start-unit-rename/before: 102/102 accounted PASS injection deploy/post-start-unit-rename/after: retained 0/102 explicit PASS positions from unchanged source injection deploy/post-start-unit-rename/after: resumed 40/102 PASS injection deploy/post-start-unit-rename/after: resumed 80/102 PASS injection deploy/post-start-unit-rename/after: 102/102 accounted PASS persistent disk deploy/post-start-unit-rename/before: retained 0/102 explicit PASS positions from unchanged source persistent disk deploy/post-start-unit-rename/before: resumed 40/102 PASS persistent disk deploy/post-start-unit-rename/before: resumed 80/102 PASS persistent disk deploy/post-start-unit-rename/before: 102/102 accounted PASS persistent disk deploy/post-start-unit-rename/after: retained 0/102 explicit PASS positions from unchanged source persistent disk deploy/post-start-unit-rename/after: resumed 40/102 PASS persistent disk deploy/post-start-unit-rename/after: resumed 80/102 PASS persistent disk deploy/post-start-unit-rename/after: 102/102 accounted PASS injection rollback/post-start-unit-rename/before: retained 0/89 explicit PASS positions from unchanged source injection rollback/post-start-unit-rename/before: resumed 40/89 PASS injection rollback/post-start-unit-rename/before: resumed 80/89 PASS injection rollback/post-start-unit-rename/before: 89/89 accounted PASS injection rollback/post-start-unit-rename/after: retained 0/89 explicit PASS positions from unchanged source injection rollback/post-start-unit-rename/after: resumed 40/89 PASS injection rollback/post-start-unit-rename/after: resumed 80/89 PASS injection rollback/post-start-unit-rename/after: 89/89 accounted PASS persistent disk rollback/post-start-unit-rename/before: retained 0/89 explicit PASS positions from unchanged source persistent disk rollback/post-start-unit-rename/before: resumed 40/89 PASS persistent disk rollback/post-start-unit-rename/before: resumed 80/89 PASS persistent disk rollback/post-start-unit-rename/before: 89/89 accounted PASS persistent disk rollback/post-start-unit-rename/after: retained 0/89 explicit PASS positions from unchanged source persistent disk rollback/post-start-unit-rename/after: resumed 40/89 PASS persistent disk rollback/post-start-unit-rename/after: resumed 80/89 PASS persistent disk rollback/post-start-unit-rename/after: 89/89 accounted PASS exhaustive failure injection resumed: PASS (4354 cases = 3314 retained + 1040 fresh; userdata unchanged) relay success: PASS relay save-fails: PASS relay target-fails: PASS relay bootstrap: PASS non-dev revision and mutable tags rejected for next/cloud: PASS data rollback options rejected: PASS build normal: PASS build dev-moved: PASS build build-fails: PASS build bad-id: PASS build wrong-label: PASS build save-fails: PASS build export-wrong-label: PASS build multiple-images: PASS build root-owned-iid: PASS deploy and code rollback dry-run without transport (next/cloud): PASS deploy regression gates: PASS (completed with retained injection positions; local mocks; no production contact) target receiver fails before producer open; no retained artifact: PASS (real user systemd) target caller lost before FIFO open cleans upload and releases lock: PASS (real user systemd) target normal: PASS (real user systemd) target failed-transaction: PASS (real user systemd) target closed-output: PASS (real user systemd) target dropped-caller: PASS (real user systemd) target locked-receipt: PASS (real user systemd) ``` Gate exit: 0. Real bootstrap/staging rehearsal exit: 0. Additional real regression and SSH-disconnect rehearsal exit: 0. Remote branch and PR head both verified as `768dbacbb6b25773af560b5e10f69a9fbd88f1da`.
kayg commented 2026-11-04 13:04:28 +00:00
Author
Owner
Copy link

Closing: superseded by #733 (named in the PR body or comments). Branch kept.

Closing: superseded by #733 (named in the PR body or comments). Branch kept.
kayg closed this pull request 2026-11-04 13:04:28 +00:00
Some checks failed
CI / web (pull_request) Failing after 52s
Details
CI / server (pull_request) Successful in 1m15s
Details
CI / rust (pull_request) Has been cancelled
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!688
No description provided.