kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 191 Pull requests 61 Projects Releases Packages Wiki Activity Actions 20

Simple prod deploy script (#765) #773

Closed
kayg wants to merge 0 commits from ops/deploy-simple-765 into dev
pull from: ops/deploy-simple-765
merge into: kayg:dev
Conversation 4 Commits - Files changed -
kayg commented 2026-10-30 13:02:28 +00:00
Owner
Copy link

Adds scripts/deploy-prod.sh for an existing image tar: preflight SSH, disk and current health; stop and archive data; retain three backups; load with rootless Podman; swap tags and restart the quadlet; verify HTTP 200 and version within 60 seconds. Failures recover the previous image and check its version. Data restoration requires explicit --restore.

Replaces the reserve plan in #765 per kayg’s 2026-10-30 instruction to keep deployment simple. Includes a short operator guide and local shim tests. No reserves, daemons, timers or deploy state files.

Validation: bash tests/deploy-prod.sh passes eight scenarios, covering success/retention, health rollback, explicit restore, version mismatch, image-load failure, backup failure, low-disk refusal and dry run. Bash syntax and staged whitespace checks pass. No Rust builds or production connections were made.

Adds `scripts/deploy-prod.sh` for an existing image tar: preflight SSH, disk and current health; stop and archive data; retain three backups; load with rootless Podman; swap tags and restart the quadlet; verify HTTP 200 and version within 60 seconds. Failures recover the previous image and check its version. Data restoration requires explicit `--restore`. Replaces the reserve plan in #765 per kayg’s 2026-10-30 instruction to keep deployment simple. Includes a short operator guide and local shim tests. No reserves, daemons, timers or deploy state files. Validation: `bash tests/deploy-prod.sh` passes eight scenarios, covering success/retention, health rollback, explicit restore, version mismatch, image-load failure, backup failure, low-disk refusal and dry run. Bash syntax and staged whitespace checks pass. No Rust builds or production connections were made.
ops: add simple production deploy script (#765)
Some checks failed
CI / server (pull_request) Successful in 1m19s
Details
CI / rust (pull_request) Successful in 18m23s
Details
CI / web (pull_request) Has been cancelled
Details
1b9b5da78e
kayg commented 2026-10-30 13:12:24 +00:00
Author
Owner
Copy link

Updated in 69d0266d:

  • Stopped backup contains only appdata/ when present; before the storage split it selects data/system/, data/notes/, data/search/, data/attachments/, and data/users/*/index.db*. Free-space sizing uses those same state paths. User photos/documents are excluded; docs explain the nightly user-file backup on backup01.
  • --restore replaces state only, preserving user files and clearing new SQLite sidecars from the failed version.
  • Both health probes share one commented health_url variable matching the quadlet's published host address (the current quadlet binds the LAN address).
  • Earlier sudo/Podman/curl calls receive /dev/null on stdin; podman load receives the full image stream. The shim actively attempts to steal stdin and checks the loaded bytes.
  • Kept three backups, :rollback, opt-in --restore, and --dry-run; deploy script is 130 lines. No infrastructure added.

Validation on 6by32-compute01 (local shims only; no prod contact, no Rust builds):

$ bash tests/deploy-prod.sh
PASS single-root backs up only state with disk budget excluding user files
PASS health probes use the quadlet published address
PASS earlier sudo, podman and curl cannot consume image stdin
PASS failed health rolls back without restoring data
PASS explicit restore recovers state and preserves user files
PASS split layout backs up appdata only with disk budget excluding userdata
PASS split restore recovers appdata and preserves userdata
PASS wrong version rolls back
PASS image load failure recovers old service
PASS backup failure restarts old service without loading
PASS insufficient disk refuses before stopping
PASS disk budget includes all selected state
PASS dry run makes no remote calls
PASS deploy script stays within 150 lines
$ bash -n scripts/deploy-prod.sh tests/deploy-prod.sh
exit 0
$ git diff --check
exit 0
$ wc -l scripts/deploy-prod.sh
130 scripts/deploy-prod.sh

Also verified that removing the stdin guard or changing the health binding makes the regression suite fail. Pushed only ops/deploy-simple-765; no merge.

Updated in `69d0266d`: - Stopped backup contains only `appdata/` when present; before the storage split it selects `data/system/`, `data/notes/`, `data/search/`, `data/attachments/`, and `data/users/*/index.db*`. Free-space sizing uses those same state paths. User photos/documents are excluded; docs explain the nightly user-file backup on backup01. - `--restore` replaces state only, preserving user files and clearing new SQLite sidecars from the failed version. - Both health probes share one commented `health_url` variable matching the quadlet's published host address (the current quadlet binds the LAN address). - Earlier sudo/Podman/curl calls receive `/dev/null` on stdin; `podman load` receives the full image stream. The shim actively attempts to steal stdin and checks the loaded bytes. - Kept three backups, `:rollback`, opt-in `--restore`, and `--dry-run`; deploy script is 130 lines. No infrastructure added. Validation on 6by32-compute01 (local shims only; no prod contact, no Rust builds): ```text $ bash tests/deploy-prod.sh PASS single-root backs up only state with disk budget excluding user files PASS health probes use the quadlet published address PASS earlier sudo, podman and curl cannot consume image stdin PASS failed health rolls back without restoring data PASS explicit restore recovers state and preserves user files PASS split layout backs up appdata only with disk budget excluding userdata PASS split restore recovers appdata and preserves userdata PASS wrong version rolls back PASS image load failure recovers old service PASS backup failure restarts old service without loading PASS insufficient disk refuses before stopping PASS disk budget includes all selected state PASS dry run makes no remote calls PASS deploy script stays within 150 lines $ bash -n scripts/deploy-prod.sh tests/deploy-prod.sh exit 0 $ git diff --check exit 0 $ wc -l scripts/deploy-prod.sh 130 scripts/deploy-prod.sh ``` Also verified that removing the stdin guard or changing the health binding makes the regression suite fail. Pushed only `ops/deploy-simple-765`; no merge.
kayg commented 2026-10-31 05:23:34 +00:00
Author
Owner
Copy link

Staging proof

Result: BLOCKED, not ready to merge. No staging run was done, and nothing was touched on prod or dev-01. (Reviewed @ 69d0266d.)

The brief said to stop if the script hardcodes prod. It does, in three places. The only override is DEPLOY_HOP, and it changes only the outer hop.

# Case Result Evidence
1 Normal deploy (backup restorable, new image, health OK, checksums) NOT RUN Blocked by hardcoded prod target (below)
2 Failed health check (broken image) NOT RUN Same
3 Manual rollback to :rollback NOT RUN Same
4 Disk-space refusal NOT RUN Same
5 Second deploy right after the first NOT RUN Same
6 stdin protection NOT RUN Same

Blocking finding: prod is hardcoded

In scripts/deploy-prod.sh:

  1. Last lines: printf -v inner 'netbird ssh --no-browser root@10.69.69.83 bash -c %q' ... has a fixed inner hop to prod root. DEPLOY_HOP only replaces the outer netbird ssh agent@8by64-compute01... hop. If DEPLOY_HOP were pointed at staging, the outer hop would land on the staging host and the inner hop would then netbird ssh root@10.69.69.83 from it. If staging can reach that address, that is a real prod deploy. I did not run it for that reason.
  2. health_url=http://10.70.3.162:8080/healthz is a literal prod address, with no override.
  3. root=/srv/calternal-app, sudo -u calternal, and XDG_RUNTIME_DIR=/run/user/1001 are literals. A fresh calternal user on another host will likely not have UID 1001. The shim tests rewrite these paths, so the tests cannot catch this.

Because of (1) and (2), the script cannot be proven on a staging host without editing it. The tests that pass only exercise shims that rewrite the prod path (tests/deploy-prod.sh substitutes /srv/calternal-app), so they do not show the real hop or the real podman/systemd calls work.

Also noted: the PR body says "eight scenarios" and describes full-data handling, while the brief mentions 14 tests, a "shared local health URL" and "matching disk sizing". The code at this SHA does not match that description (health URL is a literal prod IP, not shared with the quadlet).

Suggested fix before staging proof (not done; I stopped per instructions)

  • Add env overrides with prod as default, e.g. DEPLOY_TARGET (inner hop, default root@10.69.69.83), DEPLOY_ROOT, DEPLOY_USER/runtime dir derived with id -u calternal on the remote, and DEPLOY_HEALTH_URL (ideally read from the quadlet PublishPort).
  • Make --dry-run print the resolved hop and URL so the target is visible before running.
  • Then rerun the 6 cases on calternal-dev-01-compute02 (user calternal with linger, Nix podman, /srv/calternal-app/data, :current from origin/dev d83e54db, deploy train/2026-10-30-bm3 cc56ef86).

Staging setup on dev-01: not created yet (nothing to reset).

## Staging proof **Result: BLOCKED, not ready to merge. No staging run was done, and nothing was touched on prod or dev-01.** (Reviewed @ 69d0266d.) The brief said to stop if the script hardcodes prod. It does, in three places. The only override is `DEPLOY_HOP`, and it changes only the outer hop. | # | Case | Result | Evidence | |---|------|--------|----------| | 1 | Normal deploy (backup restorable, new image, health OK, checksums) | NOT RUN | Blocked by hardcoded prod target (below) | | 2 | Failed health check (broken image) | NOT RUN | Same | | 3 | Manual rollback to `:rollback` | NOT RUN | Same | | 4 | Disk-space refusal | NOT RUN | Same | | 5 | Second deploy right after the first | NOT RUN | Same | | 6 | stdin protection | NOT RUN | Same | ### Blocking finding: prod is hardcoded In `scripts/deploy-prod.sh`: 1. Last lines: `printf -v inner 'netbird ssh --no-browser root@10.69.69.83 bash -c %q' ...` has a fixed inner hop to prod root. `DEPLOY_HOP` only replaces the outer `netbird ssh agent@8by64-compute01...` hop. If `DEPLOY_HOP` were pointed at staging, the outer hop would land on the staging host and the inner hop would then `netbird ssh root@10.69.69.83` from it. If staging can reach that address, that is a **real prod deploy**. I did not run it for that reason. 2. `health_url=http://10.70.3.162:8080/healthz` is a literal prod address, with no override. 3. `root=/srv/calternal-app`, `sudo -u calternal`, and `XDG_RUNTIME_DIR=/run/user/1001` are literals. A fresh `calternal` user on another host will likely not have UID 1001. The shim tests rewrite these paths, so the tests cannot catch this. Because of (1) and (2), the script cannot be proven on a staging host without editing it. The tests that pass only exercise shims that rewrite the prod path (`tests/deploy-prod.sh` substitutes `/srv/calternal-app`), so they do not show the real hop or the real podman/systemd calls work. Also noted: the PR body says "eight scenarios" and describes full-`data` handling, while the brief mentions 14 tests, a "shared local health URL" and "matching disk sizing". The code at this SHA does not match that description (health URL is a literal prod IP, not shared with the quadlet). ### Suggested fix before staging proof (not done; I stopped per instructions) - Add env overrides with prod as default, e.g. `DEPLOY_TARGET` (inner hop, default `root@10.69.69.83`), `DEPLOY_ROOT`, `DEPLOY_USER`/runtime dir derived with `id -u calternal` on the remote, and `DEPLOY_HEALTH_URL` (ideally read from the quadlet `PublishPort`). - Make `--dry-run` print the resolved hop and URL so the target is visible before running. - Then rerun the 6 cases on calternal-dev-01-compute02 (user `calternal` with linger, Nix podman, `/srv/calternal-app/data`, `:current` from origin/dev d83e54db, deploy `train/2026-10-30-bm3` cc56ef86). Staging setup on dev-01: not created yet (nothing to reset).
kayg commented 2026-10-31 05:54:11 +00:00
Author
Owner
Copy link

Superseded by #786: same script, now with DEPLOY_TARGET/DEPLOY_ROOT/DEPLOY_USER/DEPLOY_HEALTH_URL, one direct hop, --dry-run target print, typed prod confirmation, and a 6-case staging proof on calternal-dev-01-compute02 (found and fixed two real bugs: incomplete state backup, busy-port restart). Please review there.

Superseded by #786: same script, now with `DEPLOY_TARGET`/`DEPLOY_ROOT`/`DEPLOY_USER`/`DEPLOY_HEALTH_URL`, one direct hop, `--dry-run` target print, typed prod confirmation, and a 6-case staging proof on calternal-dev-01-compute02 (found and fixed two real bugs: incomplete state backup, busy-port restart). Please review there.
kayg commented 2026-11-04 13:04:39 +00:00
Author
Owner
Copy link

Closing: head 69d0266d is already in dev (reached via 2f304154); nothing left to merge. Branch kept.

Closing: head 69d0266d is already in dev (reached via 2f304154); nothing left to merge. Branch kept.
kayg closed this pull request 2026-11-04 13:04:39 +00:00
Some checks failed
CI / server (pull_request) Successful in 1m19s
Details
CI / rust (pull_request) Successful in 18m23s
Details
CI / web (pull_request) Has been cancelled
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!773
No description provided.