kayg/calternal
Watch
1
0
Fork
You've already forked calternal
 0
Code Issues 191 Pull requests 61 Projects Releases Packages Wiki Activity Actions 20

ops: targetable deploy script with staging proof (supersedes #773) #786

Closed
kayg wants to merge 0 commits from ops/deploy-simple-765-r2 into dev
pull from: ops/deploy-simple-765-r2
merge into: kayg:dev
Conversation 2 Commits - Files changed -
kayg commented 2026-10-31 05:54:10 +00:00
Owner
Copy link

Supersedes #773 (same simple Bash deploy script, now targetable and proven on staging). Closes #765.

What changed since #773 (2 files of script/tests, 1 doc)

  • One direct hop, no jump host: netbird ssh --no-browser $DEPLOY_TARGET. Default root@calternal-cloud.vpn.kayg.org.
  • Overrides, all defaulting to prod: DEPLOY_TARGET, DEPLOY_ROOT (/srv/calternal-app), DEPLOY_USER (calternal), DEPLOY_HEALTH_URL (quadlet PublishPort), DEPLOY_SSH (ssh command prefix, used by tests). DEPLOY_HOP is gone.
  • The app user's UID is read on the target (id -u), no more /run/user/1001.
  • --dry-run prints target, root, user, health URL and the steps; connects to nothing.
  • With the prod target the script prints it and wants the hostname typed; --yes-prod skips it. No terminal and no flag: refuses.
  • Shim tests now use a fake target/user/uid/root/health URL and fail if any of them is wrong, and grep the script so that calternal-cloud, 10.70.3.162, /srv/calternal-app appear only once each (the defaults), and 10.69.69.83, 8by64, DEPLOY_HOP, /run/user/<n> not at all. 18 tests, script is 148 lines.
  • Two bugs found by the staging run and fixed here (they were real, and the shims could not see them):
    1. Backup missed non-rebuildable state. #773 saved only users/*/index.db*. docs/storage.md says users/*/notes/history.db (edits, versions; not rebuildable), meta.db, attachments/index.db, search/ and .calternal-install-id are state too. Now included. Photos/Documents/Notes still excluded.
    2. Restart can fail on a busy port. After the old container stops, its published port stays busy for a while (TIME_WAIT) and pasta cannot bind; systemd then gives up after 5 quick tries ("Start request repeated too quickly"). This made both the new start and the rollback start fail on the first staging run. The script now does reset-failed + restart, retried for about 1 minute (3 lines).
      Also quieter: tar socket warning (admin.sock) and the per-second curl error lines.

Staging proof

Target: calternal-dev-01-compute02 (Debian 13, x86_64). Setup: user calternal (uid 30034, linger), rootless Nix podman 5.8.9 (+ passt, crun, conmon, and apt uidmap; quadlet generator linked from the Nix podman), /srv/calternal-app/data in the prod layout with generated data (3 users, 9 notes, 1 attachment), real images built from deploy/Containerfile at origin/dev d83e54db (:current, quadlet like prod: same unit, PublishPort changed to 127.0.0.1:8080, UserNS=keep-id:uid=10001). Deployed train/2026-10-30-bm3 cc56ef86 from the control host. "Version" is 0.1.0 on both commits, so "new image serving" is shown by image ID. Every run had DEPLOY_TARGET=root@calternal-dev-01-compute02.vpn.kayg.org; prod was never contacted (dry-run output has no prod value; checked with grep).

# Case Result Evidence
1 Normal deploy PASS Container image went cb3d084a7975 (d83) to 5b468d456d0f (cc56), health healthy, /healthz 200 0.1.0 verified by script. User-content checksum (10 files of Notes/Documents) efff01ef2eee3c1f before and after; invite.key unchanged. Backup backup-pre-cc56ef86-*.tgz (48 KB, 33 state files). Restored it over a copy of the data with state removed: old image started on it, /healthz 200 and notes.list returned the notes.
2 Broken image (entrypoint sleep infinity) PASS Script waited 60 s, printed "Deploy failed; rolling back image", "Rollback healthy; previous version is running", exit 1. Container back on 5b468d456d0f, healthy, checksum unchanged, backup kept.
3 Manual rollback to :rollback PASS After deploy 1: :rollback = d83. podman tag :rollback :current; systemctl --user -M calternal@ restart calternal.service gave container image cb3d084a7975 (d83), healthy, checksum unchanged.
4 Disk-space refusal PASS DEPLOY_ROOT=/mnt/small (40 MB loop fs, 28 MB free, with a copy of data): "Insufficient disk at /mnt/small", exit 1, before stop/backup/load; real service stayed up on the same image.
5 Immediate second deploy PASS Two deploys of cc56ef86 back to back: both exit 0, both verified, two backups, healthy, checksum unchanged. Known limit: :rollback now equals :current (d83 still reachable by its SHA tag). Documented, not fixed.
6 stdin protection PASS, weak evidence Real chain (ssh to sudo to podman load) received the whole 45 MB archive intact: loaded image ID equals the source. But a negative control (every </dev/null removed) also succeeded on this stack: nothing on dev-01 consumed stdin, so the guard is proven only by the shim tests (a shim that tries to read stdin). Kept as cheap insurance.

Also seen: first run (before the two fixes) failed on the busy port, and the rollback failed the same way ("ROLLBACK FAILED; manual recovery required", exit 1), and the service had to be started by hand. That is the run that found bug 2.

Finding outside this PR

train/2026-10-30-bm3 (cc56ef86) cannot build with deploy/Containerfile: bun install --frozen-lockfile fails because packages/text is a workspace but the Containerfile copies only editor/ui/web. I built it with a patched copy (two extra COPY packages/text ... lines, kept outside the repo). The Containerfile needs the same two lines before that train ships.

Exact commands

# control host, from this branch. Never without DEPLOY_TARGET for staging.
export DEPLOY_TARGET=root@calternal-dev-01-compute02.vpn.kayg.org DEPLOY_HEALTH_URL=http://127.0.0.1:8080/healthz
scripts/deploy-prod.sh --dry-run cc56ef86.tar cc56ef86 0.1.0
scripts/deploy-prod.sh cc56ef86.tar cc56ef86 0.1.0
DEPLOY_ROOT=/mnt/small scripts/deploy-prod.sh cc56ef86.tar cc56ef86 0.1.0   # case 4
# images: on dev-01 as root, via slot (claude has no docker):
#   HOME=/home/claude /home/claude/bin/slot docker build -f deploy/Containerfile -t localhost/calternal:<sha> .   (in worktrees ~claude/Developer/Worktrees/stg-<sha>)
#   docker save localhost/calternal:<sha> > <sha>.tar     # uncompressed; copied to the control host
# bad image: FROM localhost/calternal:cc56ef86 / ENTRYPOINT ["sleep","infinity"]   (tag badc0de1)
bash tests/deploy-prod.sh        # run on dev-01: all 18 PASS

Reset staging

On dev-01 as root: bash /root/stg/reset.sh stops the service, restores /srv/calternal-app/data from /root/stg/golden-data.tgz, removes backups, reloads d83e54db as :current (no :rollback), starts the quadlet and prints the state (/root/stg/state.sh). bash /root/stg/seed.sh recreates the golden data from scratch (new users/notes). Images for the runs: /root/stg/{cc56ef86,badc0de1}.tar. Staging is left running d83e54db.

Self-review

Round 1 only (the staging run did the finding); no new problems after the two fixes, so no simpler redesign needed. Open risks: prod's podman/network stack may differ from staging (pasta here), so the busy-port retry is defensive; :rollback is clobbered by a same-SHA redeploy; stdin guard has no real-stack proof.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xi6YxsuZ7m2ESGC7aS91AN

Supersedes #773 (same simple Bash deploy script, now targetable and proven on staging). Closes #765. ## What changed since #773 (2 files of script/tests, 1 doc) - One direct hop, no jump host: `netbird ssh --no-browser $DEPLOY_TARGET`. Default `root@calternal-cloud.vpn.kayg.org`. - Overrides, all defaulting to prod: `DEPLOY_TARGET`, `DEPLOY_ROOT` (`/srv/calternal-app`), `DEPLOY_USER` (`calternal`), `DEPLOY_HEALTH_URL` (quadlet `PublishPort`), `DEPLOY_SSH` (ssh command prefix, used by tests). `DEPLOY_HOP` is gone. - The app user's UID is read on the target (`id -u`), no more `/run/user/1001`. - `--dry-run` prints target, root, user, health URL and the steps; connects to nothing. - With the prod target the script prints it and wants the hostname typed; `--yes-prod` skips it. No terminal and no flag: refuses. - Shim tests now use a fake target/user/uid/root/health URL and fail if any of them is wrong, and grep the script so that `calternal-cloud`, `10.70.3.162`, `/srv/calternal-app` appear only once each (the defaults), and `10.69.69.83`, `8by64`, `DEPLOY_HOP`, `/run/user/<n>` not at all. 18 tests, script is 148 lines. - Two bugs found by the staging run and fixed here (they were real, and the shims could not see them): 1. **Backup missed non-rebuildable state.** #773 saved only `users/*/index.db*`. `docs/storage.md` says `users/*/notes/history.db` (edits, versions; not rebuildable), `meta.db`, `attachments/index.db`, `search/` and `.calternal-install-id` are state too. Now included. Photos/Documents/Notes still excluded. 2. **Restart can fail on a busy port.** After the old container stops, its published port stays busy for a while (TIME_WAIT) and pasta cannot bind; systemd then gives up after 5 quick tries ("Start request repeated too quickly"). This made both the new start and the rollback start fail on the first staging run. The script now does `reset-failed` + restart, retried for about 1 minute (3 lines). Also quieter: `tar` socket warning (`admin.sock`) and the per-second curl error lines. ## Staging proof Target: `calternal-dev-01-compute02` (Debian 13, x86_64). Setup: user `calternal` (uid 30034, linger), rootless **Nix podman 5.8.9** (+ passt, crun, conmon, and apt `uidmap`; quadlet generator linked from the Nix podman), `/srv/calternal-app/data` in the prod layout with generated data (3 users, 9 notes, 1 attachment), real images built from `deploy/Containerfile` at `origin/dev` d83e54db (`:current`, quadlet like prod: same unit, `PublishPort` changed to `127.0.0.1:8080`, `UserNS=keep-id:uid=10001`). Deployed `train/2026-10-30-bm3` cc56ef86 from the control host. "Version" is `0.1.0` on both commits, so "new image serving" is shown by image ID. Every run had `DEPLOY_TARGET=root@calternal-dev-01-compute02.vpn.kayg.org`; prod was never contacted (dry-run output has no prod value; checked with grep). | # | Case | Result | Evidence | |---|------|--------|----------| | 1 | Normal deploy | PASS | Container image went `cb3d084a7975` (d83) to `5b468d456d0f` (cc56), health `healthy`, `/healthz` 200 `0.1.0` verified by script. User-content checksum (10 files of Notes/Documents) `efff01ef2eee3c1f` before and after; `invite.key` unchanged. Backup `backup-pre-cc56ef86-*.tgz` (48 KB, 33 state files). Restored it over a copy of the data with state removed: old image started on it, `/healthz` 200 and `notes.list` returned the notes. | | 2 | Broken image (entrypoint `sleep infinity`) | PASS | Script waited 60 s, printed "Deploy failed; rolling back image", "Rollback healthy; previous version is running", exit 1. Container back on `5b468d456d0f`, healthy, checksum unchanged, backup kept. | | 3 | Manual rollback to `:rollback` | PASS | After deploy 1: `:rollback` = d83. `podman tag :rollback :current; systemctl --user -M calternal@ restart calternal.service` gave container image `cb3d084a7975` (d83), healthy, checksum unchanged. | | 4 | Disk-space refusal | PASS | `DEPLOY_ROOT=/mnt/small` (40 MB loop fs, 28 MB free, with a copy of data): "Insufficient disk at /mnt/small", exit 1, before stop/backup/load; real service stayed up on the same image. | | 5 | Immediate second deploy | PASS | Two deploys of cc56ef86 back to back: both exit 0, both verified, two backups, healthy, checksum unchanged. Known limit: `:rollback` now equals `:current` (d83 still reachable by its SHA tag). Documented, not fixed. | | 6 | stdin protection | PASS, weak evidence | Real chain (ssh to sudo to podman load) received the whole 45 MB archive intact: loaded image ID equals the source. But a negative control (every `</dev/null` removed) **also succeeded** on this stack: nothing on dev-01 consumed stdin, so the guard is proven only by the shim tests (a shim that tries to read stdin). Kept as cheap insurance. | Also seen: first run (before the two fixes) failed on the busy port, and the rollback failed the same way ("ROLLBACK FAILED; manual recovery required", exit 1), and the service had to be started by hand. That is the run that found bug 2. ### Finding outside this PR `train/2026-10-30-bm3` (cc56ef86) cannot build with `deploy/Containerfile`: `bun install --frozen-lockfile` fails because `packages/text` is a workspace but the Containerfile copies only editor/ui/web. I built it with a patched copy (two extra `COPY packages/text ...` lines, kept outside the repo). The Containerfile needs the same two lines before that train ships. ## Exact commands ```sh # control host, from this branch. Never without DEPLOY_TARGET for staging. export DEPLOY_TARGET=root@calternal-dev-01-compute02.vpn.kayg.org DEPLOY_HEALTH_URL=http://127.0.0.1:8080/healthz scripts/deploy-prod.sh --dry-run cc56ef86.tar cc56ef86 0.1.0 scripts/deploy-prod.sh cc56ef86.tar cc56ef86 0.1.0 DEPLOY_ROOT=/mnt/small scripts/deploy-prod.sh cc56ef86.tar cc56ef86 0.1.0 # case 4 # images: on dev-01 as root, via slot (claude has no docker): # HOME=/home/claude /home/claude/bin/slot docker build -f deploy/Containerfile -t localhost/calternal:<sha> . (in worktrees ~claude/Developer/Worktrees/stg-<sha>) # docker save localhost/calternal:<sha> > <sha>.tar # uncompressed; copied to the control host # bad image: FROM localhost/calternal:cc56ef86 / ENTRYPOINT ["sleep","infinity"] (tag badc0de1) bash tests/deploy-prod.sh # run on dev-01: all 18 PASS ``` ## Reset staging On dev-01 as root: `bash /root/stg/reset.sh` stops the service, restores `/srv/calternal-app/data` from `/root/stg/golden-data.tgz`, removes backups, reloads d83e54db as `:current` (no `:rollback`), starts the quadlet and prints the state (`/root/stg/state.sh`). `bash /root/stg/seed.sh` recreates the golden data from scratch (new users/notes). Images for the runs: `/root/stg/{cc56ef86,badc0de1}.tar`. Staging is left running d83e54db. ## Self-review Round 1 only (the staging run did the finding); no new problems after the two fixes, so no simpler redesign needed. Open risks: prod's podman/network stack may differ from staging (pasta here), so the busy-port retry is defensive; `:rollback` is clobbered by a same-SHA redeploy; stdin guard has no real-stack proof. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Xi6YxsuZ7m2ESGC7aS91AN
ops: add simple production deploy script (#765)
Some checks failed
CI / server (pull_request) Successful in 1m19s
Details
CI / rust (pull_request) Successful in 18m23s
Details
CI / web (pull_request) Has been cancelled
Details
1b9b5da78e 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xi6YxsuZ7m2ESGC7aS91AN
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xi6YxsuZ7m2ESGC7aS91AN
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xi6YxsuZ7m2ESGC7aS91AN
ops: quieter health probe, document manual rollback
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details
1820e80422 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xi6YxsuZ7m2ESGC7aS91AN
kayg commented 2026-10-31 05:54:53 +00:00
Author
Owner
Copy link

Merged by the orchestrator on kayg's D4 approval ("as long as it's proven to work"): staging proof 6/6 on dev-01. Note: in the single-root layout the backup also includes users' notes and attachments (safer, larger). Follow-ups: same-SHA redeploy overwrites :rollback; deploy/Containerfile needs packages/text copied before BM3 can build an image.

Merged by the orchestrator on kayg's D4 approval ("as long as it's proven to work"): staging proof 6/6 on dev-01. Note: in the single-root layout the backup also includes users' notes and attachments (safer, larger). Follow-ups: same-SHA redeploy overwrites `:rollback`; `deploy/Containerfile` needs `packages/text` copied before BM3 can build an image.
kayg commented 2026-11-04 13:04:41 +00:00
Author
Owner
Copy link

Closing: head 1820e804 is already in dev (reached via 2f304154); nothing left to merge. Branch kept.

Closing: head 1820e804 is already in dev (reached via 2f304154); nothing left to merge. Branch kept.
kayg closed this pull request 2026-11-04 13:04:41 +00:00
Some checks failed
CI / server (pull_request) Has been cancelled
Details
CI / rust (pull_request) Has been cancelled
Details
CI / web (pull_request) Has been cancelled
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backend
Rust server

  
design
Mockups, UI and TASTE

  
docs
ADRs, CONTEXT, TASTE

  
idea
Parked idea, never worked until promoted

  
infra
Deploy, CI, VMs

  
perf
Performance and budgets

  
reliability
Data integrity, corruption traps

  
web
Web client

No labels
backend
design
docs
idea
infra
perf
reliability
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal!786
No description provided.