kayg/calternal

Watch

Fork 0

DEPLOY: preserve a recoverable release and keep raw service logs out of job output #732

New issue

Open

opened 2026-10-02 13:06:52 +00:00 by kayg · 4 comments

kayg commented

2026-10-02 13:06:52 +00:00

Owner

Defensive sec-admin-deploy audit, assigned through #663.

Source: origin/dev c4a61e8cf0.
Also present in origin/job/merge-round-7a. Source review only; no production access or exploit execution.

A failed deploy has no rollback and copies raw service logs.
deploy/deploy-cloud.sh:36-49 replaces the mutable image tag, installs the
Quadlet, restarts, then only exits on health failure. It does not retain
and restore the previous image ID and Quadlet. It prints 60 raw journal
lines. crates/calternal-auth/src/api.rs:202 intentionally logs the
first-owner setup URL (DESIGN §7). Thus the diagnostic copy can carry a
setup credential to the invoking job's output. Do not copy raw service
logs. Print a fixed failure and a local diagnostic instruction. Keep
immutable release identities and rollback state. Check migration
compatibility before any binary rollback. Test failure paths with fake
service and image commands; do not deploy to test this change.

Fix acceptance: add the tests described above, run touched-crate gates, and verify no secrets appear in reports.

Defensive sec-admin-deploy audit, assigned through #663. Source: origin/dev c4a61e8cf090170f35b1bed3350d9de20c83ecd5. Also present in origin/job/merge-round-7a. Source review only; no production access or exploit execution. 3. **A failed deploy has no rollback and copies raw service logs.** `deploy/deploy-cloud.sh:36-49` replaces the mutable image tag, installs the Quadlet, restarts, then only exits on health failure. It does not retain and restore the previous image ID and Quadlet. It prints 60 raw journal lines. `crates/calternal-auth/src/api.rs:202` intentionally logs the first-owner setup URL (DESIGN §7). Thus the diagnostic copy can carry a setup credential to the invoking job's output. Do not copy raw service logs. Print a fixed failure and a local diagnostic instruction. Keep immutable release identities and rollback state. Check migration compatibility before any binary rollback. Test failure paths with fake service and image commands; do not deploy to test this change. Fix acceptance: add the tests described above, run touched-crate gates, and verify no secrets appear in reports.

kayg commented

2026-10-02 13:09:59 +00:00

Author

Owner

The same script also has no bounded image-retention step. It exports the mutable main tag only (line 37), so older VM images can lose their tag without being removed. Keep explicit immutable release IDs for rollback and remove only obsolete release images after successful validation; do not broadly prune Agent or rollback images. Disk growth is reasoned from source, not measured on production.

kayg referenced this issue

2026-10-02 13:11:51 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg referenced this issue

2026-10-02 13:12:42 +00:00

PERF: instant everywhere — app-wide speed architecture rules, audit and shared primitives #663

kayg commented

2026-10-02 13:16:08 +00:00

Author

Owner

Starting #732 on branch job/deployfix-732. Base: origin/dev c4a61e8cf0. Current head: 2f4482ded0.

Starting #732 on branch job/deployfix-732. Base: origin/dev c4a61e8cf090170f35b1bed3350d9de20c83ecd5. Current head: 2f4482ded066d9c5d9c59130377907f7fd2916c9.

kayg commented

2026-10-02 14:08:01 +00:00

Author

Owner

Finding: DESIGN §49 I6 says a binary rollback across an unrelated incompatible Plugin schema change requires the matching Index snapshot. The server snapshots the Index before migrations in crates/calternal-server/src/wire.rs. The deploy helper therefore rolls back only when migration SQL and Rust registration sources are unchanged from the saved release SHA. When the source check is unknown, it stops the unhealthy candidate and leaves the Index intact instead of starting an older binary. This is a fail-closed recovery path; a migration-changing release still needs a matching-snapshot recovery procedure if its health check fails.

Finding: DESIGN §49 I6 says a binary rollback across an unrelated incompatible Plugin schema change requires the matching Index snapshot. The server snapshots the Index before migrations in `crates/calternal-server/src/wire.rs`. The deploy helper therefore rolls back only when migration SQL and Rust registration sources are unchanged from the saved release SHA. When the source check is unknown, it stops the unhealthy candidate and leaves the Index intact instead of starting an older binary. This is a fail-closed recovery path; a migration-changing release still needs a matching-snapshot recovery procedure if its health check fails.

kayg commented

2026-10-02 14:24:24 +00:00

Author

Owner

#732 is implemented on job/deployfix-732.

Head: 376ed5afbd1bfed0ac3cb44472a3ae2b2c7319e2.

Built

Cloud images use the full Git SHA. Host state keeps the active release and two prior healthy releases with their matching Quadlets. Failed health checks restore the last healthy release when migration compatibility is established.
A healthy deploy removes releases beyond the retention set, removes dangling images without Podman's --all, and drops the legacy main alias.
Failures print a fixed class and the host journal instruction. The deploy helper never reads or forwards journal contents.
--dry-run prints the plan without a build or host connection.

Files: deploy/deploy-cloud.sh, deploy/cloud/deploy-release.sh, deploy/cloud/migration-compat.sh, deploy/cloud/README.md, tests/deploy/test_cloud_deploy.py.

Decisions and known gaps

Before the first SHA-tagged release, the old main image has no known Git SHA. The helper retains it by image ID and uses origin/dev as the migration comparison baseline.
A migration change or unknown baseline blocks binary rollback. The current comparison against origin/dev returned unknown because this branch already contains migration source changes. In that case the helper stops the unhealthy service and leaves the Index intact; an owner must restore the matching Index snapshot before a manual old-binary rollback. DESIGN §49 requires this protection. This means automatic rollback is conditional for migration-changing releases.
The local Podman smoke used the cached localhost/calternal-cloud:main image with the server entrypoint directly. Its /healthz passed. The full nested-Podman runtime entrypoint did not reach health during an earlier attempt, so nested Agent startup was not verified.
No production connection, deploy, push or merge was performed. git fetch origin && git merge origin/dev reported Already up to date.
No UI changed; UX and screenshot checks do not apply. apps/web/build was absent and no Rust crate changed.

Gates and local validation

$ PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests/deploy -p 'test_*.py'
........
----------------------------------------------------------------------
Ran 8 tests in 9.886s

OK
exit: 0
$ bash -n deploy/deploy-cloud.sh deploy/cloud/deploy-release.sh deploy/cloud/migration-compat.sh
exit: 0
$ deploy/deploy-cloud.sh --dry-run
dry run: build and deploy localhost/calternal-cloud:376ed5afbd1bfed0ac3cb44472a3ae2b2c7319e2
dry run: retain the active release and previous two; roll back if /healthz fails
dry run: prune unused images only after the candidate is healthy
exit: 0
$ git show --check --oneline HEAD
376ed5afb fix(deploy): commit release state before pruning
exit: 0
$ git diff --check
exit: 0
$ migration source comparison against origin/dev
unknown
exit: 0
$ cargo clean
     Removed 0 files
exit: 0
$ local Podman smoke
local Podman smoke: server /healthz returned success

Rust formatting, clippy and test gates were not run because this job changed no Rust crate. Web check and test gates were not run because this job changed no web code.

#732 is implemented on `job/deployfix-732`. Head: `376ed5afbd1bfed0ac3cb44472a3ae2b2c7319e2`. ## Built - Cloud images use the full Git SHA. Host state keeps the active release and two prior healthy releases with their matching Quadlets. Failed health checks restore the last healthy release when migration compatibility is established. - A healthy deploy removes releases beyond the retention set, removes dangling images without Podman's `--all`, and drops the legacy `main` alias. - Failures print a fixed class and the host journal instruction. The deploy helper never reads or forwards journal contents. - `--dry-run` prints the plan without a build or host connection. Files: `deploy/deploy-cloud.sh`, `deploy/cloud/deploy-release.sh`, `deploy/cloud/migration-compat.sh`, `deploy/cloud/README.md`, `tests/deploy/test_cloud_deploy.py`. ## Decisions and known gaps - Before the first SHA-tagged release, the old `main` image has no known Git SHA. The helper retains it by image ID and uses `origin/dev` as the migration comparison baseline. - A migration change or unknown baseline blocks binary rollback. The current comparison against `origin/dev` returned `unknown` because this branch already contains migration source changes. In that case the helper stops the unhealthy service and leaves the Index intact; an owner must restore the matching Index snapshot before a manual old-binary rollback. DESIGN §49 requires this protection. This means automatic rollback is conditional for migration-changing releases. - The local Podman smoke used the cached `localhost/calternal-cloud:main` image with the server entrypoint directly. Its `/healthz` passed. The full nested-Podman runtime entrypoint did not reach health during an earlier attempt, so nested Agent startup was not verified. - No production connection, deploy, push or merge was performed. `git fetch origin && git merge origin/dev` reported `Already up to date.` - No UI changed; UX and screenshot checks do not apply. `apps/web/build` was absent and no Rust crate changed. ## Gates and local validation ```text $ PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests/deploy -p 'test_*.py' ........ ---------------------------------------------------------------------- Ran 8 tests in 9.886s OK exit: 0 $ bash -n deploy/deploy-cloud.sh deploy/cloud/deploy-release.sh deploy/cloud/migration-compat.sh exit: 0 $ deploy/deploy-cloud.sh --dry-run dry run: build and deploy localhost/calternal-cloud:376ed5afbd1bfed0ac3cb44472a3ae2b2c7319e2 dry run: retain the active release and previous two; roll back if /healthz fails dry run: prune unused images only after the candidate is healthy exit: 0 $ git show --check --oneline HEAD 376ed5afb fix(deploy): commit release state before pruning exit: 0 $ git diff --check exit: 0 $ migration source comparison against origin/dev unknown exit: 0 $ cargo clean Removed 0 files exit: 0 $ local Podman smoke local Podman smoke: server /healthz returned success ``` Rust formatting, clippy and test gates were not run because this job changed no Rust crate. Web check and test gates were not run because this job changed no web code.

kayg referenced this issue

2026-10-02 17:33:01 +00:00

Plan: merge order and conflicts for merge round 7b #867

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/calternal#732

No description provided.

Rows
Columns