Add host: staging write failures, leftover temp bases, uninstall idempotency, unknown commit outcome, mesh-state list (J349 P3s, #394) #423

New issue

Open

opened 2026-11-07 14:20:35 +00:00 by kayg · 1 comment

kayg commented

2026-11-07 14:20:35 +00:00

Owner

Follow-up from J349's re-review of the D359 Take over slice (#394, comment 84681). The orchestrator decided these stay off addhost-takeover-d359: most are already on dev, and all of them fail safe (the old install is never lost). J349's root evidence is in compute01:/root/j349/ev/.

A failed write in staging leaves a rollback the server can't finish (failure-sweep.txt, steps 7 and 9).
- If printf "$kind" > "$stage/kind" fails, it leaves an empty kind, and ROLLBACK_TRANSACTION then exits 71 (kind=), although .bootstrap-acquiring beside it says fresh.
- An empty $stage/lease makes retire_setup_lease exit 1.
- Restore then refuses (72) with a base in place. The old install stays whole in the set-aside, but out of service with no way back inside the product.
- A power cut during staging gives the same empty-file shape.
- The lines are unchanged from dev (install.rs 1577/1680 on dev).
- Suggested fix: read kind only when [ -s ], else take the journal's; treat an empty lease as no lease.
A leftover temporary base from another transaction is never removed (j349-only.txt, [stale] rows, and rows 1–3 of [no-rollback], temp_left=True).
- $base.bootstrap-new-<tx> (only an owner id and a journal, no secrets) stays when a cut between mkdir and mv is followed by no rollback of that same transaction.
- Check reports INFRATERNAL_SETUP_ABSENT beside it.
- Suggested fix: Check or staging removes, under the setup guard, any private bootstrap-new-* of ours that holds only those two files.
The late-failure uninstall isn't idempotent across a cut inside the binary's own cleanup.
- crates/plugins/remote/src/uninstall.rs removes the unit and launcher (line 634) before remove_files, and validate requires the unit.
- A retry then exits 71, and the next Check reports unproven. The agent is already disabled, so nothing runs.
- On dev already; Take over and Re-enroll now reach it too.
A commit whose result never reaches the server still leaves an agent running for a revoked id.
- An SSH drop after the host committed leaves committed() false. ROLLBACK_TRANSACTION then exits 0 (owner matches, no stage, no journal), the restore answers untouched, and the server revokes the host.
- Fresh installs on dev have the same gap. It can be recovered: a re-add offers Take over of the revoked id.
- Suggested fix: also call uninstall_committed when the commit's outcome is unknown. It is owner-guarded and does nothing while a stage exists.
The mesh-state allowlist will drift.
- validate_upgrade_sessions accepts exactly six mesh-* names (from crates/plugins/mesh/src/node_state.rs). When the mesh plugin adds a state file, every real root install reads unproven and Take over is never offered. A commit-cut set-aside is then kept, with its old key, until the next commit.
- This fails safe.
- Suggested fix: a test that ties the list to node_state.rs.

Already fixed on addhost-takeover-d359: J349's P2-1 (sync before a commit deletes, f6138bc6f) and P3-3 (prefix drop-ins, 422dbb213).

Follow-up from J349's re-review of the D359 Take over slice (#394, comment 84681). The orchestrator decided these stay off `addhost-takeover-d359`: most are already on dev, and all of them fail safe (the old install is never lost). J349's root evidence is in `compute01:/root/j349/ev/`. 1. **A failed write in staging leaves a rollback the server can't finish** (`failure-sweep.txt`, steps 7 and 9). - If `printf "$kind" > "$stage/kind"` fails, it leaves an empty `kind`, and `ROLLBACK_TRANSACTION` then exits 71 (`kind=`), although `.bootstrap-acquiring` beside it says `fresh`. - An empty `$stage/lease` makes `retire_setup_lease` exit 1. - Restore then refuses (72) with a base in place. The old install stays whole in the set-aside, but out of service with no way back inside the product. - A power cut during staging gives the same empty-file shape. - The lines are unchanged from dev (`install.rs` 1577/1680 on dev). - **Suggested fix:** read `kind` only when `[ -s ]`, else take the journal's; treat an empty lease as no lease. 2. **A leftover temporary base from another transaction is never removed** (`j349-only.txt`, `[stale]` rows, and rows 1–3 of `[no-rollback]`, `temp_left=True`). - `$base.bootstrap-new-<tx>` (only an owner id and a journal, no secrets) stays when a cut between `mkdir` and `mv` is followed by no rollback of that same transaction. - Check reports `INFRATERNAL_SETUP_ABSENT` beside it. - **Suggested fix:** Check or staging removes, under the setup guard, any private `bootstrap-new-*` of ours that holds only those two files. 3. **The late-failure uninstall isn't idempotent across a cut inside the binary's own cleanup.** - `crates/plugins/remote/src/uninstall.rs` removes the unit and launcher (line 634) before `remove_files`, and `validate` requires the unit. - A retry then exits 71, and the next Check reports `unproven`. The agent is already disabled, so nothing runs. - On dev already; Take over and Re-enroll now reach it too. 4. **A commit whose result never reaches the server still leaves an agent running for a revoked id.** - An SSH drop after the host committed leaves `committed()` false. `ROLLBACK_TRANSACTION` then exits 0 (owner matches, no stage, no journal), the restore answers `untouched`, and the server revokes the host. - Fresh installs on dev have the same gap. It can be recovered: a re-add offers Take over of the revoked id. - **Suggested fix:** also call `uninstall_committed` when the commit's outcome is unknown. It is owner-guarded and does nothing while a stage exists. 5. **The mesh-state allowlist will drift.** - `validate_upgrade_sessions` accepts exactly six `mesh-*` names (from `crates/plugins/mesh/src/node_state.rs`). When the mesh plugin adds a state file, every real root install reads `unproven` and Take over is never offered. A commit-cut set-aside is then kept, with its old key, until the next commit. - This fails safe. - **Suggested fix:** a test that ties the list to `node_state.rs`. Already fixed on `addhost-takeover-d359`: J349's P2-1 (sync before a commit deletes, `f6138bc6f`) and P3-3 (prefix drop-ins, `422dbb213`).

kayg referenced this issue

2026-11-07 14:50:47 +00:00

Add host (root): upload fails with exit 71 and every retry repeats it; message tells the owner to inspect folders (J303) #394

kayg commented

2026-11-07 14:50:47 +00:00

Author

Owner

(J315, seen on compute01) The hidden-holder rule matches any same-user infraternal session scope. hidden_holder → install_process treats every hidden process of the same uid in an infraternal-session-*.scope (or ending in infraternal-agent.service) as a holder of this install, whichever install started it. With J347's shims running as claude, 24 oracle tests read busy (73). Root runs are unaffected. On a real host it would only matter for an account running two installs; it fails safe (busy). Suggested fix: tie a scope process to this install, for example by its argv0 resolved in its own root, as install_process already does for paths.

6. **(J315, seen on compute01) The hidden-holder rule matches any same-user infraternal session scope.** `hidden_holder` → `install_process` treats every hidden process of the same uid in an `infraternal-session-*.scope` (or ending in `infraternal-agent.service`) as a holder of *this* install, whichever install started it. With J347's shims running as `claude`, 24 oracle tests read busy (73). Root runs are unaffected. On a real host it would only matter for an account running two installs; it fails safe (busy). Suggested fix: tie a scope process to this install, for example by its argv0 resolved in its own root, as `install_process` already does for paths.

No labels

launch-qa

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/infraternal#423

No description provided.

Rows
Columns