Remove leaves the infraternal agent running: uninstall never starts on the host

kayg commented

2026-11-06 19:39:31 +00:00

Owner

Found by J320 on the test server and confirmed by J325 on both 74b9c4d80 (production) and 37c93637e (#305 comment 82827).

After Remove of an added rootless host:

the toast says "revoked, cleanup pending… Copy the command and run it there";
45 s later the infraternal agent is still running on the host, with its files and unit in place;
re-adding the host fails with "could not verify ownership or permissions of its files";
the server entry stays at "removal pending · access revoked".

The host journal shows the server dropping the infraternal agent connection about 0.3 s after the click, and the uninstall helper never starts on the host.

Logs, host-state snapshots and screenshots: compute01 ~claude/j325/.

Found by J320 on the test server and confirmed by J325 on both 74b9c4d80 (production) and 37c93637e (#305 comment 82827). After **Remove** of an added rootless host: - the toast says "revoked, cleanup pending… Copy the command and run it there"; - 45 s later the infraternal agent is still running on the host, with its files and unit in place; - re-adding the host fails with "could not verify ownership or permissions of its files"; - the server entry stays at "removal pending · access revoked". The host journal shows the server dropping the infraternal agent connection about 0.3 s after the click, and the uninstall helper never starts on the host. Logs, host-state snapshots and screenshots: compute01 `~claude/j325/`.

kayg commented

2026-11-06 19:43:24 +00:00

Author

Owner

J328 start

Branch fix-remove-uninstall-409, base origin/dev fba745051.

Plan (about 3–4 h):

Rebuild J325's scripted run on compute01: throwaway rootless account + sshd under systemd-run, own CA, throwaway branch server, own software-passkey copy. Mesh claim held while it runs. Never test server or production. (~45 min)
Find why UNINSTALL never reaches or never gets accepted by the infraternal agent. From reading so far: the ~0.3 s link cut matches the normal authority publish right after the removal transaction commits. The real question is why request_uninstall's control_call returns not-accepted (→ unreachable) right away. First suspect: infraternal-agent/src/plugins.rs registers UninstallPlugin only if let Ok(..) of a full artifact validation at agent start, and silently skips it on failure. That makes UNINSTALL an unknown action, with nothing logged. (~45 min)
Test-first fix, then real end-to-end runs for rootless and root: remove → agent stops, install gone, entry dropped after the receipt; re-add; offline host's pending cleanup finishes when it comes back. (~1.5 h)
If the removal flow turns out to have overlapping paths (server cut, cleanup outbox, manual uninstall), I'll write that up here with a single-path proposal before patching further.

## J328 start Branch `fix-remove-uninstall-409`, base origin/dev `fba745051`. **Plan (about 3–4 h):** 1. Rebuild J325's scripted run on compute01: throwaway rootless account + sshd under systemd-run, own CA, throwaway branch server, own software-passkey copy. Mesh claim held while it runs. Never test server or production. (~45 min) 2. Find why UNINSTALL never reaches or never gets accepted by the infraternal agent. From reading so far: the ~0.3 s link cut matches the normal authority publish right after the removal transaction commits. The real question is why `request_uninstall`'s `control_call` returns not-`accepted` (→ `unreachable`) right away. First suspect: `infraternal-agent/src/plugins.rs` registers `UninstallPlugin` only `if let Ok(..)` of a full artifact validation at agent start, and silently skips it on failure. That makes UNINSTALL an unknown action, with nothing logged. (~45 min) 3. Test-first fix, then real end-to-end runs for rootless and root: remove → agent stops, install gone, entry dropped after the receipt; re-add; offline host's pending cleanup finishes when it comes back. (~1.5 h) 4. If the removal flow turns out to have overlapping paths (server cut, cleanup outbox, manual uninstall), I'll write that up here with a single-path proposal before patching further.

kayg commented

2026-11-06 20:05:50 +00:00

Author

Owner

J328 finding 1: root cause. The server never sends the uninstall (regression from #285, `6ca0c6a74`, 2026-11-02)

EmbeddedAgent::control_call hands each control action to the host actor. The actor (crates/infraternal-server/src/embedded.rs, Command::Control) checks a per-action allow-list before anything goes on the wire:

let supported = match action {
    "transport.csr" | "transport.certificate" => …supports_renewal(),
    "transport.csr.rotate" | "transport.identity.rotate.certificate" | "transport.identity.rotate.ack" => …supports_identity_rotation(),
    _ => false,          // ← "remote.host.uninstall" lands here
};
if !supported { reply.send(Err(AgentError::Unsupported)); … }

Before 6ca0c6a74 ("schedule Mesh and host key rotation", #285), one check covered every control call, the uninstall included. That commit replaced it with this list and left remote.host.uninstall out. Since then every Remove goes like this:

request_uninstall gets Unsupported at once, without touching the wire. That is why the infraternal agent's journal shows nothing.
host_cleanup='unreachable', so the toast says "infraternal is still installed… Copy the command".
The removal transaction commits, and publishing the revoked authority cuts the link. That is the ~0.3 s BrokenPipe. The cut is correct (D352). It only looked like the cause.
The agent's cleanup-only reconnect delivers terminal.login.revoke, and the receipt drops the entry. The install stays, and re-add hits .bootstrap-owner (exit 71).

Why no gate caught it: the real-binary removal check (bootstrap_breakit2_tests.rs, BREAKIT2_REMOVE) calls this same request_uninstall path, but only when that environment variable is set, and the gate doesn't set it.

Repro on unfixed 37c93637e (J325's release build, throwaway server https://localhost:45328, rootless j328t@127.0.0.1:2328 behind a throwaway sshd, own software passkey under ~claude/j328/priv; logs in compute01 ~claude/j328/logs/repro37c-*):

20:03:43.235 A-remove REMOVE_CLICKED
20:03:44.101 A-remove TOAST 127.0.0.1: revoked, cleanup pending. infraternal is still installed on it. Copy the command and run it there.
20:04:44.836 HOST_NOT_CLEAN A-after-remove after 60 s running true installed true
   3100286 … /home/j328t/.local/share/infraternal/bin/current agent --data …   (unit: active)
LEDGER A-after-remove
remote_hosts [('855c1fc6…', 'rootless', 1793995425)]
remote_login_cleanup [('855c1fc6…', 1)]          ← the D352 receipt did arrive
remote_host_removals [('855c1fc6…', 1, 'unreachable')]
20:04:57 C-add (re-add) FAILURE_DETAILS … could not verify ownership or permissions of its files … Failed step: Copy infraternal to the host

A correction to the issue text: the server entry does retire once the cleanup receipt arrives (confirmed=1, revoked_at set). The stuck "removal pending" was not a server outbox failure in this run.

Fix (test first): the allow-list gets an explicit arm for remote.host.uninstall that requires only a remote link. Every infraternal agent build answers a stream-0 plugin call, by running its uninstall handler or by refusing. Red test remote_uninstall_delivery_tests.rs: real TLS, production router, a real enrolled certificate, and the agent side played by hand. It asserts that the server writes the uninstall call on stream 0, with this host's id and dry_run:false, and that an accepted reply comes back to the removal as requested. Red/green run queued on compute01 (slot queue position ~19).

## J328 finding 1: root cause. The server never sends the uninstall (regression from #285, 6ca0c6a74, 2026-11-02) `EmbeddedAgent::control_call` hands each control action to the host actor. The actor (`crates/infraternal-server/src/embedded.rs`, `Command::Control`) checks a per-action allow-list before anything goes on the wire: ```rust let supported = match action { "transport.csr" | "transport.certificate" => …supports_renewal(), "transport.csr.rotate" | "transport.identity.rotate.certificate" | "transport.identity.rotate.ack" => …supports_identity_rotation(), _ => false, // ← "remote.host.uninstall" lands here }; if !supported { reply.send(Err(AgentError::Unsupported)); … } ``` Before 6ca0c6a74 ("schedule Mesh and host key rotation", #285), one check covered every control call, the uninstall included. That commit replaced it with this list and left `remote.host.uninstall` out. Since then every Remove goes like this: 1. `request_uninstall` gets `Unsupported` at once, without touching the wire. That is why the infraternal agent's journal shows nothing. 2. `host_cleanup='unreachable'`, so the toast says "infraternal is still installed… Copy the command". 3. The removal transaction commits, and publishing the revoked authority cuts the link. That is the ~0.3 s `BrokenPipe`. The cut is correct (D352). It only looked like the cause. 4. The agent's cleanup-only reconnect delivers `terminal.login.revoke`, and the receipt drops the entry. The install stays, and re-add hits `.bootstrap-owner` (exit 71). Why no gate caught it: the real-binary removal check (`bootstrap_breakit2_tests.rs`, `BREAKIT2_REMOVE`) calls this same `request_uninstall` path, but only when that environment variable is set, and the gate doesn't set it. **Repro on unfixed 37c93637e** (J325's release build, throwaway server `https://localhost:45328`, rootless `j328t@127.0.0.1:2328` behind a throwaway sshd, own software passkey under `~claude/j328/priv`; logs in compute01 `~claude/j328/logs/repro37c-*`): ``` 20:03:43.235 A-remove REMOVE_CLICKED 20:03:44.101 A-remove TOAST 127.0.0.1: revoked, cleanup pending. infraternal is still installed on it. Copy the command and run it there. 20:04:44.836 HOST_NOT_CLEAN A-after-remove after 60 s running true installed true 3100286 … /home/j328t/.local/share/infraternal/bin/current agent --data … (unit: active) LEDGER A-after-remove remote_hosts [('855c1fc6…', 'rootless', 1793995425)] remote_login_cleanup [('855c1fc6…', 1)] ← the D352 receipt did arrive remote_host_removals [('855c1fc6…', 1, 'unreachable')] 20:04:57 C-add (re-add) FAILURE_DETAILS … could not verify ownership or permissions of its files … Failed step: Copy infraternal to the host ``` A correction to the issue text: the server entry **does** retire once the cleanup receipt arrives (`confirmed=1`, `revoked_at` set). The stuck "removal pending" was not a server outbox failure in this run. **Fix (test first):** the allow-list gets an explicit arm for `remote.host.uninstall` that requires only a remote link. Every infraternal agent build answers a stream-0 plugin call, by running its uninstall handler or by refusing. Red test `remote_uninstall_delivery_tests.rs`: real TLS, production router, a real enrolled certificate, and the agent side played by hand. It asserts that the server writes the uninstall call on stream 0, with this host's id and `dry_run:false`, and that an `accepted` reply comes back to the removal as `requested`. Red/green run queued on compute01 (slot queue position ~19).

kayg commented

2026-11-06 20:08:10 +00:00

Author

Owner

J328 finding 2: the removal flow has three overlapping paths, and none owns the uninstall end to end

Path	When it runs	What it delivers	Gap
Live call in `remote_removal::publish`	once, inside the SQLite writer transaction, only if the host is connected right then	`remote.host.uninstall`	Broken since #285 (finding 1). Even when fixed: (a) a network round trip of up to 5 s holds the writer lock; (b) it races the D352 receipt. The helper retires every session and runs `disable --now` on the agent. The receipt arrives only on the agent's next (cleanup-only) reconnect after the cut. If the helper wins, nothing ever confirms, and the entry stays "removal pending" forever.
Durable outbox (`remote_login_cleanup` → cleanup-only reconnect → `deliver_login_cleanup`)	on every reconnect until the receipt	`terminal.login.revoke` only	No uninstall. A host that was offline at Remove comes back, its entry is dropped, and its install keeps running and keeps being refused. Re-add then fails on `.bootstrap-owner`.
Manual `--bootstrap-uninstall <id>`	shown once in the toast, and only when the live call failed	—	It is the only uninstall an offline host ever gets.

Proposal: one path, the outbox

Remove commits intent only. In one transaction: the cleanup row, plus the removal row with uninstall and host_cleanup='pending'. No network call under the writer lock. The commit cuts the link, as today (D352).
The outbox delivers everything, in order, on the cleanup-only reconnect. For a just-cut online host that is the immediate reconnect; for an offline host it is its return:
1. terminal.login.revoke, which yields the durable D352 receipt;
2. if the removal asked for it and it has not been accepted yet: remote.host.uninstall;
3. one transaction records host_cleanup (requested / refused) and confirms the receipt, which drops the entry.
A reply lost before step 3 confirms nothing, so the next reconnect retries. Because the server confirms right after the agent accepts, the helper stopping the agent can no longer strand the entry.
The cleanup link's write filter also lets the one uninstall call through. It still passes no terminal, plugin or native-stream frames.
The toast says the host "uninstalls itself as soon as it connects", and it always carries the manual command as a fallback, for a host that never comes back.
Legacy non-UUID owners (test and legacy ledgers only; real accounts are UUIDs, as my repro's remote_login_cleanup row shows) have no outbox. They keep the one immediate attempt.

What we lose: the toast can no longer list, at click time, the kernel objects the host leaves because its install did not create them (#235's "infraternal0 stays…" line). The host still leaves them in place; only that one line goes. If the owner wants it back, it needs a column on remote_host_removals and a UI read, as a follow-up.

I'm implementing this on fix-remove-uninstall-409 as separate commits after the one-line root-cause fix. So the fix alone can still be taken if the owner prefers the smaller change.

## J328 finding 2: the removal flow has three overlapping paths, and none owns the uninstall end to end | Path | When it runs | What it delivers | Gap | |---|---|---|---| | **Live call** in `remote_removal::publish` | once, inside the SQLite writer transaction, only if the host is connected right then | `remote.host.uninstall` | Broken since #285 (finding 1). Even when fixed: (a) a network round trip of up to 5 s holds the writer lock; (b) it races the D352 receipt. The helper retires every session and runs `disable --now` on the agent. The receipt arrives only on the agent's *next* (cleanup-only) reconnect after the cut. If the helper wins, nothing ever confirms, and the entry stays "removal pending" forever. | | **Durable outbox** (`remote_login_cleanup` → cleanup-only reconnect → `deliver_login_cleanup`) | on every reconnect until the receipt | `terminal.login.revoke` only | **No uninstall.** A host that was offline at Remove comes back, its entry is dropped, and its install keeps running and keeps being refused. Re-add then fails on `.bootstrap-owner`. | | **Manual** `--bootstrap-uninstall <id>` | shown once in the toast, and only when the live call failed | — | It is the only uninstall an offline host ever gets. | ### Proposal: one path, the outbox 1. **Remove commits intent only.** In one transaction: the cleanup row, plus the removal row with `uninstall` and `host_cleanup='pending'`. No network call under the writer lock. The commit cuts the link, as today (D352). 2. **The outbox delivers everything, in order,** on the cleanup-only reconnect. For a just-cut online host that is the immediate reconnect; for an offline host it is its return: 1. `terminal.login.revoke`, which yields the durable D352 receipt; 2. if the removal asked for it and it has not been accepted yet: `remote.host.uninstall`; 3. one transaction records `host_cleanup` (`requested` / `refused`) and confirms the receipt, which drops the entry. A reply lost before step 3 confirms nothing, so the next reconnect retries. Because the server confirms right after the agent accepts, the helper stopping the agent can no longer strand the entry. 3. **The cleanup link's write filter** also lets the one uninstall call through. It still passes no terminal, plugin or native-stream frames. 4. **The toast** says the host "uninstalls itself as soon as it connects", and it always carries the manual command as a fallback, for a host that never comes back. 5. **Legacy non-UUID owners** (test and legacy ledgers only; real accounts are UUIDs, as my repro's `remote_login_cleanup` row shows) have no outbox. They keep the one immediate attempt. What we lose: the toast can no longer list, at click time, the kernel objects the host leaves because its install did not create them (#235's "infraternal0 stays…" line). The host still leaves them in place; only that one line goes. If the owner wants it back, it needs a column on `remote_host_removals` and a UI read, as a follow-up. I'm implementing this on `fix-remove-uninstall-409` as separate commits after the one-line root-cause fix. So the fix alone can still be taken if the owner prefers the smaller change.

kayg commented

2026-11-06 20:15:39 +00:00

Author

Owner

J328 progress (21:20 CET)

Pushed fix-remove-uninstall-409 at e7647ead8 (rebased on origin/dev fba745051):

7497a582c fix(remote): send Remove's uninstall request to the host. The allow-list arm, plus the red-first delivery test remote_uninstall_delivery_tests.rs.
0d8ce8de0 refactor(remote): the removal outbox alone asks a host to uninstall (the single path proposed above). Tests in remote_removal_outbox_tests.rs: receipt → uninstall → one confirming transaction; a lost answer is asked again on the next reconnect; a refusal retires the entry with unreachable.
e7647ead8 fix(tracer): a pending uninstall is not a failure toast.

Waiting for a compute01 build slot (queue position 19 since 20:47). One slot run does: two red proofs (each fix reverse-applied on the final tree), green, verify quick, then the release build for the end-to-end runs. Script ~claude/j328/red.sh, log ~claude/j328/logs/red.log, target ~claude/.cache/codex-scratch/j328/target.

Next: end-to-end runs on the release build. Rootless: throwaway account j328t + sshd under /root/j328. Root: a throwaway AlmaLinux qemu VM, because compute01's own /var/lib/infraternal belongs to another job. Repro evidence: ~claude/j328/logs/repro37c-*.

## J328 progress (21:20 CET) Pushed `fix-remove-uninstall-409` at `e7647ead8` (rebased on origin/dev `fba745051`): 1. `7497a582c` fix(remote): send Remove's uninstall request to the host. The allow-list arm, plus the red-first delivery test `remote_uninstall_delivery_tests.rs`. 2. `0d8ce8de0` refactor(remote): the removal outbox alone asks a host to uninstall (the single path proposed above). Tests in `remote_removal_outbox_tests.rs`: receipt → uninstall → one confirming transaction; a lost answer is asked again on the next reconnect; a refusal retires the entry with `unreachable`. 3. `e7647ead8` fix(tracer): a pending uninstall is not a failure toast. Waiting for a compute01 build slot (queue position 19 since 20:47). One slot run does: two red proofs (each fix reverse-applied on the final tree), green, `verify quick`, then the release build for the end-to-end runs. Script `~claude/j328/red.sh`, log `~claude/j328/logs/red.log`, target `~claude/.cache/codex-scratch/j328/target`. Next: end-to-end runs on the release build. Rootless: throwaway account `j328t` + sshd under `/root/j328`. Root: a throwaway AlmaLinux qemu VM, because compute01's own `/var/lib/infraternal` belongs to another job. Repro evidence: `~claude/j328/logs/repro37c-*`.

kayg commented

2026-11-06 20:46:33 +00:00

Author

Owner

J328 paused for compute01 maintenance (02:15–02:45 IST)

build-slot admission is frozen for the VM resize. My queued slot run (~claude/j328/red.sh: red proofs, green, verify quick, release build) never started: it had waited since 02:17 IST and was at queue position ~3. I stopped it by PID before the restart, so it was cut by the maintenance, not by a failure. I also powered off my throwaway root VM (/root/j328/vm) cleanly. The rootless test account is already removed, and I hold no mesh claim.

Code is pushed at e7647ead8. Once "compute01 back, 5 slots" is posted on #402, I'll resubmit the same slot run once, in the foreground, then run the end-to-end runs (rootless, then root VM).

## J328 paused for compute01 maintenance (02:15–02:45 IST) build-slot admission is frozen for the VM resize. My queued slot run (`~claude/j328/red.sh`: red proofs, green, `verify quick`, release build) **never started**: it had waited since 02:17 IST and was at queue position ~3. I stopped it by PID before the restart, so it was cut by the maintenance, not by a failure. I also powered off my throwaway root VM (`/root/j328/vm`) cleanly. The rootless test account is already removed, and I hold no mesh claim. Code is pushed at `e7647ead8`. Once "compute01 back, 5 slots" is posted on #402, I'll resubmit the same slot run once, in the foreground, then run the end-to-end runs (rootless, then root VM).

kayg commented

2026-11-06 22:13:40 +00:00

Author

Owner

J328 progress (04:45 IST)

compute01 is back. My resubmitted slot run hit two harness mistakes of mine, each found in the slot's first minutes, then stopped by PID and fixed:

the targeted test run needed --features remote (the workspace gate gets it through feature unification): 0 tests ran;
the outbox test's #[path] inside an inline mod tests needed ../../: it did not compile.

Head is now b627895ae (7497a582c fix, 1d8bd2917 outbox, b627895ae toast). I also tightened confirm_removal_cleanup: its host_cleanup update now requires this person's unconfirmed cleanup, like the receipt statements beside it. Waiting in the slot queue (position ~19). Logs: ~claude/j328/logs/red.log, and the earlier attempts in red-*.log.

## J328 progress (04:45 IST) compute01 is back. My resubmitted slot run hit two harness mistakes of mine, each found in the slot's first minutes, then stopped by PID and fixed: - the targeted test run needed `--features remote` (the workspace gate gets it through feature unification): 0 tests ran; - the outbox test's `#[path]` inside an inline `mod tests` needed `../../`: it did not compile. Head is now `b627895ae` (7497a582c fix, 1d8bd2917 outbox, b627895ae toast). I also tightened `confirm_removal_cleanup`: its `host_cleanup` update now requires this person's unconfirmed cleanup, like the receipt statements beside it. Waiting in the slot queue (position ~19). Logs: `~claude/j328/logs/red.log`, and the earlier attempts in `red-*.log`.

kayg commented

2026-11-07 00:20:46 +00:00

Author

Owner

J328 progress (06:30 IST)

Release build of the fix is ready: ~claude/j328/release-b627895ae. The final head 14b4d118d differs from it only in test-only code and formatting.
The fixed test run (~claude/j328/tests.sh: red1, red2, green, verify quick at 14b4d118d) is queued at position ~25. The earlier slot run's tests did not compile because of the test module's #[path] (fixed). That run's log is ~claude/j328/logs/red.log.
Blocked on the compute01 mesh claim: ~claude/mesh-vm.claim has said J197 2026-11-06T22:41:10Z for about 2 h, and no J197 process is running. I'm waiting, not removing it. Orchestrator: please check whether it is stale. The end-to-end runs (rootless, then the root VM) start as soon as I hold it.

## J328 progress (06:30 IST) - Release build of the fix is ready: `~claude/j328/release-b627895ae`. The final head `14b4d118d` differs from it only in test-only code and formatting. - The fixed test run (`~claude/j328/tests.sh`: red1, red2, green, `verify quick` at `14b4d118d`) is queued at position ~25. The earlier slot run's tests did not compile because of the test module's `#[path]` (fixed). That run's log is `~claude/j328/logs/red.log`. - **Blocked on the compute01 mesh claim:** `~claude/mesh-vm.claim` has said `J197 2026-11-06T22:41:10Z` for about 2 h, and no J197 process is running. I'm waiting, not removing it. Orchestrator: please check whether it is stale. The end-to-end runs (rootless, then the root VM) start as soon as I hold it.

kayg commented

2026-11-07 03:25:15 +00:00

Author

Owner

J328: red/green proven; gate queued but blocked on compute01 disk

Red/green at c98b7cdaa (log: compute01 ~claude/j328/logs/tests.log):

RED1, with the embedded.rs allow-list arm reverse-applied: uninstall_delivery::removal_uninstall_reaches_the_connected_agent ... FAILED, panicked at remote_uninstall_delivery_tests.rs:109 ("the server never sent the uninstall request to the connected agent"). That is the production bug.
RED2, with the outbox delivery in remote_runtime.rs reverse-applied: all 3 removal_outbox_tests FAILED at remote_removal_outbox_tests.rs:194 ("the server sent no control call" after the receipt).
GREEN (uninstall removal cleanup revocation revoke filters, --features remote): 127 passed; 1 failed. The failure was my new wording test, and it found a real copy bug, see below.

Copy bug found (it exists today too): for a rootless host, the leftover-install messages said "remove it there as the account it was added with with: ". It is fixed in its own commit, e20c06ec0, with a test covering both rootless messages. The new pending message uses the same "… by running: " form.

verify quick at c98b7cdaa: fmt, doc coverage, D351 shape lints, doc comments on diff, fixed test waits, and workspace clippy all PASS. Unit tests (changed) failed only on that wording test. Log: ~claude/j328/logs/verify-quick.log.

Head e20c06ec0 (on origin/dev fba745051, unchanged):

aa564b116 fix(remote): send Remove's uninstall request to the host
0cfeac439 refactor(remote): the removal outbox alone asks a host to uninstall
7e96ea09b fix(tracer): a pending uninstall is not a failure toast
e20c06ec0 fix(remote): a rootless leftover's command says "added with" once

Blocked, orchestrator:

verify fast is queued (~claude/j328/gate.sh), but build-slot waits: compute01 has 21 GB free (546/591 G used; below the 25 G floor). My own use is an 11 G target that the gate needs, plus 0.5 G (the throwaway VM disk).
The mesh claim still reads J197 2026-11-06T22:41:10Z, so the end-to-end runs can't start.

## J328: red/green proven; gate queued but blocked on compute01 disk **Red/green at `c98b7cdaa`** (log: compute01 `~claude/j328/logs/tests.log`): - **RED1**, with the `embedded.rs` allow-list arm reverse-applied: `uninstall_delivery::removal_uninstall_reaches_the_connected_agent ... FAILED`, panicked at `remote_uninstall_delivery_tests.rs:109` ("the server never sent the uninstall request to the connected agent"). That is the production bug. - **RED2**, with the outbox delivery in `remote_runtime.rs` reverse-applied: all 3 `removal_outbox_tests` FAILED at `remote_removal_outbox_tests.rs:194` ("the server sent no control call" after the receipt). - **GREEN** (`uninstall removal cleanup revocation revoke` filters, `--features remote`): `127 passed; 1 failed`. The failure was my new wording test, and it found a real copy bug, see below. **Copy bug found (it exists today too):** for a rootless host, the leftover-install messages said "remove it there as the account it was added with **with**: <command>". It is fixed in its own commit, `e20c06ec0`, with a test covering both rootless messages. The new pending message uses the same "… by running: <command>" form. `verify quick` at `c98b7cdaa`: fmt, doc coverage, D351 shape lints, doc comments on diff, fixed test waits, and workspace clippy all PASS. Unit tests (changed) failed only on that wording test. Log: `~claude/j328/logs/verify-quick.log`. **Head `e20c06ec0`** (on origin/dev `fba745051`, unchanged): 1. `aa564b116` fix(remote): send Remove's uninstall request to the host 2. `0cfeac439` refactor(remote): the removal outbox alone asks a host to uninstall 3. `7e96ea09b` fix(tracer): a pending uninstall is not a failure toast 4. `e20c06ec0` fix(remote): a rootless leftover's command says "added with" once **Blocked, orchestrator:** - `verify fast` is queued (`~claude/j328/gate.sh`), but build-slot waits: compute01 has **21 GB free** (546/591 G used; below the 25 G floor). My own use is an 11 G target that the gate needs, plus 0.5 G (the throwaway VM disk). - The mesh claim still reads `J197 2026-11-06T22:41:10Z`, so the end-to-end runs can't start.

kayg commented

2026-11-07 04:44:49 +00:00

Author

Owner

J328: first full gate at `e20c06ec0`, and a fix for a test that failed under load

verify fast at e20c06ec0 (compute01, log ~claude/j328/logs/gate-e20c06ec0.log): every row PASS except:

test: 954 passed; 1 failed. The failure was my lost_uninstall_answer_is_asked_again_on_the_next_reconnect, panicked at remote_removal_outbox_tests.rs:194:56. Under full-suite load, the server's first cleanup task had not yet seen the dropped link, so the per-host cleanup lane refused the test's reconnect (WouldBlock). The test assumed a fixed 300 ms sleep was enough. Fixed: the test now reconnects the way a real infraternal agent does, retrying until the server asks for the receipt, with no fixed sleep. Product behaviour is unchanged: a real agent retries its reconnect.
static builds: No space left on device while building musl OpenSSL, because compute01's disk filled up during the run. Environmental. I dropped my own 15 G incremental cache to help: free space went from 7.7 G to 27 G.

New head 2db5024f4: aa564b116 fix, 5ede0f539 outbox, 83bd06181 toast, 2db5024f4 wording. The final gate is queued. Mesh claim still J197 2026-11-06T22:41:10Z, so the end-to-end runs are still waiting on it.

## J328: first full gate at `e20c06ec0`, and a fix for a test that failed under load `verify fast` at `e20c06ec0` (compute01, log `~claude/j328/logs/gate-e20c06ec0.log`): every row PASS except: - **test**: `954 passed; 1 failed`. The failure was my `lost_uninstall_answer_is_asked_again_on_the_next_reconnect`, panicked at `remote_removal_outbox_tests.rs:194:56`. Under full-suite load, the server's first cleanup task had not yet seen the dropped link, so the per-host cleanup lane refused the test's reconnect (WouldBlock). The test assumed a fixed 300 ms sleep was enough. **Fixed:** the test now reconnects the way a real infraternal agent does, retrying until the server asks for the receipt, with no fixed sleep. Product behaviour is unchanged: a real agent retries its reconnect. - **static builds**: `No space left on device` while building musl OpenSSL, because compute01's disk filled up during the run. Environmental. I dropped my own 15 G incremental cache to help: free space went from 7.7 G to 27 G. New head **`2db5024f4`**: aa564b116 fix, 5ede0f539 outbox, 83bd06181 toast, 2db5024f4 wording. The final gate is queued. Mesh claim still `J197 2026-11-06T22:41:10Z`, so the end-to-end runs are still waiting on it.

kayg commented

2026-11-07 06:10:37 +00:00

Author

Owner

J328 `verify fast` at `2db5024f4` (compute01; log `~claude/j328/logs/gate-2db5024f4.log`)

| Rust gate                 | Result       |
+---------------------------+--------------+
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | FAIL (101)   |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| privileged integration    | SKIP         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
VERIFY_EXIT=1

Every row passes except test, and the only failing suite is crates/plugins/ssh/tests/install.rs:

---- earlier_binary_cache_is_refreshed_before_owned_recovery stdout ----

thread 'earlier_binary_cache_is_refreshed_before_owned_recovery' (2526049) panicked at crates/plugins/ssh/tests/install.rs:2265:17:
assertion `left == right` failed: Setup is already running on this host. Wait for it to finish, then try again.
  left: Failed
 right: Done
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test result: FAILED. 96 passed; 9 failed; 2 ignored; 0 measured; 0 filtered out; finished in 304.50s

Not from this change:

This branch touches only crates/infraternal-server and apps/tracer, and plugins/ssh does not depend on infraternal-server. The failing test binary is byte-identical between my two gate runs: target/debug/deps/install-a8f6df12f81dec5d.
The same binary passed in my previous gate run at e20c06ec0: test result: ok. 105 passed; 0 failed; 2 ignored … finished in 14.19s (gate-e20c06ec0.log). This time it took 304.50 s, more than 20× slower, and the failures are setup-lock refusals (exit 73, "Setup is already running on this host"). That is host load during this run (5 slots, disk churn), not code.
My own tests all pass in this run. The suite containing uninstall_delivery, removal_outbox_tests, pending_uninstall_says… and rootless_manual_wording_reads_once is among the 150 test result: ok lines.
static builds now passes; the earlier ENOSPC was environmental.

Still to do: the end-to-end runs. They are blocked on the compute01 mesh claim, which has said J197 2026-11-06T22:41:10Z for about 8 h.

## J328 `verify fast` at `2db5024f4` (compute01; log `~claude/j328/logs/gate-2db5024f4.log`) ``` | Rust gate | Result | +---------------------------+--------------+ | Nix Cargo vendoring | PASS | | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | FAIL (101) | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | SKIP | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | static builds | PASS | +---------------------------+--------------+ VERIFY_EXIT=1 ``` Every row passes except **test**, and the only failing suite is `crates/plugins/ssh/tests/install.rs`: ``` ---- earlier_binary_cache_is_refreshed_before_owned_recovery stdout ---- thread 'earlier_binary_cache_is_refreshed_before_owned_recovery' (2526049) panicked at crates/plugins/ssh/tests/install.rs:2265:17: assertion `left == right` failed: Setup is already running on this host. Wait for it to finish, then try again. left: Failed right: Done note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace test result: FAILED. 96 passed; 9 failed; 2 ignored; 0 measured; 0 filtered out; finished in 304.50s ``` **Not from this change:** - This branch touches only `crates/infraternal-server` and `apps/tracer`, and `plugins/ssh` does not depend on `infraternal-server`. The failing test binary is byte-identical between my two gate runs: `target/debug/deps/install-a8f6df12f81dec5d`. - The **same binary** passed in my previous gate run at `e20c06ec0`: `test result: ok. 105 passed; 0 failed; 2 ignored … finished in 14.19s` (`gate-e20c06ec0.log`). This time it took 304.50 s, more than 20× slower, and the failures are setup-lock refusals (exit 73, "Setup is already running on this host"). That is host load during this run (5 slots, disk churn), not code. - My own tests all pass in this run. The suite containing `uninstall_delivery`, `removal_outbox_tests`, `pending_uninstall_says…` and `rootless_manual_wording_reads_once` is among the 150 `test result: ok` lines. - `static builds` now passes; the earlier ENOSPC was environmental. Still to do: the end-to-end runs. They are blocked on the compute01 mesh claim, which has said `J197 2026-11-06T22:41:10Z` for about 8 h.

kayg referenced this issue

2026-11-07 06:47:46 +00:00

Root add fails on SELinux-enforcing hosts: systemd may not execute /var/lib/infraternal/bin/current #417

kayg commented

2026-11-07 06:53:14 +00:00

Author

Owner

J328 end-to-end, first pass: rootless passes all four steps; root found two more problems

Release build of b627895ae (~claude/j328/release-b627895ae), throwaway server https://localhost:45328, own software passkey, mesh claim held.

Rootless `j328t@127.0.0.1:2328`: PASS

Log ~claude/j328/logs/fix-rootless-scenario.log, host snapshots in fix-rootless-state-*.txt, screenshots in ~claude/j328/shots/fix-rootless-*.

RESULTS {"A_add":true,"A_toast":"127.0.0.1: access revoked. It uninstalls infraternal as soon as it connects. If it never comes back, copy the command and run it there. | Copy command","A_host_clean":true,"A_entry_gone":true,"B_readd":true,"C_toast":"…","C_still_pending":true,"C_host_clean":true,"C_entry_gone":true,"D_readd":true}
06:30:16.623Z A-remove REMOVE_CLICKED
06:30:21.039Z HOST_CLEAN A-after-remove after 2.6 s        (agent gone, install gone)
LEDGER A-after-remove: remote_login_cleanup confirmed=1, remote_host_removals host_cleanup='requested', remote_host_bootstrap []
06:31:07.809Z ENTRY_STUCK C-offline-wait live=1 pending_cleanup=1   (expected: the agent is stopped, so the entry stays pending)
06:31:11.003Z HOST_CLEAN C-after-return after 2.8 s
LEDGER C-after-return: confirmed=1, host_cleanup='requested'

Host journal: the outbox in action, online, then the offline host's return.

07:30:17 current: Infraternal agent HTTPS server link closed (error Some(BrokenPipe)); reconnecting
07:30:17 systemd: Started run-p3128331-i7322443.service - [systemd-run] …/bin/live --bootstrap-uninstall.
07:30:18 systemd: Stopped infraternal-agent.service
07:31:08 systemd: Started run-p3144813-i7338925.service - [systemd-run] …/bin/live --bootstrap-uninstall.   ← agent started again after being offline
07:31:08 systemd: Stopped infraternal-agent.service

Root `root@127.0.0.1:2338` (throwaway AlmaLinux 10 qemu VM): two problems

New issue #417: a root add fails on SELinux-enforcing hosts. avc: denied { execute } … scontext=init_t tcontext=var_lib_t name="payload", so systemd may not exec /var/lib/infraternal/bin/current. This is unrelated to #409. For this proof I set only the throwaway VM to permissive.
#409, root only: the outbox delivered everything. Ledger: host_cleanup='requested', confirmed=1, and the helper unit started at the click second. But the helper itself exited 1. Run by hand: infraternal was not removed: Connection refused (os error 111). The login revocation, which the outbox correctly delivers first (D352), kills a root login's unit outright (infraternal-login-…service: code=killed, status=9/KILL, D354). Its shim never removes sessions/sessions/<id>.sock, and the helper refused the whole uninstall over that one stale socket. Rootless shims are retired through the shim protocol and clean up after themselves, so rootless never hit it.

Fix fae08b029: a refused or vanished session socket counts as retired, because a shim listens for its whole life, and the helper removes the stale socket. It removes only a real socket owned by the installation's uid, since Linux also answers ECONNREFUSED for a regular file. Tests: stale socket → retired and removed; a regular file at that path → still refused and kept.

On the redesign rule: this is a new hole, but in a different mechanism (the helper's tolerance of already-revoked sessions), not another hole in the delivery path. The fix closes the whole class: any already-dead session, whoever killed it.

Next: red/green for fae08b029, a new release build, a root rerun (fresh VM overlay), then the final gate.

## J328 end-to-end, first pass: rootless passes all four steps; root found two more problems Release build of `b627895ae` (`~claude/j328/release-b627895ae`), throwaway server `https://localhost:45328`, own software passkey, mesh claim held. ### Rootless `j328t@127.0.0.1:2328`: PASS Log `~claude/j328/logs/fix-rootless-scenario.log`, host snapshots in `fix-rootless-state-*.txt`, screenshots in `~claude/j328/shots/fix-rootless-*`. ``` RESULTS {"A_add":true,"A_toast":"127.0.0.1: access revoked. It uninstalls infraternal as soon as it connects. If it never comes back, copy the command and run it there. | Copy command","A_host_clean":true,"A_entry_gone":true,"B_readd":true,"C_toast":"…","C_still_pending":true,"C_host_clean":true,"C_entry_gone":true,"D_readd":true} 06:30:16.623Z A-remove REMOVE_CLICKED 06:30:21.039Z HOST_CLEAN A-after-remove after 2.6 s (agent gone, install gone) LEDGER A-after-remove: remote_login_cleanup confirmed=1, remote_host_removals host_cleanup='requested', remote_host_bootstrap [] 06:31:07.809Z ENTRY_STUCK C-offline-wait live=1 pending_cleanup=1 (expected: the agent is stopped, so the entry stays pending) 06:31:11.003Z HOST_CLEAN C-after-return after 2.8 s LEDGER C-after-return: confirmed=1, host_cleanup='requested' ``` Host journal: the outbox in action, online, then the offline host's return. ``` 07:30:17 current: Infraternal agent HTTPS server link closed (error Some(BrokenPipe)); reconnecting 07:30:17 systemd: Started run-p3128331-i7322443.service - [systemd-run] …/bin/live --bootstrap-uninstall. 07:30:18 systemd: Stopped infraternal-agent.service 07:31:08 systemd: Started run-p3144813-i7338925.service - [systemd-run] …/bin/live --bootstrap-uninstall. ← agent started again after being offline 07:31:08 systemd: Stopped infraternal-agent.service ``` ### Root `root@127.0.0.1:2338` (throwaway AlmaLinux 10 qemu VM): two problems 1. **New issue #417:** a root add fails on SELinux-enforcing hosts. `avc: denied { execute } … scontext=init_t tcontext=var_lib_t name="payload"`, so systemd may not exec `/var/lib/infraternal/bin/current`. This is unrelated to #409. For this proof I set only the throwaway VM to permissive. 2. **#409, root only:** the outbox delivered everything. Ledger: `host_cleanup='requested'`, `confirmed=1`, and the helper unit started at the click second. But the **helper itself exited 1**. Run by hand: `infraternal was not removed: Connection refused (os error 111)`. The login revocation, which the outbox correctly delivers first (D352), kills a root login's unit outright (`infraternal-login-…service: code=killed, status=9/KILL`, D354). Its shim never removes `sessions/sessions/<id>.sock`, and the helper refused the whole uninstall over that one stale socket. Rootless shims are retired through the shim protocol and clean up after themselves, so rootless never hit it. **Fix `fae08b029`:** a refused or vanished session socket counts as retired, because a shim listens for its whole life, and the helper removes the stale socket. It removes only a real socket owned by the installation's uid, since Linux also answers ECONNREFUSED for a regular file. Tests: stale socket → retired and removed; a regular file at that path → still refused and kept. On the redesign rule: this is a new hole, but in a different mechanism (the helper's tolerance of already-revoked sessions), not another hole in the delivery path. The fix closes the whole class: any already-dead session, whoever killed it. Next: red/green for `fae08b029`, a new release build, a root rerun (fresh VM overlay), then the final gate.

kayg commented

2026-11-07 09:41:09 +00:00

Author

Owner

J328 progress (15:10 IST)

Helper fix fae08b029, red/green (compute01 ~claude/j328/logs/run3.log; the plugin's tests need --features agent,tokio/rt-multi-thread when run on their own):

RED3, with the retire_shim connect handling reverted: a_stale_session_socket_counts_as_retired_and_is_removed ... FAILED (panicked at uninstall.rs:1083, "a stale socket refused the uninstall"). The regular-file guard passes.
GREEN: test result: ok. 24 passed; 0 failed (uninstall, retire filters).

Release build of fae08b029 is ready: ~claude/j328/release-fae08b029. A fresh root VM is booted and set permissive (#417). The root end-to-end rerun is waiting on the mesh claim (J207 2026-11-07T06:55:47Z). After it come a rootless rerun on the same binary and the final verify fast.

## J328 progress (15:10 IST) Helper fix `fae08b029`, red/green (compute01 `~claude/j328/logs/run3.log`; the plugin's tests need `--features agent,tokio/rt-multi-thread` when run on their own): - **RED3**, with the `retire_shim` connect handling reverted: `a_stale_session_socket_counts_as_retired_and_is_removed ... FAILED` (panicked at `uninstall.rs:1083`, "a stale socket refused the uninstall"). The regular-file guard passes. - **GREEN**: `test result: ok. 24 passed; 0 failed` (`uninstall`, `retire` filters). Release build of `fae08b029` is ready: `~claude/j328/release-fae08b029`. A fresh root VM is booted and set permissive (#417). The root end-to-end rerun is waiting on the mesh claim (`J207 2026-11-07T06:55:47Z`). After it come a rootless rerun on the same binary and the final `verify fast`.

kayg commented

2026-11-07 11:29:51 +00:00

Author

Owner

J328 end-to-end proof on the final build: rootless and root both PASS

Release build of fae08b029 (~claude/j328/release-fae08b029), throwaway server https://localhost:45328, own software passkey, mesh claim held 11:23–11:29 UTC. Every phase runs through the UI: A add + shell, then Remove while online; B re-add; C stop the infraternal agent, Remove while offline, start it again; D re-add.

Root `root@127.0.0.1:2338`: throwaway AlmaLinux 10 VM (SELinux set permissive only because of #417)

Log ~claude/j328/logs/fix2-root-scenario.log, snapshots fix2-root-state-*.txt.

11:24:48.570Z A-remove REMOVE_CLICKED
11:24:52.933Z HOST_CLEAN A-after-remove after 3.7 s      (no infraternal process; /var/lib/infraternal, /usr/local/bin/infraternal and the unit are gone)
11:24:53.019Z ENTRY_GONE A-after-remove after 0.1 s
11:25:44.933Z ENTRY_STUCK C-offline-wait live=1 pending_cleanup=1   (expected while offline)
11:25:49.441Z HOST_CLEAN C-after-return after 3.7 s
11:25:49.497Z ENTRY_GONE C-after-return after 0.1 s
RESULTS … "A_host_clean":true,"A_entry_gone":true,"B_readd":true,"C_still_pending":true,"C_host_clean":true,"C_entry_gone":true,"D_readd":true
LEDGER: remote_host_removals host_cleanup='requested' and remote_login_cleanup confirmed=1 for both removed hosts

VM journal: the stale-socket case that fae08b029 fixes, now passing.

11:24:48 systemd: infraternal-login-…-0.service: Main process exited, code=killed, status=9/KILL   ← login revocation (the receipt step)
11:24:49 systemd: Started run-p4192-i4193.service - [systemd-run] /var/lib/infraternal/bin/live --bootstrap-uninstall.
11:24:50 systemd: Stopped infraternal-agent.service
11:25:45 systemd: infraternal-login-…-0.service: … status=9/KILL                                  ← offline host came back
11:25:45 systemd: Started run-p4835-i4836.service - … --bootstrap-uninstall.
11:25:46 systemd: Stopped infraternal-agent.service

Rootless `j328t@127.0.0.1:2328`, same build

Log ~claude/j328/logs/fix2-rootless-scenario.log.

RESULTS "A_add":true "A_host_clean":true "A_entry_gone":true "B_readd":true "C_still_pending":true "C_host_clean":true "C_entry_gone":true "D_readd":true
HOST_CLEAN A-after-remove after 0.4 s · HOST_CLEAN C-after-return after 1.9 s

Toast (production build, 1440×900)

The pending uninstall is no longer an error: it shows the neutral toast with Copy command as the fallback. Screenshot attached (fix2-rootless-C-remove-offline-toast.png). The root run's toast screenshot caught a later frame without the toast; its text is in the log above. Not mine: the details panel in the background shows the server's own machine, J325's finding A.

Cleanup: tunnel, root helper, test account, sshd and VM are stopped or removed; the claim is released. The final verify fast at fae08b029 is running.

## J328 end-to-end proof on the final build: rootless and root both PASS Release build of `fae08b029` (`~claude/j328/release-fae08b029`), throwaway server `https://localhost:45328`, own software passkey, mesh claim held 11:23–11:29 UTC. Every phase runs through the UI: A add + shell, then Remove while online; B re-add; C stop the infraternal agent, Remove while offline, start it again; D re-add. ### Root `root@127.0.0.1:2338`: throwaway AlmaLinux 10 VM (SELinux set permissive only because of #417) Log `~claude/j328/logs/fix2-root-scenario.log`, snapshots `fix2-root-state-*.txt`. ``` 11:24:48.570Z A-remove REMOVE_CLICKED 11:24:52.933Z HOST_CLEAN A-after-remove after 3.7 s (no infraternal process; /var/lib/infraternal, /usr/local/bin/infraternal and the unit are gone) 11:24:53.019Z ENTRY_GONE A-after-remove after 0.1 s 11:25:44.933Z ENTRY_STUCK C-offline-wait live=1 pending_cleanup=1 (expected while offline) 11:25:49.441Z HOST_CLEAN C-after-return after 3.7 s 11:25:49.497Z ENTRY_GONE C-after-return after 0.1 s RESULTS … "A_host_clean":true,"A_entry_gone":true,"B_readd":true,"C_still_pending":true,"C_host_clean":true,"C_entry_gone":true,"D_readd":true LEDGER: remote_host_removals host_cleanup='requested' and remote_login_cleanup confirmed=1 for both removed hosts ``` VM journal: the stale-socket case that `fae08b029` fixes, now passing. ``` 11:24:48 systemd: infraternal-login-…-0.service: Main process exited, code=killed, status=9/KILL ← login revocation (the receipt step) 11:24:49 systemd: Started run-p4192-i4193.service - [systemd-run] /var/lib/infraternal/bin/live --bootstrap-uninstall. 11:24:50 systemd: Stopped infraternal-agent.service 11:25:45 systemd: infraternal-login-…-0.service: … status=9/KILL ← offline host came back 11:25:45 systemd: Started run-p4835-i4836.service - … --bootstrap-uninstall. 11:25:46 systemd: Stopped infraternal-agent.service ``` ### Rootless `j328t@127.0.0.1:2328`, same build Log `~claude/j328/logs/fix2-rootless-scenario.log`. ``` RESULTS "A_add":true "A_host_clean":true "A_entry_gone":true "B_readd":true "C_still_pending":true "C_host_clean":true "C_entry_gone":true "D_readd":true HOST_CLEAN A-after-remove after 0.4 s · HOST_CLEAN C-after-return after 1.9 s ``` ### Toast (production build, 1440×900) The pending uninstall is no longer an error: it shows the neutral toast with Copy command as the fallback. Screenshot attached (`fix2-rootless-C-remove-offline-toast.png`). The root run's toast screenshot caught a later frame without the toast; its text is in the log above. Not mine: the details panel in the background shows the server's own machine, J325's finding A. Cleanup: tunnel, root helper, test account, sshd and VM are stopped or removed; the claim is released. The final `verify fast` at `fae08b029` is running.

fix2-rootless-C-remove-offline-toast.png

194 KiB

kayg commented

2026-11-07 12:49:10 +00:00

Author

Owner

J328: merged origin/dev; a constraint for D362's implementation; one more load flake

Merged origin/dev (37 commits, 21b3c84ce, clean merge) into the branch: head is now d2472d8c3. The final verify fast is re-queued on that head. I stopped the run on the pre-merge head; its log is ~claude/j328/logs/gate-fae08b029-premerge.log.

Load flake in that pre-merge run: crates/plugins/ssh/tests/previous_attempt_shell.rs → ownership shell oracle failed … AssertionError: 73 != 0 (setup lock busy, exit 73). Its test binary previous_attempt_shell-3b7caef3262a2e08 is byte-identical to the one that passed in my e20c06ec0 gate (test result: ok. 2 passed). The earlier run before it died of No space left on device mid-compile (gate-fae08b029-enospc.log).

For whoever implements D362 (sole-holder revoke ends every process of the account; not in code yet): in the removal outbox, the login revocation comes first (D352 receipt), then the uninstall request. On a rootless host the infraternal agent runs as that account. If D362's "end every remaining process" includes the agent, the agent dies at step 1, and a sole-holder rootless removal never receives its uninstall request. The install would stay, and so would the same re-add failure as #409. Two options:

(a) D362 spares this installation's own agent and its uninstall helper (they are infraternal's own processes, identifiable by unit or cgroup);
(b) when the revocation is part of a removal that will uninstall, the uninstall's own retirement does the "stop everything" step.

My recommendation is (a): it keeps one ordering for every revoke. The owner decides.

## J328: merged origin/dev; a constraint for D362's implementation; one more load flake **Merged origin/dev** (37 commits, `21b3c84ce`, clean merge) into the branch: head is now **`d2472d8c3`**. The final `verify fast` is re-queued on that head. I stopped the run on the pre-merge head; its log is `~claude/j328/logs/gate-fae08b029-premerge.log`. **Load flake in that pre-merge run:** `crates/plugins/ssh/tests/previous_attempt_shell.rs` → `ownership shell oracle failed … AssertionError: 73 != 0` (setup lock busy, exit 73). Its test binary `previous_attempt_shell-3b7caef3262a2e08` is byte-identical to the one that passed in my `e20c06ec0` gate (`test result: ok. 2 passed`). The earlier run before it died of `No space left on device` mid-compile (`gate-fae08b029-enospc.log`). **For whoever implements D362** (sole-holder revoke ends every process of the account; not in code yet): in the removal outbox, the login revocation comes **first** (D352 receipt), then the uninstall request. On a rootless host the infraternal agent runs **as that account**. If D362's "end every remaining process" includes the agent, the agent dies at step 1, and a sole-holder rootless removal never receives its uninstall request. The install would stay, and so would the same re-add failure as #409. Two options: - (a) D362 spares this installation's own agent and its uninstall helper (they are infraternal's own processes, identifiable by unit or cgroup); - (b) when the revocation is part of a removal that will uninstall, the uninstall's own retirement does the "stop everything" step. My recommendation is (a): it keeps one ordering for every revoke. The owner decides.

kayg commented

2026-11-07 17:11:03 +00:00

Author

Owner

J328 finished: branch `fix-remove-uninstall-409`, head `baade609a` (0 behind origin/dev `6b1bdb81b`). Not merged.

Root cause (#409)

Since #285 (6ca0c6a74), the host actor's per-action control allow-list held only the transport.* certificate actions. So remote.host.uninstall got Unsupported inside the server and was never sent. Remove recorded unreachable, committed the revocation (which cut the link: the ~0.3 s BrokenPipe), and left the infraternal agent running. Re-add then failed on .bootstrap-owner.

Commits


`aa564b116`	fix(remote): send Remove's uninstall request to the host. The allow-list arm, plus a red-first test on real TLS.
`5ede0f539`	refactor(remote): the removal outbox alone asks a host to uninstall, the single path proposed above. Order: receipt (D352) → uninstall → one confirming transaction. No network call under the writer lock. A lost answer is asked again on the next reconnect.
`83bd06181`	fix(tracer): a pending uninstall gets a neutral toast, with Copy command as the fallback.
`2db5024f4`	fix(remote): the rootless leftover message said "added with with:" (a bug that exists today).
`fae08b029`	fix(remote): the uninstall helper treats a dead session's socket as retired. Root logins are SIGKILLed by the revocation, so their socket stays behind and the helper refused the whole uninstall (found by the root end-to-end run).
`99ae8b1e6`	test(remote): each outbox test enrolls its own host. The shared fixture's host 42 made parallel tests refuse each other in the process-wide cleanup lane.
`d2472d8c3`, `baade609a`	merges of origin/dev

Red/green

RED1, fix reverted: uninstall_delivery … FAILED at remote_uninstall_delivery_tests.rs:109 ("never sent the uninstall request").
RED2, outbox reverted: all 3 removal_outbox_tests FAILED at :194 ("the server sent no control call").
RED3, helper fix reverted: a_stale_session_socket_counts_as_retired_and_is_removed … FAILED.
GREEN at baade609a: removal_outbox 3/3 passes in parallel, 3 runs in a row; filters uninstall removal cleanup revoke revocation: test result: ok. 129 passed; 0 failed; 5 ignored; plugin-remote uninstall retire: 24 passed; 0 failed.

Real end-to-end (comments above): both PASS on the final build

Rootless: clean 0.4–2.6 s after Remove; re-add OK; the offline host's pending cleanup completes 1.9–2.8 s after it returns; re-add OK.
Root (AlmaLinux 10 VM): clean 3.7 s after Remove; re-add OK; the offline host completes 3.7 s after it returns; re-add OK.

Gate: `scripts/verify fast` at `baade609a`, compute01 (`~claude/j328/logs/final.log`), verbatim

| Rust gate                 | Result       |
+---------------------------+--------------+
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | FAIL (101)   |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| privileged integration    | PASS         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
VERIFY_EXIT=1

The only failing row, test, is not this change. Its one failing suite was crates/plugins/ssh/tests/install.rs (98 passed; 9 failed … 310.70s): 7× "Setup is already running" / exit 73, plus an_owner_record_for_another_host_is_set_up_elsewhere (UnprovenFiles ≠ UnrecognizedInstall). Re-run through build-slot (~claude/j328/logs/devcheck.log), that test passes on unmodified origin/dev and on baade609a:

== 6b1bdb81b Merge fix-mac-login-cloexec …
test an_owner_record_for_another_host_is_set_up_elsewhere ... ok
== baade609a Merge origin/dev into fix-remove-uninstall-409 (#409)
test an_owner_record_for_another_host_is_set_up_elsewhere ... ok

When it failed, compute01 had another job's real root install at /var/lib/infraternal; it is gone now. The exit-73 failures are setup-lock contention on a box running 5 gates at load ~36: the same suite passed 105/105 in 14 s in my e20c06ec0 gate. My earlier gate runs also hit agent_daemon and infraternal-agent quota tests under load, which pass 3/3 alone (logs gate-*.log).

Static sizes (report only, D243): x86_64 27821808 bytes, aarch64 25024200 bytes. The gate's printed delta is against its fixed old baseline. I did not build pristine dev statically for a per-branch delta.

For kayg to decide

The single path (5ede0f539): the uninstall now goes only through the outbox. The one thing lost is the click-time line naming kernel objects the host leaves ("infraternal0 stays because…"). The host still leaves them, unreported. If you would rather take only the one-line fix, aa564b116 stands alone; but then an offline host is never uninstalled, and the receipt race remains.
D362 implementation constraint: the revocation comes before the uninstall. If "end every process of the account" also kills a rootless host's own infraternal agent, a sole-holder removal never gets its uninstall. My recommendation: spare the installation's own agent and helper.
#417 (new): root adds fail on SELinux-enforcing hosts (init_t may not exec var_lib_t).

Cleanup: target dirs deleted; throwaway account, sshd, VM disks and tunnel removed; no claim held. Logs and scripts kept in compute01 ~claude/j328/.

## J328 finished: branch `fix-remove-uninstall-409`, head **`baade609a`** (0 behind origin/dev `6b1bdb81b`). Not merged. ### Root cause (#409) Since #285 (`6ca0c6a74`), the host actor's per-action control allow-list held only the `transport.*` certificate actions. So `remote.host.uninstall` got `Unsupported` inside the server and was never sent. Remove recorded `unreachable`, committed the revocation (which cut the link: the ~0.3 s `BrokenPipe`), and left the infraternal agent running. Re-add then failed on `.bootstrap-owner`. ### Commits | | | |---|---| | `aa564b116` | fix(remote): send Remove's uninstall request to the host. The allow-list arm, plus a red-first test on real TLS. | | `5ede0f539` | refactor(remote): **the removal outbox alone asks a host to uninstall**, the single path proposed above. Order: receipt (D352) → uninstall → one confirming transaction. No network call under the writer lock. A lost answer is asked again on the next reconnect. | | `83bd06181` | fix(tracer): a pending uninstall gets a neutral toast, with Copy command as the fallback. | | `2db5024f4` | fix(remote): the rootless leftover message said "added with **with**:" (a bug that exists today). | | `fae08b029` | fix(remote): the uninstall helper treats a dead session's socket as retired. Root logins are SIGKILLed by the revocation, so their socket stays behind and the helper refused the whole uninstall (found by the root end-to-end run). | | `99ae8b1e6` | test(remote): each outbox test enrolls its own host. The shared fixture's host 42 made parallel tests refuse each other in the process-wide cleanup lane. | | `d2472d8c3`, `baade609a` | merges of origin/dev | ### Red/green - RED1, fix reverted: `uninstall_delivery … FAILED` at `remote_uninstall_delivery_tests.rs:109` ("never sent the uninstall request"). - RED2, outbox reverted: all 3 `removal_outbox_tests` FAILED at `:194` ("the server sent no control call"). - RED3, helper fix reverted: `a_stale_session_socket_counts_as_retired_and_is_removed … FAILED`. - GREEN at `baade609a`: `removal_outbox` 3/3 passes **in parallel**, 3 runs in a row; filters `uninstall removal cleanup revoke revocation`: `test result: ok. 129 passed; 0 failed; 5 ignored`; plugin-remote `uninstall retire`: `24 passed; 0 failed`. ### Real end-to-end (comments above): both PASS on the final build - Rootless: clean 0.4–2.6 s after Remove; re-add OK; the offline host's pending cleanup completes 1.9–2.8 s after it returns; re-add OK. - Root (AlmaLinux 10 VM): clean 3.7 s after Remove; re-add OK; the offline host completes 3.7 s after it returns; re-add OK. ### Gate: `scripts/verify fast` at `baade609a`, compute01 (`~claude/j328/logs/final.log`), verbatim ``` | Rust gate | Result | +---------------------------+--------------+ | Nix Cargo vendoring | PASS | | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | FAIL (101) | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | PASS | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | static builds | PASS | +---------------------------+--------------+ VERIFY_EXIT=1 ``` **The only failing row, `test`, is not this change.** Its one failing suite was `crates/plugins/ssh/tests/install.rs` (`98 passed; 9 failed … 310.70s`): 7× "Setup is already running" / exit 73, plus `an_owner_record_for_another_host_is_set_up_elsewhere` (`UnprovenFiles` ≠ `UnrecognizedInstall`). Re-run through build-slot (`~claude/j328/logs/devcheck.log`), that test passes on **unmodified origin/dev and on `baade609a`**: ``` == 6b1bdb81b Merge fix-mac-login-cloexec … test an_owner_record_for_another_host_is_set_up_elsewhere ... ok == baade609a Merge origin/dev into fix-remove-uninstall-409 (#409) test an_owner_record_for_another_host_is_set_up_elsewhere ... ok ``` When it failed, compute01 had another job's real root install at `/var/lib/infraternal`; it is gone now. The exit-73 failures are setup-lock contention on a box running 5 gates at load ~36: the same suite passed `105/105` in 14 s in my `e20c06ec0` gate. My earlier gate runs also hit `agent_daemon` and `infraternal-agent` quota tests under load, which pass 3/3 alone (logs `gate-*.log`). Static sizes (report only, D243): x86_64 `27821808` bytes, aarch64 `25024200` bytes. The gate's printed delta is against its fixed old baseline. I did not build pristine dev statically for a per-branch delta. ### For kayg to decide 1. **The single path** (`5ede0f539`): the uninstall now goes only through the outbox. The one thing lost is the click-time line naming kernel objects the host leaves ("infraternal0 stays because…"). The host still leaves them, unreported. If you would rather take only the one-line fix, `aa564b116` stands alone; but then an offline host is never uninstalled, and the receipt race remains. 2. **D362 implementation constraint:** the revocation comes before the uninstall. If "end every process of the account" also kills a rootless host's own infraternal agent, a sole-holder removal never gets its uninstall. My recommendation: spare the installation's own agent and helper. 3. **#417 (new):** root adds fail on SELinux-enforcing hosts (`init_t` may not exec `var_lib_t`). Cleanup: target dirs deleted; throwaway account, sshd, VM disks and tunnel removed; no claim held. Logs and scripts kept in compute01 `~claude/j328/`.

kayg commented

2026-11-07 17:20:18 +00:00

Author

Owner

J351 independent correctness/security review starts. Review branch review-J351 pinned to fix-remove-uninstall-409 head baade609ae; base origin/dev 6b1bdb81b3. Plan (~75 minutes): read decisions and #285/#409; inspect identity binding, outbox ordering, caller authorization and socket cleanup; run focused removal/cleanup/revocation tests on dev-02 through checked build-slot launcher; attempt isolated rootless end-to-end if practical; post APPROVE or REQUEST CHANGES with evidence. Per common brief reviewer rule, read author gate receipt rather than rerunning quick/fast. No merge, deploy or push to dev.

J351 independent correctness/security review starts. Review branch review-J351 pinned to fix-remove-uninstall-409 head baade609ae4bf50bdb93558a3a54fb4bbccba672; base origin/dev 6b1bdb81b3d8eb90716643e7d1bd03f61ce37827. Plan (~75 minutes): read decisions and #285/#409; inspect identity binding, outbox ordering, caller authorization and socket cleanup; run focused removal/cleanup/revocation tests on dev-02 through checked build-slot launcher; attempt isolated rootless end-to-end if practical; post APPROVE or REQUEST CHANGES with evidence. Per common brief reviewer rule, read author gate receipt rather than rerunning quick/fast. No merge, deploy or push to dev.

kayg commented

2026-11-07 17:26:28 +00:00

Author

Owner

J351 finding P3: the pending-uninstall toast is not neutral yet. host-removal.on-demand.ts:59 omits tone when asking, but packages/ui/src/toast.ts:375-382 defaults that to ok and the success tick unless an icon is supplied. The production screenshot attached to comment 84420 visibly shows a green check while uninstall is still pending: https://git.kayg.org/attachments/d46fed6d-bb2e-4ce4-a901-1a4b06dc8d33 . TASTE ("The icon matches the news") calls for an info mark for neutral news. Keep the Copy command action and set the existing shared neutral/info icon on the asking branch. The requested copy and the "with with" repair are otherwise present. Nonblocking polish finding; correctness/security review and focused dev-02 tests continue.

J351 finding P3: the pending-uninstall toast is not neutral yet. `host-removal.on-demand.ts:59` omits tone when asking, but `packages/ui/src/toast.ts:375-382` defaults that to `ok` and the success tick unless an `icon` is supplied. The production screenshot attached to comment 84420 visibly shows a green check while uninstall is still pending: https://git.kayg.org/attachments/d46fed6d-bb2e-4ce4-a901-1a4b06dc8d33 . TASTE ("The icon matches the news") calls for an info mark for neutral news. Keep the Copy command action and set the existing shared neutral/info icon on the asking branch. The requested copy and the "with with" repair are otherwise present. Nonblocking polish finding; correctness/security review and focused dev-02 tests continue.

kayg commented

2026-11-07 17:32:33 +00:00

Author

Owner

J351 P2 finding (the trade-off J328 disclosed in 82885/84890): the outbox discards the accepted uninstall response's left list in remote_runtime.rs:132, so a normal UUID-owner removal no longer reports deliberately retained kernel objects. Previously request_uninstall passed this list to outcome, which printed the allow-listed object names. This is an observability regression, not an ownership/deletion escape: the helper still preserves objects it did not create. Keep the outbox ordering, but retain and expose the allow-listed left report asynchronously rather than choosing between a safe outbox and the report. Nonblocking under D206; explicitly recorded so the loss is not mistaken for feature parity with the one-line fix. Also update contracts/remote.md:483 and its finish-result description to match the asynchronous outbox and pending cleanup state.

J351 P2 finding (the trade-off J328 disclosed in 82885/84890): the outbox discards the accepted uninstall response's `left` list in `remote_runtime.rs:132`, so a normal UUID-owner removal no longer reports deliberately retained kernel objects. Previously `request_uninstall` passed this list to `outcome`, which printed the allow-listed object names. This is an observability regression, not an ownership/deletion escape: the helper still preserves objects it did not create. Keep the outbox ordering, but retain and expose the allow-listed `left` report asynchronously rather than choosing between a safe outbox and the report. Nonblocking under D206; explicitly recorded so the loss is not mistaken for feature parity with the one-line fix. Also update `contracts/remote.md:483` and its finish-result description to match the asynchronous outbox and `pending` cleanup state.

kayg commented

2026-11-07 17:34:00 +00:00

Author

Owner

J351 independent review: APPROVE

Reviewed fix-remove-uninstall-409 at baade609ae, against origin/dev 6b1bdb81b3. No P0/P1 findings. P2/P3 below are nonblocking under D206; this is a code-review approval, not a claim that the merge gate is green.

Findings

P2: the outbox loses the retained-kernel-object report, as J328 already disclosed. remote_runtime.rs matches Accepted(_), dropping the allow-listed names that the former immediate path passed into outcome. The helper still preserves unrelated objects; the loss is the owner's report. Retain/expose that result asynchronously. Finding comment 84925. The contract at contracts/remote.md:483 also still describes a network handoff inside the writer transaction and omits the new pending cleanup state; reconcile it with the chosen flow.
P3: pending uninstall still gets a green success tick. Omitting tone defaults to ok; it does not choose the shared toast's neutral icon. Set its existing info icon for the pending branch. The production screenshot on comment 84420 visibly reproduces it; finding comment 84915. Copy command is present, and both rootless “with with” messages are repaired.

Correctness and security

Keep the outbox, rather than taking only aa564b116. The one-line fix restores online delivery but still leaves offline hosts installed and lets the uninstall stop the daemon before its next-connection revocation receipt. The redesign uses the existing durable tables and cleanup admission, avoids network work under SQLite's writer lock for real UUID accounts, and implements D134/D352 more completely. The non-UUID legacy/test path remains immediate, so “one path” has that explicit exception.

The order is revoke → durable daemon reply → uninstall → one SQLite transaction recording acceptance/refusal, confirming the receipt and retiring the entry. Before the receipt, or after it with no uninstall answer, nothing advances and a surviving installation retries on reconnect. Reconstruction of a fresh server runtime reads the same retained SQLite intent. Duplicate confirmation cannot change an already-recorded acceptance. SQLite rolls back an interrupted confirming transaction. After commit the exact old certificate no longer obtains cleanup admission.

Important limit: a successfully written acceptance can arm the helper even if the server never receives it or crashes before committing. If the helper then completes, that old installation cannot reconnect to finish the ledger, and the entry can remain pending. Do not claim unconditional completion after every lost reply. This does not authorize new work or threaten a replacement installation; D355's subsequent no-install/Start-fresh proof provides reconciliation. requested is acceptance, not proof that the helper completed. Expired/individually revoked certificates also remain refused by existing cleanup admission; this patch does not add indefinite offline recovery.

A stale uninstall cannot target a fresh installation through this path. Admission binds the exact certificate fingerprint and proof to the host UUID; authenticate checks the proof's host matches admitted authority. Removal records are keyed by that UUID, not address. Bootstrap explicitly excludes removed identities from replacement/re-enrollment as that same identity (bootstrap_remote::check_stuck_install); fresh installs receive a new UUID. The uninstall handler compares the request against its local context and validates .bootstrap-owner. The independent helper takes the shared setup lock and revalidates that same identity before stopping anything. Old manual commands are bound the same way. My added test sends an old control request to a different installation and verifies refusal, intact artifacts and no armed plan.

Caller scope is correct. control_call is crate-private, and production uninstall calls originate in removal. UNINSTALL is absent from the server/browser descriptor and has api_scope: None; widening the actor's wire allow-list does not make raw uninstall key-callable. A properly scoped key can still request the separately authorized, dry-run-bound Remove action, as D333 explicitly requires. Cleanup links are private actors, unpublished to normal routing, and their write filter admits only stream-0 liveness, revoke and uninstall.

fae08b029 is safe within the installation's private session directory. Cleanup first validates the installation and the private, same-UID sessions directory. On a refused/missing endpoint it uses symlink_metadata, refuses non-sockets and foreign owners, and unlinks only the local socket entry. A regular file, symlink or foreign-UID socket is preserved. This does not delete a socket merely because its filename resembles ours elsewhere. Existing root E2E evidence shows D354's killed login unit leaving precisely this stale socket. D353/D356 revocation still precedes uninstall; J328's D362 constraint remains real: future sole-holder process sweeping must allow this installation to complete its uninstall handoff.

Independent validation and artifacts

Launcher checked outside the slot: nix develop ~/worktrees/J351 -c cargo --version → cargo 1.99.0 (5f94df478 2026-08-27). Ran on dev-02, CARGO_INCREMENTAL=0, one target, -j3, through ~/infraternal-ops/build-slot nice -n10 nix develop ~/worktrees/J351 -c bash .../run-focused.sh:

cargo test -j3 -p infraternal-server --features remote --lib -- uninstall removal cleanup revoke revocation
test result: ok. 131 passed; 0 failed; 5 ignored; 0 measured; 878 filtered out
SERVER_TEST_EXIT=0
cargo test -j3 -p infraternal-plugin-remote --features agent,tokio/rt-multi-thread --lib -- uninstall retire
test result: ok. 27 passed; 0 failed; 0 ignored; 0 measured; 109 filtered out
PLUGIN_TEST_EXIT=0

Attached focused.log contains the full unedited output. Five added independent tests cover pre-receipt disconnect, new-runtime reconstruction plus duplicate confirmation, stale installation identity, foreign UID and symlink guards. The reconstruction test uses a new EnrollmentStore/runtime over the same SQLite pool; it is not a process-kill E2E. Review tests only, pushed on review-J351 at 1edb120458; attached patch. The five ignored tests require configured release hosts, exclusive ports or scale/measurement environments; they were not silently counted as passes.

A fresh real rootless E2E was not rerun: compute01 had all five slots occupied and four queued jobs (attached capacity receipt), and recreating the root account/sshd/browser fixture was not practical for this review. I read the actual ~claude/j328/logs/fix2-rootless-scenario.log and fix2-root-scenario.log, not only the finish comment: both record successful online removal, re-add, offline return cleanup and second re-add. Those are J328's fae08b029 release receipts, not a fresh baade609a E2E claim. The affected removal/uninstall/toast production files are unchanged between that release and the reviewed head. Screenshot: https://git.kayg.org/attachments/d46fed6d-bb2e-4ce4-a901-1a4b06dc8d33 .

Author's gate, verified from the actual log

Per the common brief's explicit reviewer rule, I did not rerun quick/fast. ~claude/j328/logs/final.log at baade609a ends with the following verbatim table:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | FAIL (101)   |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| privileged integration    | PASS         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
VERIFY_EXIT=1

It is a failed gate, not a green one. The failure is the SSH install suite (98 passed, 9 failed), with setup-lock refusals and the owner-record classification case. I also read devcheck.log: that classification test passes on both pristine origin/dev and baade609a. The remaining contention attribution is the author's evidence, not an independent full-suite baseline rerun here. The orchestrator still needs the merge gate to pass.

Evidence retained locally under /home/kayg/.cache/codex-scratch/J351/evidence/ and on dev-02 at /home/claude/.cache/codex-scratch/J351/evidence/focused.log. Own dev-02 target deleted; no own processes, accounts, units, servers or mesh claim remain. No merge or deployment; no push to dev.

Verdict: APPROVE. Reviewed head: baade609ae.

Verdict: APPROVE; head: baade609ae; comment id: 84927.

## J351 independent review: APPROVE Reviewed `fix-remove-uninstall-409` at **baade609ae4bf50bdb93558a3a54fb4bbccba672**, against `origin/dev` **6b1bdb81b3d8eb90716643e7d1bd03f61ce37827**. No P0/P1 findings. P2/P3 below are nonblocking under D206; this is a code-review approval, not a claim that the merge gate is green. ### Findings - **P2: the outbox loses the retained-kernel-object report**, as J328 already disclosed. `remote_runtime.rs` matches `Accepted(_)`, dropping the allow-listed names that the former immediate path passed into `outcome`. The helper still preserves unrelated objects; the loss is the owner's report. Retain/expose that result asynchronously. Finding comment 84925. The contract at `contracts/remote.md:483` also still describes a network handoff inside the writer transaction and omits the new `pending` cleanup state; reconcile it with the chosen flow. - **P3: pending uninstall still gets a green success tick.** Omitting tone defaults to `ok`; it does not choose the shared toast's neutral icon. Set its existing info icon for the pending branch. The production screenshot on comment 84420 visibly reproduces it; finding comment 84915. Copy command is present, and both rootless “with with” messages are repaired. ### Correctness and security **Keep the outbox, rather than taking only aa564b116.** The one-line fix restores online delivery but still leaves offline hosts installed and lets the uninstall stop the daemon before its next-connection revocation receipt. The redesign uses the existing durable tables and cleanup admission, avoids network work under SQLite's writer lock for real UUID accounts, and implements D134/D352 more completely. The non-UUID legacy/test path remains immediate, so “one path” has that explicit exception. The order is revoke → durable daemon reply → uninstall → one SQLite transaction recording acceptance/refusal, confirming the receipt and retiring the entry. Before the receipt, or after it with no uninstall answer, nothing advances and a surviving installation retries on reconnect. Reconstruction of a fresh server runtime reads the same retained SQLite intent. Duplicate confirmation cannot change an already-recorded acceptance. SQLite rolls back an interrupted confirming transaction. After commit the exact old certificate no longer obtains cleanup admission. **Important limit:** a successfully written acceptance can arm the helper even if the server never receives it or crashes before committing. If the helper then completes, that old installation cannot reconnect to finish the ledger, and the entry can remain pending. Do not claim unconditional completion after every lost reply. This does not authorize new work or threaten a replacement installation; D355's subsequent no-install/Start-fresh proof provides reconciliation. `requested` is acceptance, not proof that the helper completed. Expired/individually revoked certificates also remain refused by existing cleanup admission; this patch does not add indefinite offline recovery. **A stale uninstall cannot target a fresh installation through this path.** Admission binds the exact certificate fingerprint and proof to the host UUID; `authenticate` checks the proof's host matches admitted authority. Removal records are keyed by that UUID, not address. Bootstrap explicitly excludes removed identities from replacement/re-enrollment as that same identity (`bootstrap_remote::check_stuck_install`); fresh installs receive a new UUID. The uninstall handler compares the request against its local context and validates `.bootstrap-owner`. The independent helper takes the shared setup lock and revalidates that same identity before stopping anything. Old manual commands are bound the same way. My added test sends an old control request to a different installation and verifies refusal, intact artifacts and no armed plan. **Caller scope is correct.** `control_call` is crate-private, and production uninstall calls originate in removal. UNINSTALL is absent from the server/browser descriptor and has `api_scope: None`; widening the actor's wire allow-list does not make raw uninstall key-callable. A properly scoped key can still request the separately authorized, dry-run-bound Remove action, as D333 explicitly requires. Cleanup links are private actors, unpublished to normal routing, and their write filter admits only stream-0 liveness, revoke and uninstall. **fae08b029 is safe within the installation's private session directory.** Cleanup first validates the installation and the private, same-UID sessions directory. On a refused/missing endpoint it uses `symlink_metadata`, refuses non-sockets and foreign owners, and unlinks only the local socket entry. A regular file, symlink or foreign-UID socket is preserved. This does not delete a socket merely because its filename resembles ours elsewhere. Existing root E2E evidence shows D354's killed login unit leaving precisely this stale socket. D353/D356 revocation still precedes uninstall; J328's D362 constraint remains real: future sole-holder process sweeping must allow this installation to complete its uninstall handoff. ### Independent validation and artifacts Launcher checked outside the slot: `nix develop ~/worktrees/J351 -c cargo --version` → `cargo 1.99.0 (5f94df478 2026-08-27)`. Ran on dev-02, CARGO_INCREMENTAL=0, one target, -j3, through `~/infraternal-ops/build-slot nice -n10 nix develop ~/worktrees/J351 -c bash .../run-focused.sh`: ``` cargo test -j3 -p infraternal-server --features remote --lib -- uninstall removal cleanup revoke revocation test result: ok. 131 passed; 0 failed; 5 ignored; 0 measured; 878 filtered out SERVER_TEST_EXIT=0 cargo test -j3 -p infraternal-plugin-remote --features agent,tokio/rt-multi-thread --lib -- uninstall retire test result: ok. 27 passed; 0 failed; 0 ignored; 0 measured; 109 filtered out PLUGIN_TEST_EXIT=0 ``` Attached `focused.log` contains the full unedited output. Five added independent tests cover pre-receipt disconnect, new-runtime reconstruction plus duplicate confirmation, stale installation identity, foreign UID and symlink guards. The reconstruction test uses a new EnrollmentStore/runtime over the same SQLite pool; it is not a process-kill E2E. Review tests only, pushed on `review-J351` at **1edb12045857d378b68d0ece9814a59c1cc5b1f2**; attached patch. The five ignored tests require configured release hosts, exclusive ports or scale/measurement environments; they were not silently counted as passes. A fresh real rootless E2E was **not rerun**: compute01 had all five slots occupied and four queued jobs (attached capacity receipt), and recreating the root account/sshd/browser fixture was not practical for this review. I read the actual `~claude/j328/logs/fix2-rootless-scenario.log` and `fix2-root-scenario.log`, not only the finish comment: both record successful online removal, re-add, offline return cleanup and second re-add. Those are J328's **fae08b029** release receipts, not a fresh baade609a E2E claim. The affected removal/uninstall/toast production files are unchanged between that release and the reviewed head. Screenshot: https://git.kayg.org/attachments/d46fed6d-bb2e-4ce4-a901-1a4b06dc8d33 . ### Author's gate, verified from the actual log Per the common brief's explicit reviewer rule, I did not rerun quick/fast. `~claude/j328/logs/final.log` at baade609a ends with the following **verbatim** table: ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | Nix Cargo vendoring | PASS | | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | FAIL (101) | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | PASS | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | static builds | PASS | +---------------------------+--------------+ VERIFY_EXIT=1 ``` It is a failed gate, not a green one. The failure is the SSH install suite (98 passed, 9 failed), with setup-lock refusals and the owner-record classification case. I also read `devcheck.log`: that classification test passes on both pristine origin/dev and baade609a. The remaining contention attribution is the author's evidence, not an independent full-suite baseline rerun here. The orchestrator still needs the merge gate to pass. Evidence retained locally under `/home/kayg/.cache/codex-scratch/J351/evidence/` and on dev-02 at `/home/claude/.cache/codex-scratch/J351/evidence/focused.log`. Own dev-02 target deleted; no own processes, accounts, units, servers or mesh claim remain. No merge or deployment; no push to dev. **Verdict: APPROVE. Reviewed head: baade609ae4bf50bdb93558a3a54fb4bbccba672.** Verdict: APPROVE; head: baade609ae4bf50bdb93558a3a54fb4bbccba672; comment id: 84927.

focused.log

32 KiB

review-tests.patch

6.4 KiB

compute-capacity.txt

944 B

author-gate-table.txt

1.2 KiB

kayg commented

2026-11-07 20:08:15 +00:00

Author

Owner

J328: J351's P2 and P3 done; head `863c810d1` (gate green)

Branch fix-remove-uninstall-409, on top of the approved baade609a; 42 commits behind origin/dev 6ffe37e06. Not merged. One commit each:

091751101 P3 fix(tracer): the pending-uninstall toast passes the shared info icon (ICON_INFO, the "Not saved" mark) instead of falling back to the default ok tick. The spec gains a case at 1440 and 393: data-icon="true", not the error tone, the info glyph's path, the tick marker hidden, and Copy command present.

863c810d1 P2 fix(remote): the outbox keeps the host's retained-kernel-object report. New migration 20261107173825_removal_retained_objects adds remote_host_removals.retained (a JSON array of server-known names; one known_left() filter covers the live reply and the stored copy). It is written in the same transaction as the answer and the receipt. When the host was connected at the click, Remove waits up to 5 s after its commit (outside the writer lock) for that transaction, then answers removed with "…Left infraternal0 because infraternal didn't create it here." and left for scripts. The CLI prints it; a new CLI test covers human and machine output. A slower or offline host gets the pending answer at once. contracts/remote.md now describes the outbox flow, and that requested means accepted, not finished (J351's note).


`091751101`	P3 fix(tracer): the pending-uninstall toast passes the shared info icon (`ICON_INFO`, the "Not saved" mark) instead of falling back to the default `ok` tick. The spec gains a case at 1440 and 393: `data-icon="true"`, not the error tone, the info glyph's path, the tick marker hidden, and Copy command present.
`863c810d1`	P2 fix(remote): the outbox keeps the host's retained-kernel-object report. New migration `20261107173825_removal_retained_objects` adds `remote_host_removals.retained` (a JSON array of server-known names; one `known_left()` filter covers the live reply and the stored copy). It is written in the same transaction as the answer and the receipt. When the host was connected at the click, Remove waits up to 5 s after its commit (outside the writer lock) for that transaction, then answers `removed` with "…Left infraternal0 because infraternal didn't create it here." and `left` for scripts. The CLI prints it; a new CLI test covers human and machine output. A slower or offline host gets the pending answer at once. `contracts/remote.md` now describes the outbox flow, and that `requested` means accepted, not finished (J351's note).

Real environment (release build of `863c810d1`, throwaway server, mesh claim held and released)

Root (AlmaLinux 10 VM, permissive for #417). A plain dummy infraternal0 was created after add, so the install did not own it. Log: ~claude/j328/logs/p2-root-scenario.log.

18:37:39.877Z A-remove REMOVE_CLICKED
18:37:40.513Z A-remove TOAST 127.0.0.1:2338 was removed. Its infraternal agent is uninstalling itself. Left infraternal0 because infraternal didn't create it here.
18:37:42.115Z HOST_CLEAN A-after-remove after 1.0 s
remote_host_removals: [('fd2d226e…', 1, 'requested', '["infraternal0"]')]
VM after removal: infraternal0 still present (untouched); /var/lib/infraternal gone

Rootless, all four phases. Log: ~claude/j328/logs/p2-rootless-scenario.log.

A-remove TOAST 127.0.0.1:2328 was removed. Its infraternal agent is uninstalling itself.   (connected: completed answer)
HOST_CLEAN A-after-remove after 0.6 s · C (offline) pending toast · HOST_CLEAN C-after-return after 2.4 s
RESULTS A_host_clean, A_entry_gone, B_readd, C_still_pending, C_host_clean, C_entry_gone, D_readd: all true

Spec on the production tracer build (pending-removal-command.spec.ts, compute01, ~claude/j328/logs/spec.log): 4 passed (1.2m). The 393 and 1440 neutral screenshots and the root result are attached.

Tests

On dev-02, outbox tests in parallel ×3: 4 passed; 0 failed each time (including the new an_unanswered_removal_wait_ends_without_an_outcome). The accept test also checks that only infraternal0 is stored from ["infraternal0","run me"] and that a concurrent waiter receives it. Filters uninstall removal cleanup revoke revocation: 130 passed; 0 failed; 5 ignored (~claude/j328/t2.log on dev-02).

`verify quick`, dev-02 (aarch64), `863c810d1`

| Rust gate                 | Result       |
+---------------------------+--------------+
| scope guard               | PASS         |
| migration names           | PASS         |
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| clippy (workspace)        | PASS         |
| unit tests (changed)      | PASS         |
| unit tests (changed, bin-only) | PASS         |
| ui literals               | PASS         |
| tracer typecheck          | PASS         |
+---------------------------+--------------+

QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done.
QUICK_EXIT=0

(The two earlier quick runs found a clippy needless_pass_by_value in known_left and a CLI test that didn't allow for line wrapping. Both were fixed in the P2 commit before this run.)

`verify fast`, compute01, `863c810d1` (`~claude/j328/logs/gate-863c810d1.log`), verbatim

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| privileged integration    | PASS         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
VERIFY_EXIT=0
PT_INTERP: absent; size: 27832144 bytes; baseline 22418256; delta +5413888 bytes (report only, D243 amended 2026-11-01)
PT_INTERP: absent; size: 25034312 bytes; baseline 20453280; delta +4581032 bytes (report only, D243 amended 2026-11-01)

The SSH install suite passed this time, so no reruns were needed. Static sizes (report only): x86_64 27832144 B, aarch64 25034312 B, against 27821808 / 25024200 at baade609a. That is +10336 B / +10112 B for P2+P3.

Cleanup: compute01 and dev-02 targets deleted; VM disks, throwaway account, sshd and tunnel removed; no claim held.

## J328: J351's P2 and P3 done; head **`863c810d1`** (gate green) Branch `fix-remove-uninstall-409`, on top of the approved `baade609a`; 42 commits behind origin/dev `6ffe37e06`. Not merged. One commit each: | | | |---|---| | `091751101` | **P3** fix(tracer): the pending-uninstall toast passes the shared info icon (`ICON_INFO`, the "Not saved" mark) instead of falling back to the default `ok` tick. The spec gains a case at 1440 and 393: `data-icon="true"`, not the error tone, the info glyph's path, the tick marker hidden, and Copy command present. | | `863c810d1` | **P2** fix(remote): the outbox keeps the host's retained-kernel-object report. New migration `20261107173825_removal_retained_objects` adds `remote_host_removals.retained` (a JSON array of server-known names; one `known_left()` filter covers the live reply and the stored copy). It is written in the same transaction as the answer and the receipt. When the host was connected at the click, Remove waits up to 5 s **after its commit** (outside the writer lock) for that transaction, then answers `removed` with "…Left infraternal0 because infraternal didn't create it here." and `left` for scripts. The CLI prints it; a new CLI test covers human and machine output. A slower or offline host gets the pending answer at once. `contracts/remote.md` now describes the outbox flow, and that `requested` means accepted, not finished (J351's note). | ### Real environment (release build of `863c810d1`, throwaway server, mesh claim held and released) - **Root** (AlmaLinux 10 VM, permissive for #417). A plain `dummy` `infraternal0` was created after add, so the install did not own it. Log: `~claude/j328/logs/p2-root-scenario.log`. ``` 18:37:39.877Z A-remove REMOVE_CLICKED 18:37:40.513Z A-remove TOAST 127.0.0.1:2338 was removed. Its infraternal agent is uninstalling itself. Left infraternal0 because infraternal didn't create it here. 18:37:42.115Z HOST_CLEAN A-after-remove after 1.0 s remote_host_removals: [('fd2d226e…', 1, 'requested', '["infraternal0"]')] VM after removal: infraternal0 still present (untouched); /var/lib/infraternal gone ``` - **Rootless**, all four phases. Log: `~claude/j328/logs/p2-rootless-scenario.log`. ``` A-remove TOAST 127.0.0.1:2328 was removed. Its infraternal agent is uninstalling itself. (connected: completed answer) HOST_CLEAN A-after-remove after 0.6 s · C (offline) pending toast · HOST_CLEAN C-after-return after 2.4 s RESULTS A_host_clean, A_entry_gone, B_readd, C_still_pending, C_host_clean, C_entry_gone, D_readd: all true ``` - **Spec on the production tracer build** (`pending-removal-command.spec.ts`, compute01, `~claude/j328/logs/spec.log`): `4 passed (1.2m)`. The 393 and 1440 neutral screenshots and the root result are attached. ### Tests On dev-02, outbox tests in parallel ×3: `4 passed; 0 failed` each time (including the new `an_unanswered_removal_wait_ends_without_an_outcome`). The accept test also checks that only `infraternal0` is stored from `["infraternal0","run me"]` and that a concurrent waiter receives it. Filters `uninstall removal cleanup revoke revocation`: `130 passed; 0 failed; 5 ignored` (`~claude/j328/t2.log` on dev-02). ### `verify quick`, dev-02 (aarch64), `863c810d1` ``` | Rust gate | Result | +---------------------------+--------------+ | scope guard | PASS | | migration names | PASS | | Nix Cargo vendoring | PASS | | fmt | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | clippy (workspace) | PASS | | unit tests (changed) | PASS | | unit tests (changed, bin-only) | PASS | | ui literals | PASS | | tracer typecheck | PASS | +---------------------------+--------------+ QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done. QUICK_EXIT=0 ``` (The two earlier quick runs found a clippy `needless_pass_by_value` in `known_left` and a CLI test that didn't allow for line wrapping. Both were fixed in the P2 commit before this run.) ### `verify fast`, compute01, `863c810d1` (`~claude/j328/logs/gate-863c810d1.log`), verbatim ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | Nix Cargo vendoring | PASS | | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | PASS | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | static builds | PASS | +---------------------------+--------------+ VERIFY_EXIT=0 PT_INTERP: absent; size: 27832144 bytes; baseline 22418256; delta +5413888 bytes (report only, D243 amended 2026-11-01) PT_INTERP: absent; size: 25034312 bytes; baseline 20453280; delta +4581032 bytes (report only, D243 amended 2026-11-01) ``` The SSH install suite passed this time, so no reruns were needed. Static sizes (report only): x86_64 27832144 B, aarch64 25034312 B, against 27821808 / 25024200 at `baade609a`. That is **+10336 B / +10112 B** for P2+P3. Cleanup: compute01 and dev-02 targets deleted; VM disks, throwaway account, sshd and tunnel removed; no claim held.

393-neutral.png

152 KiB

1440-neutral.png

165 KiB

p2-root-A-remove-toast.png

202 KiB

kayg referenced this issue

2026-11-08 04:42:20 +00:00

J300: host-removal confirmation is badly designed (wrong anchor, sentence button, legible glass) #391

kayg referenced this issue

2026-11-08 14:24:16 +00:00

doc.js over its size budget on dev #428

kayg referenced this issue

2026-11-08 16:59:54 +00:00

dev: core-screen #rtt and platform-consistency specs fail #430

kayg referenced this issue

2026-11-08 18:39:03 +00:00

dev: core-screen #rtt and platform-consistency specs fail #430

kayg referenced this issue

2026-11-08 18:55:37 +00:00

Gate flakes, round 2 #355

kayg referenced this issue

2026-11-08 19:07:00 +00:00

Gate flakes, round 2 #355

kayg referenced this issue

2026-11-08 22:45:16 +00:00

Gate flakes, round 2 #355

kayg referenced this issue

2026-11-08 22:59:47 +00:00

Gate flakes, round 2 #355

kayg referenced this issue

2026-11-08 23:11:54 +00:00

Gate flakes, round 2 #355

kayg referenced this issue

2026-11-09 05:05:52 +00:00

Gate flakes, round 2 #355

kayg referenced this issue

2026-11-09 05:52:10 +00:00

Gate flakes, round 2 #355

kayg referenced this issue

2026-11-09 07:01:53 +00:00

Gate flakes, round 2 #355

kayg referenced this issue

2026-11-09 12:22:57 +00:00

Gate flakes, round 2 #355

kayg referenced this issue

2026-11-09 14:36:00 +00:00

Gate flakes, round 2 #355

kayg referenced this issue

2026-11-09 18:51:54 +00:00

Gate flakes, round 2 #355

Rows
Columns

Remove leaves the infraternal agent running: uninstall never starts on the host #409

J328 start

J328 finding 1: root cause. The server never sends the uninstall (regression from #285, 6ca0c6a74, 2026-11-02)

J328 finding 2: the removal flow has three overlapping paths, and none owns the uninstall end to end

Proposal: one path, the outbox

J328 progress (21:20 CET)

J328 paused for compute01 maintenance (02:15–02:45 IST)

J328 progress (04:45 IST)

J328 progress (06:30 IST)

J328: red/green proven; gate queued but blocked on compute01 disk

J328: first full gate at e20c06ec0, and a fix for a test that failed under load

J328 verify fast at 2db5024f4 (compute01; log ~claude/j328/logs/gate-2db5024f4.log)

J328 end-to-end, first pass: rootless passes all four steps; root found two more problems

Rootless j328t@127.0.0.1:2328: PASS

Root root@127.0.0.1:2338 (throwaway AlmaLinux 10 qemu VM): two problems

J328 progress (15:10 IST)

J328 end-to-end proof on the final build: rootless and root both PASS

Root root@127.0.0.1:2338: throwaway AlmaLinux 10 VM (SELinux set permissive only because of #417)

Rootless j328t@127.0.0.1:2328, same build

Toast (production build, 1440×900)

J328: merged origin/dev; a constraint for D362's implementation; one more load flake

J328 finished: branch fix-remove-uninstall-409, head baade609a (0 behind origin/dev 6b1bdb81b). Not merged.

Root cause (#409)

Commits

Red/green

Real end-to-end (comments above): both PASS on the final build

Gate: scripts/verify fast at baade609a, compute01 (~claude/j328/logs/final.log), verbatim

For kayg to decide

J351 independent review: APPROVE

Findings

Correctness and security

Independent validation and artifacts

Author's gate, verified from the actual log

J328: J351's P2 and P3 done; head 863c810d1 (gate green)

Real environment (release build of 863c810d1, throwaway server, mesh claim held and released)

Tests

verify quick, dev-02 (aarch64), 863c810d1

verify fast, compute01, 863c810d1 (~claude/j328/logs/gate-863c810d1.log), verbatim

J328 finding 1: root cause. The server never sends the uninstall (regression from #285, `6ca0c6a74`, 2026-11-02)

J328: first full gate at `e20c06ec0`, and a fix for a test that failed under load

J328 `verify fast` at `2db5024f4` (compute01; log `~claude/j328/logs/gate-2db5024f4.log`)

Rootless `j328t@127.0.0.1:2328`: PASS

Root `root@127.0.0.1:2338` (throwaway AlmaLinux 10 qemu VM): two problems

Root `root@127.0.0.1:2338`: throwaway AlmaLinux 10 VM (SELinux set permissive only because of #417)

Rootless `j328t@127.0.0.1:2328`, same build

J328 finished: branch `fix-remove-uninstall-409`, head `baade609a` (0 behind origin/dev `6b1bdb81b`). Not merged.

Gate: `scripts/verify fast` at `baade609a`, compute01 (`~claude/j328/logs/final.log`), verbatim

J328: J351's P2 and P3 done; head `863c810d1` (gate green)

Real environment (release build of `863c810d1`, throwaway server, mesh claim held and released)

`verify quick`, dev-02 (aarch64), `863c810d1`

`verify fast`, compute01, `863c810d1` (`~claude/j328/logs/gate-863c810d1.log`), verbatim