kayg/infraternal
Watch
1
0
Fork
You've already forked infraternal
 0
Code Issues 356 Pull requests Projects Releases Packages Wiki Activity Actions

J395: eliminate spurious Add host setup-lock refusals #439

New issue
Open
opened 2026-11-08 20:52:00 +00:00 by kayg · 20 comments
kayg commented 2026-11-08 20:52:00 +00:00
Owner
Copy link

Investigate and fix intermittent exit 73 without weakening concurrent setup exclusion. References #432 (comment 86899, J376) and #401 (J381 AlmaLinux reproduction). Production read-only; minipc01 preserved. Plan: trace lock lifecycle and J381 evidence; deterministic red regression; minimal fix; quick and fast gates on compute01; 50 rootless and 50 root Add host runs against per-run guest copies and a loopback throwaway server; prove concurrent exclusion. Initial estimate: 4 hours including build-slot queue and integration sweeps.

Investigate and fix intermittent exit 73 without weakening concurrent setup exclusion. References #432 (comment 86899, J376) and #401 (J381 AlmaLinux reproduction). Production read-only; minipc01 preserved. Plan: trace lock lifecycle and J381 evidence; deterministic red regression; minimal fix; quick and fast gates on compute01; 50 rootless and 50 root Add host runs against per-run guest copies and a loopback throwaway server; prove concurrent exclusion. Initial estimate: 4 hours including build-slot queue and integration sweeps.
kayg commented 2026-11-08 20:52:16 +00:00
Author
Owner
Copy link

J395 started on branch fix-setup-lock-race, base 753fcd657. Worktree: /home/kayg/Developer/Worktrees/infraternal-j395; compute01 mirror and evidence will be ~/worktrees/j395 and ~/.cache/codex-scratch/j395/evidence. Plan and estimate are in the issue body. No production writes or minipc01 access.

J395 started on branch `fix-setup-lock-race`, base `753fcd657`. Worktree: `/home/kayg/Developer/Worktrees/infraternal-j395`; compute01 mirror and evidence will be `~/worktrees/j395` and `~/.cache/codex-scratch/j395/evidence`. Plan and estimate are in the issue body. No production writes or minipc01 access.
kayg commented 2026-11-08 21:02:50 +00:00
Author
Owner
Copy link

Finding: J385's #421 comment 87619 identifies four install failures, each on a rollback retry after an intentionally interrupted/refused first rollback. retire_setup_lease trusts a matching PID/start-time as evidence the lease is still alive, then refuses if fd 7 is absent. A terminated lease can remain a zombie until its parent reaps it: /proc/PID/stat still has the same birth time, while its descriptors are gone. Load extends this window.

Deterministic red on unmodified 753fcd657: create an exited child, retain it with waitid(WEXITED|WNOWAIT), publish its PID/birth as the lease receipt, then call the shipping retirement code. Result: owner_pid=1517633 state=Z birth=6575474 retirement_exit=73, expected 0. Receipt: compute01 ~/.cache/codex-scratch/j395/evidence/zombie-red.log; reproducer ~/.cache/codex-scratch/j395/zombie-repro.py. Added a regression that also retains a genuinely held journal flock and expects 73 in that case. This explains the four rollback retry failures; J381's install refusal still needs qualification rather than attribution by assumption.

A lease-only probe passed 50 three-lease handoff sequences per shell (sh, bash) both on compute01 and the AlmaLinux guest: evidence/lease-probe.log, evidence/guest-lease-probe.log. These are isolated shell probes, not the required Add host sweep. Quick is queued on compute01, PID 1437357; no other heavy J395 run is active. Honcho context lookup timed out; Forgejo remains the record.

Finding: J385's #421 comment 87619 identifies four install failures, each on a rollback retry after an intentionally interrupted/refused first rollback. `retire_setup_lease` trusts a matching PID/start-time as evidence the lease is still alive, then refuses if fd 7 is absent. A terminated lease can remain a zombie until its parent reaps it: `/proc/PID/stat` still has the same birth time, while its descriptors are gone. Load extends this window. Deterministic red on unmodified `753fcd657`: create an exited child, retain it with `waitid(WEXITED|WNOWAIT)`, publish its PID/birth as the lease receipt, then call the shipping retirement code. Result: `owner_pid=1517633 state=Z birth=6575474 retirement_exit=73`, expected 0. Receipt: compute01 `~/.cache/codex-scratch/j395/evidence/zombie-red.log`; reproducer `~/.cache/codex-scratch/j395/zombie-repro.py`. Added a regression that also retains a genuinely held journal flock and expects 73 in that case. This explains the four rollback retry failures; J381's install refusal still needs qualification rather than attribution by assumption. A lease-only probe passed 50 three-lease handoff sequences per shell (`sh`, `bash`) both on compute01 and the AlmaLinux guest: `evidence/lease-probe.log`, `evidence/guest-lease-probe.log`. These are isolated shell probes, not the required Add host sweep. Quick is queued on compute01, PID 1437357; no other heavy J395 run is active. Honcho context lookup timed out; Forgejo remains the record.
kayg commented 2026-11-08 21:11:41 +00:00
Author
Owner
Copy link

Confirmed again with the shipping lease launcher, rather than only a synthetic receipt: actual-lease-repro.py starts a real orphan setup lease, retires it successfully, keeps that exited child unreaped with Linux child-subreaper semantics, then retries retirement. Unmodified source refuses the retry:

actual_lease_pid=1582648 birth=6626688 first_retirement=0 unreaped=True retry_exit=73
REPRO_EXIT=1

Receipt: compute01 ~/.cache/codex-scratch/j395/evidence/actual-lease-red.log. This is a process-lifecycle synchronization reproducer, without CPU stress or speed assertions. The loaded install-suite proof is prepared with 16 test threads and the same delayed-reaping condition, so it will exercise the real rollback paths deterministically rather than rely on the host scheduler reproducing the window. Quick remains queued first as required.

Proposed minimal fix keeps journal validation, the parent guard, and PID/birth/exact-descriptor checks before signalling. Missing descriptors or an exit racing kill defer to the existing journal flock, which remains the final authority. No waiting budget, ownership check, dependency, or live-setup guard is removed. It also retains the existing refusal for a redirected stage. The regression checks both an unlocked journal (success despite the zombie) and a separately held journal (73).

Confirmed again with the shipping lease launcher, rather than only a synthetic receipt: `actual-lease-repro.py` starts a real orphan setup lease, retires it successfully, keeps that exited child unreaped with Linux child-subreaper semantics, then retries retirement. Unmodified source refuses the retry: ```text actual_lease_pid=1582648 birth=6626688 first_retirement=0 unreaped=True retry_exit=73 REPRO_EXIT=1 ``` Receipt: compute01 `~/.cache/codex-scratch/j395/evidence/actual-lease-red.log`. This is a process-lifecycle synchronization reproducer, without CPU stress or speed assertions. The loaded install-suite proof is prepared with 16 test threads and the same delayed-reaping condition, so it will exercise the real rollback paths deterministically rather than rely on the host scheduler reproducing the window. Quick remains queued first as required. Proposed minimal fix keeps journal validation, the parent guard, and PID/birth/exact-descriptor checks before signalling. Missing descriptors or an exit racing `kill` defer to the existing journal flock, which remains the final authority. No waiting budget, ownership check, dependency, or live-setup guard is removed. It also retains the existing refusal for a redirected stage. The regression checks both an unlocked journal (success despite the zombie) and a separately held journal (73).
actual-lease-red.log
434 B
actual-lease-green.log
85 B
kayg commented 2026-11-08 21:17:50 +00:00
Author
Owner
Copy link

Crash-safety checkpoint: the in-progress fix and regression are snapshotted to wip/j395; the final atomic commit on fix-setup-lock-race waits for the required checks. Local branch has rebased onto 0a5432d2c (the #421 gate-path fix); SSH product source is unchanged by that rebase. VM baseline/quick still uses 753fcd657 and the added regression; the final gate will use the final branch head.

The real shipped-lease red/green pair is now complete. Before: first retirement 0, held-unreaped retry 73. Fixed: first retirement 0, held-unreaped retry 0. Green receipt: compute01 ~/.cache/codex-scratch/j395/evidence/actual-lease-green.log; the fixed shell under test is ~/.cache/codex-scratch/j395/setup_lock.fixed.sh. This is not a replacement for the pending parallel Cargo install suite and 50-per-mode Add host proof.

Quick PID 1437357 is first in compute01's slot queue. The only J395 target will be ~/worktrees/j395/target; evidence and scratch helpers are outside it. No J395 server or guest is running while queued. The disposable diagnostic guest was stopped by recorded PID 1445124. Production and minipc01 remain untouched.

Crash-safety checkpoint: the in-progress fix and regression are snapshotted to `wip/j395`; the final atomic commit on `fix-setup-lock-race` waits for the required checks. Local branch has rebased onto `0a5432d2c` (the #421 gate-path fix); SSH product source is unchanged by that rebase. VM baseline/quick still uses `753fcd657` and the added regression; the final gate will use the final branch head. The real shipped-lease red/green pair is now complete. Before: first retirement 0, held-unreaped retry 73. Fixed: first retirement 0, held-unreaped retry 0. Green receipt: compute01 `~/.cache/codex-scratch/j395/evidence/actual-lease-green.log`; the fixed shell under test is `~/.cache/codex-scratch/j395/setup_lock.fixed.sh`. This is not a replacement for the pending parallel Cargo install suite and 50-per-mode Add host proof. Quick PID 1437357 is first in compute01's slot queue. The only J395 target will be `~/worktrees/j395/target`; evidence and scratch helpers are outside it. No J395 server or guest is running while queued. The disposable diagnostic guest was stopped by recorded PID 1445124. Production and minipc01 remain untouched.
kayg commented 2026-11-08 21:29:59 +00:00
Author
Owner
Copy link

J395 checkpoint: recovery snapshot wip/j395-checkpoint2 at f8d506c48df35cb1e1b4cc2dcad7373259bb45c8 preserves the fix, deterministic Linux regression and updated proof notes on base 0a5432d2c. Final atomic commit is pending verification.

compute01 baseline scripts/verify quick is still running (PID 1437357), has cleared workspace clippy and is building SSH unit-test dependencies. Source there retains the original lock shell until the Rust regression and parallel install suite capture red. The standalone real-lease red/green evidence is already recorded above. The single target directory remains /home/claude/worktrees/j395/target; logs and prepared subsequent runners remain under ~/.cache/codex-scratch/j395/.

Next: run red-green.sh through build-slot/nix, inspect retained-orphan parallel-suite results, commit the verified change, full fast gate and 50 rootless + 50 root fresh-overlay Add hosts. No guest sweep result claimed yet.

J395 checkpoint: recovery snapshot `wip/j395-checkpoint2` at `f8d506c48df35cb1e1b4cc2dcad7373259bb45c8` preserves the fix, deterministic Linux regression and updated proof notes on base `0a5432d2c`. Final atomic commit is pending verification. compute01 baseline `scripts/verify quick` is still running (PID 1437357), has cleared workspace clippy and is building SSH unit-test dependencies. Source there retains the original lock shell until the Rust regression and parallel install suite capture red. The standalone real-lease red/green evidence is already recorded above. The single target directory remains `/home/claude/worktrees/j395/target`; logs and prepared subsequent runners remain under `~/.cache/codex-scratch/j395/`. Next: run `red-green.sh` through build-slot/nix, inspect retained-orphan parallel-suite results, commit the verified change, full fast gate and 50 rootless + 50 root fresh-overlay Add hosts. No guest sweep result claimed yet.
kayg commented 2026-11-08 21:44:44 +00:00
Author
Owner
Copy link

J395 finding: the parallel install suite now reproduces the refusal deterministically on baseline, and passes with the minimal retirement fix.

compute01, base 753fcd65788a3cb1121d7dd4dc2272b48edc699f, 16 test threads. Starting /proc/loadavg: 10.91 11.04 11.71. A scratch Linux child-subreaper retained exited orphan leases to make delayed init reaping deterministic.

  • New Rust regression, original shell: expected [Some(0), Some(73)], actual [Some(73), Some(73)]; exit 101. It tests both a free journal and a genuinely held journal with the exact same unreaped lease receipt.
  • Original full install suite: 91 passed, 17 failed, 2 ignored, exit 101; 602 retained zombies. All 17 failures are SetupRunning / exit 73. This includes the buffering-reader test from #432 and all four tests from #421 comment 87619, plus fresh-install/bridge/restaging paths.
  • Fixed lease suite: 3 passed, 0 failed, including the live-journal refusal.
  • Fixed full install suite under the same harness: 108 passed, 0 failed, 2 ignored, exit 0; 652 retained zombies. active_transaction_is_live_between_ssh_setup_commands, concurrent_attempts_cannot_replace_or_clean_each_others_transaction and setup_transaction_holds_a_live_lock_between_shell_commands all pass.

Root cause: /proc/PID/stat and the recorded birth time can still match after the lease exits, although the kernel has closed its descriptors. Retirement incorrectly treated the missing descriptor as a live competing setup. The fix requires PID/birth/exact descriptor before signalling, then uses the already-existing journal flock as the sole proof of completed retirement. A concurrent journal holder still refuses. No new sleep, timeout or dependency.

Receipts on compute01: ~/.cache/codex-scratch/j395/evidence/{lease-regression-red,lease-regression-green,install-parallel-red,install-parallel-green}.log; runner and subreaper beside evidence/. Fixed quick is running. Next are the atomic commit, final gate and the 50+50 fresh-guest Add host sweep. The historical #401 run has no exact shell trace, so this is a reproduced mechanism in the same fresh-install path, not a claimed retrospective trace of its individual failed run.

J395 finding: the parallel install suite now reproduces the refusal deterministically on baseline, and passes with the minimal retirement fix. compute01, base `753fcd65788a3cb1121d7dd4dc2272b48edc699f`, 16 test threads. Starting `/proc/loadavg`: `10.91 11.04 11.71`. A scratch Linux child-subreaper retained exited orphan leases to make delayed init reaping deterministic. - New Rust regression, original shell: expected `[Some(0), Some(73)]`, actual `[Some(73), Some(73)]`; exit 101. It tests both a free journal and a genuinely held journal with the exact same unreaped lease receipt. - Original full install suite: **91 passed, 17 failed, 2 ignored**, exit 101; 602 retained zombies. All 17 failures are `SetupRunning` / exit 73. This includes the buffering-reader test from #432 and all four tests from #421 comment 87619, plus fresh-install/bridge/restaging paths. - Fixed lease suite: **3 passed, 0 failed**, including the live-journal refusal. - Fixed full install suite under the same harness: **108 passed, 0 failed, 2 ignored**, exit 0; **652 retained zombies**. `active_transaction_is_live_between_ssh_setup_commands`, `concurrent_attempts_cannot_replace_or_clean_each_others_transaction` and `setup_transaction_holds_a_live_lock_between_shell_commands` all pass. Root cause: `/proc/PID/stat` and the recorded birth time can still match after the lease exits, although the kernel has closed its descriptors. Retirement incorrectly treated the missing descriptor as a live competing setup. The fix requires PID/birth/exact descriptor before signalling, then uses the already-existing journal flock as the sole proof of completed retirement. A concurrent journal holder still refuses. No new sleep, timeout or dependency. Receipts on compute01: `~/.cache/codex-scratch/j395/evidence/{lease-regression-red,lease-regression-green,install-parallel-red,install-parallel-green}.log`; runner and subreaper beside `evidence/`. Fixed quick is running. Next are the atomic commit, final gate and the 50+50 fresh-guest Add host sweep. The historical #401 run has no exact shell trace, so this is a reproduced mechanism in the same fresh-install path, not a claimed retrospective trace of its individual failed run.
install-parallel-red.log
19 KiB
install-parallel-green.log
8.1 KiB
lease-regression-red.log
2.4 KiB
lease-regression-green.log
2 KiB
kayg commented 2026-11-08 21:46:18 +00:00
Author
Owner
Copy link

J395 commit checkpoint: pushed atomic commit b079bb74fee6f1065c112aae978d1cb83f714b33 on fix-setup-lock-race, based on origin/dev 0a5432d2c1347c57bf1a6c57e368877599d57587. Remote worktree now tracks that exact head, clean.

The fixed quick pre-check and deterministic red/green run completed successfully (RED_GREEN_EXIT=0). Full scripts/verify fast is queued on compute01 through build-slot / nice -n10 nix develop -c, cargo jobs 4, PID 2234754; full log ~/.cache/codex-scratch/j395/evidence/fast.log. The one target remains ~/worktrees/j395/target.

Prepared own-loopback-server and fresh guest-overlay runners remain under J395 scratch and /root/j395-{start-guest,sweep}.sh; guest concurrency checks will verify 73 while a shell holds the shipped guard and 0 after release, separately for rootless and root. Still pending: final fast result, building own server/payload/frontend and 50 successful real Add hosts per mode. No merge/deployment, production or minipc01 touched.

J395 commit checkpoint: pushed atomic commit `b079bb74fee6f1065c112aae978d1cb83f714b33` on `fix-setup-lock-race`, based on origin/dev `0a5432d2c1347c57bf1a6c57e368877599d57587`. Remote worktree now tracks that exact head, clean. The fixed quick pre-check and deterministic red/green run completed successfully (`RED_GREEN_EXIT=0`). Full `scripts/verify fast` is queued on compute01 through build-slot / `nice -n10 nix develop -c`, cargo jobs 4, PID 2234754; full log `~/.cache/codex-scratch/j395/evidence/fast.log`. The one target remains `~/worktrees/j395/target`. Prepared own-loopback-server and fresh guest-overlay runners remain under J395 scratch and `/root/j395-{start-guest,sweep}.sh`; guest concurrency checks will verify 73 while a shell holds the shipped guard and 0 after release, separately for rootless and root. Still pending: final fast result, building own server/payload/frontend and 50 successful real Add hosts per mode. No merge/deployment, production or minipc01 touched.
kayg commented 2026-11-08 22:04:10 +00:00
Author
Owner
Copy link

J395b started (successor to J395, which a provider filter stopped). Branch fix-setup-lock-race, head b079bb74fee6f1065c112aae978d1cb83f714b33, base dev 0a5432d2c. The compute01 worktree ~/worktrees/j395 is at that head and clean.

J395's verify fast is still running in slot 3 (PID 2234754, 17m elapsed, log ~/.cache/codex-scratch/j395/evidence/fast.log, at SSH plugin compile). I'll wait for it and not start a second gate.

Plan: (1) quote the fast result; (2) own loopback server + fresh per-run guest overlay on compute01 with J395's runners: 50 rootless + 50 root Add hosts, then for each mode the guard-held refusal (73) and the success after release; (3) self-review the diff for any weakening against a real concurrent setup. Estimate ~2 h. No production writes, no minipc01, no merge or deploy.

J395b started (successor to J395, which a provider filter stopped). Branch `fix-setup-lock-race`, head `b079bb74fee6f1065c112aae978d1cb83f714b33`, base dev `0a5432d2c`. The compute01 worktree `~/worktrees/j395` is at that head and clean. J395's `verify fast` is still running in slot 3 (PID 2234754, 17m elapsed, log `~/.cache/codex-scratch/j395/evidence/fast.log`, at SSH plugin compile). I'll wait for it and not start a second gate. Plan: (1) quote the fast result; (2) own loopback server + fresh per-run guest overlay on compute01 with J395's runners: 50 rootless + 50 root Add hosts, then for each mode the guard-held refusal (73) and the success after release; (3) self-review the diff for any weakening against a real concurrent setup. Estimate ~2 h. No production writes, no minipc01, no merge or deploy.
kayg commented 2026-11-08 23:03:22 +00:00
Author
Owner
Copy link

J395b result 1: J395's verify fast on b079bb74f (PID 2234754) finished with FAST_EXIT=1. One row failed: doc comments on diff. Verbatim:

crates/plugins/ssh/src/install.rs:950: script needs a doc comment (D345)

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | FAIL (1)     |
| fixed test waits          | PASS         |
| privileged integration    | PASS         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
FAST_EXIT=1

Log: compute01 ~/.cache/codex-scratch/j395/evidence/fast.log. The fix's comment edit inside the private Installation::script makes the D351 row count that fn, and it had no /// doc. Static sizes: aarch64 25305312 bytes, report-only delta +4852032 against the row's own baseline; x86 inside the accepted delta (PASS).

Fixed in a separate doc-only commit, 5f84474d07281e7aa55ce624c8f65ebf13cd35ce (pushed to fix-setup-lock-race). The compute01 worktree is now at that head. The proof build (server, musl payload, web) is queued through build-slot (PID 3372758, log evidence/proof-build.log). Once binaries exist, the 50+50 sweep runs, and verify fast is re-queued on the new head at the same time.

J395b result 1: J395's `verify fast` on `b079bb74f` (PID 2234754) finished with **FAST_EXIT=1**. One row failed: `doc comments on diff`. Verbatim: ```text crates/plugins/ssh/src/install.rs:950: script needs a doc comment (D345) ``` ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | Nix Cargo vendoring | PASS | | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | FAIL (1) | | fixed test waits | PASS | | privileged integration | PASS | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | static builds | PASS | +---------------------------+--------------+ FAST_EXIT=1 ``` Log: compute01 `~/.cache/codex-scratch/j395/evidence/fast.log`. The fix's comment edit inside the private `Installation::script` makes the D351 row count that fn, and it had no `///` doc. Static sizes: aarch64 25305312 bytes, report-only delta +4852032 against the row's own baseline; x86 inside the accepted delta (PASS). Fixed in a separate doc-only commit, `5f84474d07281e7aa55ce624c8f65ebf13cd35ce` (pushed to `fix-setup-lock-race`). The compute01 worktree is now at that head. The proof build (server, musl payload, web) is queued through build-slot (PID 3372758, log `evidence/proof-build.log`). Once binaries exist, the 50+50 sweep runs, and `verify fast` is re-queued on the new head at the same time.
kayg commented 2026-11-08 23:35:02 +00:00
Author
Owner
Copy link

J395b progress: proof build on 5f84474d0 passed (PROOF_BUILD_EXIT=0, evidence/proof-build.log; server sha256 817e3762…, musl payload 77d6bd98…).

Harness finding, not a product failure: the first rootless sweep run hit HOST_BOOTSTRAP_RESULT timeout. The Add host had actually succeeded. The server assigned host id 0057c34b…, an infraternal agent link opened, and the screenshot shows a live [kayg@j311b-alma ~]$ shell. On this build the wizard closes and opens the shell by itself, while J395's scratch flow.ts only counted a done row or an open-terminal button. I patched the scratch flow to also accept "wizard closed + kayg@127.0.0.1, shell, connected window" (original kept as flow.ts.j395-orig). The miss is preserved in evidence/harness-miss-run-kayg-01-kayg/ (server.log, 3-timeout.png). Its server.log has no setup-running or exit-73 lines, only the expected key-less first sign-in failure_kind="auth".

Rootless run 1 (fresh overlay, own loopback server) passed, with the shipped-guard concurrency check:

HOST_BOOTSTRAP_RESULT done
mode=kayg concurrent_exit=73
mode=kayg after_release_exit=0
ADDHOST_RUN_EXIT=0

Receipt: evidence/run-kayg-01-kayg/run.txt. Rootless 2–50, then root 1–50, are running back to back (root PID 3639556, log evidence/sweep-chain.log, about 2 min per run). verify fast on 5f84474d0 is running (evidence/fast2.log).

J395b progress: proof build on `5f84474d0` passed (`PROOF_BUILD_EXIT=0`, `evidence/proof-build.log`; server sha256 `817e3762…`, musl payload `77d6bd98…`). Harness finding, not a product failure: the first rootless sweep run hit `HOST_BOOTSTRAP_RESULT timeout`. The Add host had actually succeeded. The server assigned host id `0057c34b…`, an infraternal agent link opened, and the screenshot shows a live `[kayg@j311b-alma ~]$` shell. On this build the wizard closes and opens the shell by itself, while J395's scratch `flow.ts` only counted a `done` row or an `open-terminal` button. I patched the scratch flow to also accept "wizard closed + `kayg@127.0.0.1, shell, connected` window" (original kept as `flow.ts.j395-orig`). The miss is preserved in `evidence/harness-miss-run-kayg-01-kayg/` (server.log, 3-timeout.png). Its server.log has no setup-running or exit-73 lines, only the expected key-less first sign-in `failure_kind="auth"`. Rootless run 1 (fresh overlay, own loopback server) passed, with the shipped-guard concurrency check: ```text HOST_BOOTSTRAP_RESULT done mode=kayg concurrent_exit=73 mode=kayg after_release_exit=0 ADDHOST_RUN_EXIT=0 ``` Receipt: `evidence/run-kayg-01-kayg/run.txt`. Rootless 2–50, then root 1–50, are running back to back (root PID 3639556, log `evidence/sweep-chain.log`, about 2 min per run). `verify fast` on `5f84474d0` is running (`evidence/fast2.log`).
kayg commented 2026-11-09 00:06:56 +00:00
Author
Owner
Copy link

J395b finding (P1): the fix does not remove every spurious refusal. Rootless runs 1–18 passed on 5f84474d0. Run 19 refused on a fresh guest overlay against its own loopback server, with no other setup anywhere:

00:04:22Z WARN host bootstrap failed host_id=unassigned ... step="connect" failure_kind="auth" exit_status=unavailable mode="rootless"   <- expected key-less first try
Infraternal agent HTTPS link ended (Other)
Infraternal agent HTTPS link ended (Other)
00:04:49Z WARN host bootstrap failed host_id=ff58187d-9116-4699-8f62-1d421cc978f9 target_id=kayg_127.0.0.1_22495 step="install" failure_kind="setup-running" exit_status=73 mode="rootless"
PROGRESS {"title":"Setup is already running on this host","rows":[],"buttons":["back-progress","retry"]}

Receipts (compute01): ~/.cache/codex-scratch/j395/evidence/run-kayg-19-kayg/{server.log,flow-add.log,3-failed.png,run.txt}, evidence/sweep-chain.log (CHAIN_EXIT=1). Load at the time: compute01 was running static builds (loadavg ~20).

Preserved field repro: guest overlay root@compute01:/root/j395-vm/run-kayg-19.qcow2, now read-only, backing /root/j311b-vm/disk.qcow2. I booted a copy only. The stage ~kayg/.local/share/infraternal/.bootstrap-lock had reached activating with the replacing-{binary,launcher,state,unit}, stopping, creating-alias and lease entries. So the 73 came from a late install command, not admission. All files there are 0 bytes, because QEMU's SIGTERM came before the guest flushed file data, so the lease PID/birth is lost. The guest journal wasn't persisted.

What this means: J395's unreaped-lease mechanism is real and fixed (red/green, suite 91/17 → 108/0). A second path still exits 73 in a fresh install under load. The live candidates, unproven: (a) retire_setup_lease's flock -w 2 7 timing out because the lease (or something that inherited fd 7) hasn't released within 2 s under load; (b) setup_guard's flock -n 9 meeting a process that inherited the parent guard. One example: prepare_join_key runs "$stage/payload" agent … prepare-join-key while the shell holds fd 9 and the re-acquired fd 7, and anything that outlives it keeps both. The next step is a sweep run with the failing shell's exit point and the fd holders (/proc/*/fd for the stage and parent) captured at refusal. That's diagnosis work on the protocol, so it isn't patched here.

Next: running the root 1–50 sweep (stops at its first failure) for root-mode data and the root concurrency check.

J395b finding (P1): **the fix does not remove every spurious refusal.** Rootless runs 1–18 passed on `5f84474d0`. **Run 19 refused** on a fresh guest overlay against its own loopback server, with no other setup anywhere: ```text 00:04:22Z WARN host bootstrap failed host_id=unassigned ... step="connect" failure_kind="auth" exit_status=unavailable mode="rootless" <- expected key-less first try Infraternal agent HTTPS link ended (Other) Infraternal agent HTTPS link ended (Other) 00:04:49Z WARN host bootstrap failed host_id=ff58187d-9116-4699-8f62-1d421cc978f9 target_id=kayg_127.0.0.1_22495 step="install" failure_kind="setup-running" exit_status=73 mode="rootless" PROGRESS {"title":"Setup is already running on this host","rows":[],"buttons":["back-progress","retry"]} ``` Receipts (compute01): `~/.cache/codex-scratch/j395/evidence/run-kayg-19-kayg/{server.log,flow-add.log,3-failed.png,run.txt}`, `evidence/sweep-chain.log` (`CHAIN_EXIT=1`). Load at the time: compute01 was running static builds (loadavg ~20). Preserved field repro: guest overlay `root@compute01:/root/j395-vm/run-kayg-19.qcow2`, now read-only, backing `/root/j311b-vm/disk.qcow2`. I booted a copy only. The stage `~kayg/.local/share/infraternal/.bootstrap-lock` had reached `activating` with the `replacing-{binary,launcher,state,unit}`, `stopping`, `creating-alias` and `lease` entries. So the 73 came from a late install command, not admission. All files there are 0 bytes, because QEMU's SIGTERM came before the guest flushed file data, so the lease PID/birth is lost. The guest journal wasn't persisted. What this means: J395's unreaped-lease mechanism is real and fixed (red/green, suite 91/17 → 108/0). A second path still exits 73 in a fresh install under load. The live candidates, unproven: (a) `retire_setup_lease`'s `flock -w 2 7` timing out because the lease (or something that inherited fd 7) hasn't released within 2 s under load; (b) `setup_guard`'s `flock -n 9` meeting a process that inherited the parent guard. One example: `prepare_join_key` runs `"$stage/payload" agent … prepare-join-key` while the shell holds fd 9 and the re-acquired fd 7, and anything that outlives it keeps both. The next step is a sweep run with the failing shell's exit point and the fd holders (`/proc/*/fd` for the stage and parent) captured at refusal. That's diagnosis work on the protocol, so it isn't patched here. Next: running the root 1–50 sweep (stops at its first failure) for root-mode data and the root concurrency check.
kayg commented 2026-11-09 00:16:00 +00:00
Author
Owner
Copy link

J395b finish. Not ready to merge: the gate is green, but the real-environment sweep found a remaining spurious refusal.

Head: 5f84474d07281e7aa55ce624c8f65ebf13cd35ce on fix-setup-lock-race (J395's b079bb74f plus a doc-only commit for the D345 row), base dev 0a5432d2c.

verify fast on that head (compute01, build-slot, nix develop -c; log ~/.cache/codex-scratch/j395/evidence/fast2.log, first line confirms the head):

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| privileged integration    | PASS         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
FAST_EXIT=0

Static: x86 28171984 bytes (+5753728 against the row's baseline), aarch64 25305312 (+4852032), both report-only. Unchanged by this branch except comments.

50/50 sweep: not achieved.

  • Rootless: 18 passed, then run 19 refused with step="install" failure_kind="setup-running" exit_status=73 ("Setup is already running on this host"), on a fresh overlay with its own server and no competitor (comment 87926; overlay preserved read-only at root@compute01:/root/j395-vm/run-kayg-19.qcow2). Rootless run 1 had first timed out on a harness detection bug (the wizard now closes by itself on success), fixed in the scratch flow; that evidence is in evidence/harness-miss-run-kayg-01-kayg/.
  • Root: 0 passed, 2 of 2 failed at step="terminal" failure_kind="setup" exit_status=unavailable after install and enrol finished. It isn't a 73 and it's deterministic on this image. Rollback leaves /var/lib/infraternal clean, with no AVCs logged. That's a separate root-mode problem (probably what J381's relabel-root runs were after) and not part of this lock fix. Evidence in evidence/failed-run-root-01-root/ and evidence/run-root-01-root/; overlay /root/j395-vm/run-root-01.qcow2 (second attempt; it replaced the first under the same name).
  • Totals: evidence/sweep-results.txt (18 rootless ADDHOST_PASS lines).

Concurrency check with the shipped guard (setup_guard.sh at the head): passes both modes. Rootless inside sweep run 1 (evidence/run-kayg-01-kayg/run.txt), and both modes standalone on a fresh overlay (evidence/concurrent-standalone-{root,kayg}.log):

mode=root concurrent_exit=73
mode=root after_release_exit=0
mode=kayg concurrent_exit=73
mode=kayg after_release_exit=0

Diff review (does it weaken the guard against a genuinely concurrent setup?) I found nothing that does:

  1. setup_guard (parent flock -n 9) is untouched, so two setup commands still exclude each other. That is what the 73 above proves.
  2. Whenever the stage directory exists, retire_setup_lease still has to take the stage flock (flock -w 2 7 on a fresh open file description) before it returns. The lease holds that lock until it exits, and its sleepers close fd 7, so an exited lease no longer counts as live, and a live holder still gets 73. The Rust regression checks both ([Some(0), Some(73)]).
  3. The only signal is still sent only when PID, birth time and the exact fd-7 directory all match, so a reused PID is never signalled. Ignoring a failed kill lets the flock decide, which is stricter than the old blanket 73.
  4. The stage-symlink refusal, the lease-file validation (owner, mode, size, format) and every caller's transaction check are unchanged.
  5. Remaining edge: a live lease whose fd 7 points to a different directory now gets no signal and is left to its bounded expiry (≤360 s), not refused. It holds no lock on the current $stage, so it can't overlap a mutation. I'm noting it, not counting it as a weakening.

Lead for the remaining 73 (unproven): commit_after takes the exclusive parent guard with -n right after terminal proof. Meanwhile runtime_lock::acquire_bound (crates/plugins/remote/src/runtime_lock.rs:168) takes a blocking shared flock on that same parent directory, for the whole call including the kernel socket dump. A session or runtime acquisition still in flight under load would make commit refuse. The run-19 stage was at activating, the commit stage, which fits. Possible next step, for the orchestrator to decide: a sweep that records the failing shell's exit point and the holders of the parent and stage fds at refusal. If confirmed, a bounded setup_guard_wait for commit_after (as restore uses) is one candidate. That's a lock-protocol decision, so I haven't changed it.

Cleanup: every J395b server and QEMU was stopped by PID (none remain), target ~/worktrees/j395/target deleted (52G), and the inspection copies removed. Kept: evidence (118M scratch) and the two failure overlays. Production and minipc01 untouched. No merge or deploy.

J395b finish. **Not ready to merge:** the gate is green, but the real-environment sweep found a remaining spurious refusal. **Head:** `5f84474d07281e7aa55ce624c8f65ebf13cd35ce` on `fix-setup-lock-race` (J395's `b079bb74f` plus a doc-only commit for the D345 row), base dev `0a5432d2c`. **verify fast** on that head (compute01, build-slot, `nix develop -c`; log `~/.cache/codex-scratch/j395/evidence/fast2.log`, first line confirms the head): ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | Nix Cargo vendoring | PASS | | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | PASS | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | static builds | PASS | +---------------------------+--------------+ FAST_EXIT=0 ``` Static: x86 28171984 bytes (+5753728 against the row's baseline), aarch64 25305312 (+4852032), both report-only. Unchanged by this branch except comments. **50/50 sweep: not achieved.** - Rootless: **18 passed, then run 19 refused** with `step="install" failure_kind="setup-running" exit_status=73` ("Setup is already running on this host"), on a fresh overlay with its own server and no competitor (comment 87926; overlay preserved read-only at `root@compute01:/root/j395-vm/run-kayg-19.qcow2`). Rootless run 1 had first timed out on a harness detection bug (the wizard now closes by itself on success), fixed in the scratch flow; that evidence is in `evidence/harness-miss-run-kayg-01-kayg/`. - Root: **0 passed, 2 of 2 failed** at `step="terminal" failure_kind="setup" exit_status=unavailable` after install and enrol finished. It isn't a 73 and it's deterministic on this image. Rollback leaves `/var/lib/infraternal` clean, with no AVCs logged. That's a separate root-mode problem (probably what J381's `relabel-root` runs were after) and not part of this lock fix. Evidence in `evidence/failed-run-root-01-root/` and `evidence/run-root-01-root/`; overlay `/root/j395-vm/run-root-01.qcow2` (second attempt; it replaced the first under the same name). - Totals: `evidence/sweep-results.txt` (18 rootless ADDHOST_PASS lines). **Concurrency check with the shipped guard (`setup_guard.sh` at the head): passes both modes.** Rootless inside sweep run 1 (`evidence/run-kayg-01-kayg/run.txt`), and both modes standalone on a fresh overlay (`evidence/concurrent-standalone-{root,kayg}.log`): ```text mode=root concurrent_exit=73 mode=root after_release_exit=0 mode=kayg concurrent_exit=73 mode=kayg after_release_exit=0 ``` **Diff review (does it weaken the guard against a genuinely concurrent setup?)** I found nothing that does: 1. `setup_guard` (parent `flock -n 9`) is untouched, so two setup commands still exclude each other. That is what the 73 above proves. 2. Whenever the stage directory exists, `retire_setup_lease` still has to take the stage flock (`flock -w 2 7` on a fresh open file description) before it returns. The lease holds that lock until it exits, and its sleepers close fd 7, so an exited lease no longer counts as live, and a live holder still gets 73. The Rust regression checks both (`[Some(0), Some(73)]`). 3. The only signal is still sent only when PID, birth time *and* the exact fd-7 directory all match, so a reused PID is never signalled. Ignoring a failed `kill` lets the flock decide, which is stricter than the old blanket 73. 4. The stage-symlink refusal, the lease-file validation (owner, mode, size, format) and every caller's transaction check are unchanged. 5. Remaining edge: a live lease whose fd 7 points to a *different* directory now gets no signal and is left to its bounded expiry (≤360 s), not refused. It holds no lock on the current `$stage`, so it can't overlap a mutation. I'm noting it, not counting it as a weakening. **Lead for the remaining 73 (unproven):** `commit_after` takes the exclusive parent guard with `-n` right after terminal proof. Meanwhile `runtime_lock::acquire_bound` (crates/plugins/remote/src/runtime_lock.rs:168) takes a blocking *shared* flock on that same parent directory, for the whole call including the kernel socket dump. A session or runtime acquisition still in flight under load would make commit refuse. The run-19 stage was at `activating`, the commit stage, which fits. Possible next step, for the orchestrator to decide: a sweep that records the failing shell's exit point and the holders of the parent and stage fds at refusal. If confirmed, a bounded `setup_guard_wait` for `commit_after` (as restore uses) is one candidate. That's a lock-protocol decision, so I haven't changed it. Cleanup: every J395b server and QEMU was stopped by PID (none remain), target `~/worktrees/j395/target` deleted (52G), and the inspection copies removed. Kept: evidence (118M scratch) and the two failure overlays. Production and minipc01 untouched. No merge or deploy.
kayg commented 2026-11-09 00:19:46 +00:00
Author
Owner
Copy link

J395b round 2 started (the orchestrator decided; owner not asked). Branch fix-setup-lock-race, head 5f84474d0.

Plan:

  1. Diagnose (scratch only, never committed). Every non-comment exit 73 in setup_guard.sh, setup_lock.sh and previous_attempt.sh is wrapped as { setup_diag <file:line>; exit 73; }. setup_diag writes /tmp/j395-diag.* on the guest: refusing site, argv tail (it names the command), parent/stage/lifetime-lock dev+inode, /proc/locks, fuser -v, a /proc/*/fd holder scan with cmdlines, and ps. Built into out-diag/ (build-slot PID 50395, evidence/diag-build.log); the tree is restored after the build. Runner collects diag files every run, plus the journal/AVC on failure. First a reproduction attempt on a copy of run 19's kept overlay, then rootless sweeps until a refusal is caught with evidence, or 100 clean runs.
  2. Fix by evidence. If it's commit_after's flock -n against runtime_lock::acquire_bound's shared lock, the regression test comes first (shared lock held across commit and released within budget → 0; held past it → 73), then a bounded exclusive wait of a few seconds, the way restore uses setup_guard_wait.
  3. Root. Confirm #401 (203/EXEC, var_lib_t, AVCs) from the guest journal on a failing root run. If confirmed, the root sweep runs with setenforce 0 on each per-run overlay, and the results say so.
  4. Fresh build of the fix head, 50 rootless + 50 root, verify fast, cleanup.
J395b round 2 started (the orchestrator decided; owner not asked). Branch `fix-setup-lock-race`, head `5f84474d0`. Plan: 1. **Diagnose (scratch only, never committed).** Every non-comment `exit 73` in `setup_guard.sh`, `setup_lock.sh` and `previous_attempt.sh` is wrapped as `{ setup_diag <file:line>; exit 73; }`. `setup_diag` writes `/tmp/j395-diag.*` on the guest: refusing site, argv tail (it names the command), parent/stage/lifetime-lock dev+inode, `/proc/locks`, `fuser -v`, a `/proc/*/fd` holder scan with cmdlines, and `ps`. Built into `out-diag/` (build-slot PID 50395, `evidence/diag-build.log`); the tree is restored after the build. Runner collects diag files every run, plus the journal/AVC on failure. First a reproduction attempt on a copy of run 19's kept overlay, then rootless sweeps until a refusal is caught with evidence, or 100 clean runs. 2. **Fix by evidence.** If it's `commit_after`'s `flock -n` against `runtime_lock::acquire_bound`'s shared lock, the regression test comes first (shared lock held across commit and released within budget → 0; held past it → 73), then a bounded exclusive wait of a few seconds, the way restore uses `setup_guard_wait`. 3. **Root.** Confirm #401 (203/EXEC, `var_lib_t`, AVCs) from the guest journal on a failing root run. If confirmed, the root sweep runs with `setenforce 0` on each per-run overlay, and the results say so. 4. Fresh build of the fix head, 50 rootless + 50 root, `verify fast`, cleanup.
kayg commented 2026-11-09 01:39:06 +00:00
Author
Owner
Copy link

J395b root cause confirmed (diagnostic build 5f84474d0+diag, scratch only: every exit 73 dumps /proc/locks, fuser -v, a /proc fd scan and the refusing command's argv).

Reproduction on run 19's kept overlay: not reproduced. Re-adding over that torn state (0-byte journal files) fails earlier at step="check" (failure_kind="setup"), with no 73 and no diagnostic file (evidence/repro19-kayg/). The original overlay is unchanged.

Caught with evidence: diagnostic rootless run 18 (evidence/diag-kayg-018-kayg/), loadavg ~25–40, same symptom as run 19:

01:32:34.819Z WARN host bootstrap failed host_id=20f10a7d-… step="install" failure_kind="setup-running" exit_status=73 mode="rootless"
01:32:35.094Z WARN infraternal agent connection failed; awaiting reconnect
PROGRESS {"title":"Setup is already running on this host",...}
TAG setup_guard:44 pid=2460 uid=1001 time=01:32:34.385855503 base=/home/kayg/.local/share/infraternal
  • Refusing command: PID 2460, setup_guard's flock -n 9 on ~/.local/share (inode 41943168). Its argv tail is the post-activation linger check (loginctl show-user "$uid" -p Linger), not the commit.
  • Holders, milliseconds later:
--- /proc/locks
1: FLOCK  ADVISORY  WRITE 2459 fc:04:110166 0 EOF
2: FLOCK  ADVISORY  WRITE 2447 fc:04:8452781 0 EOF
3: FLOCK  ADVISORY  WRITE 1087 00:1b:1685 0 EOF
--- fuser
Specified filename /nonexistent does not exist.
                     USER        PID ACCESS COMMAND
/home/kayg/.local/share:
                     kayg       2460 f.... sh
                     kayg       2465 f.... sh
--- fd holders

No lock remains on 41943168, and only the refusing shell and its own diagnostic subshell have it open. Whatever held the guard released it almost at once. The stage lock (lease 2447) and the runtime lifetime lock (2459) are held, and the agent was starting right then.

  • Who else locks that directory: only runtime_lock::acquire_bound (crates/plugins/remote/src/runtime_lock.rs:168, parent.lock_shared()). The infraternal agent calls it at startup (crates/infraternal-agent/src/plugin_control.rs:567). The only other shared flock in the tree, login_binary.rs, locks a private image file.

Root cause: systemd starts the new infraternal agent during activation. While admitting its runtime, the agent takes a brief shared lock on the install root's parent. On a loaded host that overlaps the next guarded setup command, whose non-blocking exclusive setup_guard refuses with 73, and the server reports a finished install as "Setup is already running". It's the mechanism the orchestrator suspected, but any guarded command after activation can hit it, not only the commit. So the fix goes in setup_guard itself: a 5 s bounded wait by default. Restore keeps 25 s, and the locked-state inspector keeps -n so host checks stay fast.

A second pattern, diagnostic run 6 (evidence/diag-kayg-006-kayg/): the first failure was step="connect" failure_kind="setup" at 60 s, a slow or abandoned command under load. Its rollback then refused with 73 while that shell (2346, fuser) still held the guard. The same bounded wait helps that rollback. The 60 s primary failure is a separate timing limit under extreme load, and it isn't what users saw as "Setup is already running".

Root failures = #401, confirmed on an enforcing per-run overlay (evidence/enf-root-001-root/fail-guest.txt):

infraternal-agent.service: Failed at step EXEC spawning /var/lib/infraternal/bin/current: Permission denied
infraternal-agent.service: Main process exited, code=exited, status=203/EXEC

(avc_count=0: a dontaudit rule hides the denial; /var/lib/infraternal is var_lib_t.) The root sweep therefore runs with setenforce 0 per overlay, and the results say so.

Next: red/green of the new regression (build-slot PID 880425, evidence/guard-rg.log).

J395b root cause confirmed (diagnostic build `5f84474d0+diag`, scratch only: every `exit 73` dumps `/proc/locks`, `fuser -v`, a /proc fd scan and the refusing command's argv). **Reproduction on run 19's kept overlay:** not reproduced. Re-adding over that torn state (0-byte journal files) fails earlier at `step="check"` (`failure_kind="setup"`), with no 73 and no diagnostic file (`evidence/repro19-kayg/`). The original overlay is unchanged. **Caught with evidence: diagnostic rootless run 18** (`evidence/diag-kayg-018-kayg/`), loadavg ~25–40, same symptom as run 19: ```text 01:32:34.819Z WARN host bootstrap failed host_id=20f10a7d-… step="install" failure_kind="setup-running" exit_status=73 mode="rootless" 01:32:35.094Z WARN infraternal agent connection failed; awaiting reconnect PROGRESS {"title":"Setup is already running on this host",...} TAG setup_guard:44 pid=2460 uid=1001 time=01:32:34.385855503 base=/home/kayg/.local/share/infraternal ``` - **Refusing command:** PID 2460, `setup_guard`'s `flock -n 9` on `~/.local/share` (inode 41943168). Its argv tail is the post-activation linger check (`loginctl show-user "$uid" -p Linger`), **not the commit**. - **Holders, milliseconds later:** ```text --- /proc/locks 1: FLOCK ADVISORY WRITE 2459 fc:04:110166 0 EOF 2: FLOCK ADVISORY WRITE 2447 fc:04:8452781 0 EOF 3: FLOCK ADVISORY WRITE 1087 00:1b:1685 0 EOF --- fuser Specified filename /nonexistent does not exist. USER PID ACCESS COMMAND /home/kayg/.local/share: kayg 2460 f.... sh kayg 2465 f.... sh --- fd holders ``` No lock remains on 41943168, and only the refusing shell and its own diagnostic subshell have it open. Whatever held the guard released it almost at once. The stage lock (lease 2447) and the runtime lifetime lock (2459) are held, and the agent was starting right then. - **Who else locks that directory:** only `runtime_lock::acquire_bound` (crates/plugins/remote/src/runtime_lock.rs:168, `parent.lock_shared()`). The infraternal agent calls it at startup (crates/infraternal-agent/src/plugin_control.rs:567). The only other shared flock in the tree, login_binary.rs, locks a private image file. **Root cause:** systemd starts the new infraternal agent during activation. While admitting its runtime, the agent takes a brief shared lock on the install root's parent. On a loaded host that overlaps the next guarded setup command, whose non-blocking exclusive `setup_guard` refuses with 73, and the server reports a finished install as "Setup is already running". It's the mechanism the orchestrator suspected, but any guarded command after activation can hit it, not only the commit. So the fix goes in `setup_guard` itself: a 5 s bounded wait by default. Restore keeps 25 s, and the locked-state inspector keeps `-n` so host checks stay fast. A second pattern, diagnostic run 6 (`evidence/diag-kayg-006-kayg/`): the first failure was `step="connect" failure_kind="setup"` at 60 s, a slow or abandoned command under load. Its rollback then refused with 73 while that shell (2346, `fuser`) still held the guard. The same bounded wait helps that rollback. The 60 s primary failure is a separate timing limit under extreme load, and it isn't what users saw as "Setup is already running". **Root failures = #401, confirmed** on an enforcing per-run overlay (`evidence/enf-root-001-root/fail-guest.txt`): ```text infraternal-agent.service: Failed at step EXEC spawning /var/lib/infraternal/bin/current: Permission denied infraternal-agent.service: Main process exited, code=exited, status=203/EXEC ``` (`avc_count=0`: a dontaudit rule hides the denial; `/var/lib/infraternal` is `var_lib_t`.) The root sweep therefore runs with `setenforce 0` per overlay, and the results say so. Next: red/green of the new regression (build-slot PID 880425, `evidence/guard-rg.log`).
kayg commented 2026-11-09 03:01:04 +00:00
Author
Owner
Copy link

J395b fix pushed.

  • 17750ccd4d6f5b4e1fd0ecc726493278897a44d2: setup guard waits briefly instead of refusing a starting agent. setup_guard uses flock -w 5 when the caller sets no wait. Restore keeps 25 s; the locked-state inspector keeps -n. Includes the regression test commit_waits_for_a_brief_shared_holder_and_refuses_a_long_one. The holder is released only once /proc/locks shows the commit queued behind it, so the outcome never depends on timing.
    • Red (shipped guard): Failure { kind: SetupRunning, message: "Setup is already running on this host. …", failed_step: Some(Install), exit_status: Some(73) }, GUARD_RED_EXIT=101
    • Green: test … ok, GUARD_GREEN_EXIT=0
    • install suite 109 passed, 0 failed, 2 ignored; lease_progress 3 passed (compute01 evidence/guard-{red,green,suite}.log, evidence/guard-rg.log)
  • 1e3f15e8bea5a81357c95dbed409bacb3a71ae16: merge of origin/dev 6b60f1fb7, clean. It includes fix-selinux-agent-exec (#401), merged since the last round. So the root sweep runs on the enforcing guest, and permissive is now only a fallback.

Harness check before the change: a permissive root Add host on the old build passed (evidence/perm-root-001-root/run.txt), so the root path itself works once #401 is out of the way.

Next: proof build of 1e3f15e8b (build-slot PID 2208318, evidence/proof-build2.log), then 50 rootless + 50 root fresh-overlay Add hosts, with the concurrency check on run 1 of each, and verify fast.

J395b fix pushed. - `17750ccd4d6f5b4e1fd0ecc726493278897a44d2`: **setup guard waits briefly instead of refusing a starting agent.** `setup_guard` uses `flock -w 5` when the caller sets no wait. Restore keeps 25 s; the locked-state inspector keeps `-n`. Includes the regression test `commit_waits_for_a_brief_shared_holder_and_refuses_a_long_one`. The holder is released only once `/proc/locks` shows the commit queued behind it, so the outcome never depends on timing. - Red (shipped guard): `Failure { kind: SetupRunning, message: "Setup is already running on this host. …", failed_step: Some(Install), exit_status: Some(73) }`, `GUARD_RED_EXIT=101` - Green: `test … ok`, `GUARD_GREEN_EXIT=0` - `install` suite **109 passed, 0 failed, 2 ignored**; `lease_progress` **3 passed** (compute01 `evidence/guard-{red,green,suite}.log`, `evidence/guard-rg.log`) - `1e3f15e8bea5a81357c95dbed409bacb3a71ae16`: **merge of origin/dev `6b60f1fb7`**, clean. It includes `fix-selinux-agent-exec` (#401), merged since the last round. So the root sweep runs on the **enforcing** guest, and permissive is now only a fallback. Harness check before the change: a permissive root Add host on the old build passed (`evidence/perm-root-001-root/run.txt`), so the root path itself works once #401 is out of the way. Next: proof build of `1e3f15e8b` (build-slot PID 2208318, `evidence/proof-build2.log`), then 50 rootless + 50 root fresh-overlay Add hosts, with the concurrency check on run 1 of each, and `verify fast`.
kayg commented 2026-11-09 04:44:29 +00:00
Author
Owner
Copy link

J395b progress, final sweep on 1e3f15e8b (fix + merged dev, enforcing guest, fresh overlay and own server per run):

  • Rootless 1–33 passed, 0 refusals. Run 1's concurrency check against the shipped guard: mode=kayg concurrent_exit=73 (after the 5 s wait), after_release_exit=0 (evidence/final-kayg-001-kayg/run.txt).
  • Run 29 passed (ADDHOST_RUN_EXIT=0), but my sweep wrapper then crashed in QEMU cleanup: a race between the /proc/PID/stat check and the read, inherited from J395's runner. The wrapper is fixed, the evidence is kept as evidence/harness-crash-final-kayg-029-kayg/, and run 29 was re-run.
  • Run 34 failed because compute01's disk filled. It failed at 04:42:58Z with step="install" failure_kind="setup" exit_status=unavailable (no 73). My queued verify fast died at about 04:43:05Z with No space left on device (os error 28) and a linker bus error (evidence/fast3-enospc.log). All guest-side collections for run 34 are empty because the guest was unreachable, as QEMU pauses on ENOSPC. Root disk was at 96% afterwards (26G free, floor 25G). Evidence: evidence/enospc-final-kayg-034-kayg/, overlay /root/j395-vm/enospc-final-kayg-034.qcow2. It's marked ADDHOST_FAIL_HOST_ENOSPC in sweep-final.txt. The rootless sweep resumed at 34 (root PID 3668776), and verify fast is re-queued (PID 3669213, evidence/fast4.log).

Orchestrator: compute01 hit 0 bytes free around 04:43Z. My one target is ~/worktrees/j395/target (33G), needed for the gate; I'll delete it at the end.

J395b progress, final sweep on `1e3f15e8b` (fix + merged dev, enforcing guest, fresh overlay and own server per run): - Rootless **1–33 passed, 0 refusals**. Run 1's concurrency check against the shipped guard: `mode=kayg concurrent_exit=73` (after the 5 s wait), `after_release_exit=0` (`evidence/final-kayg-001-kayg/run.txt`). - Run 29 passed (`ADDHOST_RUN_EXIT=0`), but my sweep wrapper then crashed in QEMU cleanup: a race between the `/proc/PID/stat` check and the read, inherited from J395's runner. The wrapper is fixed, the evidence is kept as `evidence/harness-crash-final-kayg-029-kayg/`, and run 29 was re-run. - **Run 34 failed because compute01's disk filled.** It failed at 04:42:58Z with `step="install" failure_kind="setup" exit_status=unavailable` (no 73). My queued `verify fast` died at about 04:43:05Z with `No space left on device (os error 28)` and a linker bus error (`evidence/fast3-enospc.log`). All guest-side collections for run 34 are empty because the guest was unreachable, as QEMU pauses on ENOSPC. Root disk was at 96% afterwards (26G free, floor 25G). Evidence: `evidence/enospc-final-kayg-034-kayg/`, overlay `/root/j395-vm/enospc-final-kayg-034.qcow2`. It's marked `ADDHOST_FAIL_HOST_ENOSPC` in `sweep-final.txt`. The rootless sweep resumed at 34 (root PID 3668776), and `verify fast` is re-queued (PID 3669213, `evidence/fast4.log`). Orchestrator: compute01 hit 0 bytes free around 04:43Z. My one target is `~/worktrees/j395/target` (33G), needed for the gate; I'll delete it at the end.
kayg commented 2026-11-09 07:02:16 +00:00
Author
Owner
Copy link

J395b sweep result on 1e3f15e8bea5a81357c95dbed409bacb3a71ae16 (setup-guard fix + merged dev with #401). Each run uses a fresh guest overlay, its own loopback server and its own software passkey, on the enforcing AlmaLinux guest (permissive=0 on every run; setenforce 0 was not needed). compute01 loadavg mostly 20–40.

HEAD=1e3f15e8bea5a81357c95dbed409bacb3a71ae16
ROOTLESS_ADD_HOSTS=50/50
ROOT_ADD_HOSTS=50/50 (enforcing)
SPURIOUS_REFUSALS=0
SET_ASIDE_RUNS_REFUSALS={'harness-crash-final-kayg-029-kayg': 0, 'enospc-final-kayg-034-kayg': 0, 'harness-trust-final-kayg-039-kayg': 0}
CONCURRENT_ROOTLESS_EXIT=73
CONCURRENT_ROOT_EXIT=73
AFTER_RELEASE_BOTH=0
SWEEP_REPORT_EXIT=0

The checker (~/.cache/codex-scratch/j395/report-final.py) checks every run's head, HOST_BOOTSTRAP_RESULT done, the terminal proof (nonce + id -un typed in the new host's terminal and read back from the guest), the PASS line, and no setup-running/73 anywhere in its server log. Report: evidence/sweep-final-report.json; per-run receipts evidence/final-{kayg,root}-NNN-*/; ledger evidence/sweep-final.txt.

Concurrency check with the shipped guard while a shell holds it (run 1 of each mode): concurrent_exit=73 (after the 5 s wait), then after_release_exit=0, in both modes.

Three rootless attempts were set aside and re-run, none a setup refusal (0 refusal lines in each server log, all kept):

  • run 29 passed, but my sweep wrapper crashed in QEMU cleanup afterwards (fixed race) → harness-crash-final-kayg-029-kayg/
  • run 34 failed when compute01's disk filled (ENOSPC at 04:43Z, comment 88314) → enospc-final-kayg-034-kayg/
  • run 39 failed when the browser script clicked "trust host key" three times under load; the page showed "Something went wrong", and the server never left awaiting host key confirmation. The script now trusts once → harness-trust-final-kayg-039-kayg/. Side note for the orchestrator: a real double-click on that button may hit the same error. It's worth a separate look; I haven't investigated it.

So the 50 rootless passes are not 50 consecutive attempts: 53 attempts, 3 set aside for the causes above. The 50 root passes are consecutive. verify fast is re-queued on the same head (see next comment).

J395b sweep result on `1e3f15e8bea5a81357c95dbed409bacb3a71ae16` (setup-guard fix + merged dev with #401). Each run uses a fresh guest overlay, its own loopback server and its own software passkey, on the **enforcing** AlmaLinux guest (`permissive=0` on every run; `setenforce 0` was not needed). compute01 loadavg mostly 20–40. ```text HEAD=1e3f15e8bea5a81357c95dbed409bacb3a71ae16 ROOTLESS_ADD_HOSTS=50/50 ROOT_ADD_HOSTS=50/50 (enforcing) SPURIOUS_REFUSALS=0 SET_ASIDE_RUNS_REFUSALS={'harness-crash-final-kayg-029-kayg': 0, 'enospc-final-kayg-034-kayg': 0, 'harness-trust-final-kayg-039-kayg': 0} CONCURRENT_ROOTLESS_EXIT=73 CONCURRENT_ROOT_EXIT=73 AFTER_RELEASE_BOTH=0 SWEEP_REPORT_EXIT=0 ``` The checker (`~/.cache/codex-scratch/j395/report-final.py`) checks every run's head, `HOST_BOOTSTRAP_RESULT done`, the terminal proof (nonce + `id -un` typed in the new host's terminal and read back from the guest), the PASS line, and no `setup-running`/73 anywhere in its server log. Report: `evidence/sweep-final-report.json`; per-run receipts `evidence/final-{kayg,root}-NNN-*/`; ledger `evidence/sweep-final.txt`. Concurrency check with the shipped guard while a shell holds it (run 1 of each mode): `concurrent_exit=73` (after the 5 s wait), then `after_release_exit=0`, in both modes. **Three rootless attempts were set aside and re-run**, none a setup refusal (0 refusal lines in each server log, all kept): - run 29 passed, but my sweep wrapper crashed in QEMU cleanup afterwards (fixed race) → `harness-crash-final-kayg-029-kayg/` - run 34 failed when compute01's disk filled (ENOSPC at 04:43Z, comment 88314) → `enospc-final-kayg-034-kayg/` - run 39 failed when the browser script clicked "trust host key" three times under load; the page showed "Something went wrong", and the server never left `awaiting host key confirmation`. The script now trusts once → `harness-trust-final-kayg-039-kayg/`. Side note for the orchestrator: a real double-click on that button may hit the same error. It's worth a separate look; I haven't investigated it. So the 50 rootless passes are not 50 consecutive attempts: 53 attempts, 3 set aside for the causes above. The 50 root passes are consecutive. `verify fast` is re-queued on the same head (see next comment).
kayg commented 2026-11-09 07:28:32 +00:00
Author
Owner
Copy link

J395b finish (round 2). Not merged, not deployed.

Head: 1e3f15e8bea5a81357c95dbed409bacb3a71ae16 on fix-setup-lock-race

  • b079bb74f: J395's unreaped-lease retirement fix
  • 5f84474d0: doc comment for the D345 row
  • 17750ccd4: setup guard waits up to 5 s instead of refusing a starting agent, plus a deterministic regression test
  • 1e3f15e8b: merge of origin/dev 6b60f1fb7 (brings the #401 SELinux fix)

Root cause (comment 88104): systemd starts the new infraternal agent during activation. While admitting its runtime, the agent takes a brief shared lock on the install root's parent (runtime_lock::acquire_bound, called from plugin_control.rs:567). On a loaded host this overlaps the next guarded setup command, whose non-blocking exclusive setup_guard (flock -n 9) refuses with 73, so a finished install is reported as "Setup is already running". Caught by the scratch diagnostic build: the refusing command was the post-activation linger check, and /proc/locks + fuser showed the holder gone milliseconds later. Fix: flock -w 5 by default. Restore keeps 25 s, and the locked-state inspector keeps -n, so host checks stay fast. A guard holder that outlasts the wait still gets 73. One that releases sooner is still refused by the admission, transaction and stage-flock checks that follow.

Regression test commit_waits_for_a_brief_shared_holder_and_refuses_a_long_one: red on the shipped guard (kind: SetupRunning … exit_status: Some(73)), green with the fix. install suite 109/0 and lease_progress 3/0.

Sweep (comment 88459): 50/50 rootless, 50/50 root, enforcing SELinux, 0 spurious refusals, every receipt checked by report-final.py (evidence/sweep-final-report.json). The concurrency check with the shipped guard gave 73 while held and 0 after release, in both modes. Three rootless attempts were set aside, none a refusal: a harness crash after a pass, compute01's disk filling up, and a triple-click on host-key trust.

Root failures from round 1 = #401, confirmed on the enforcing guest (Failed at step EXEC spawning /var/lib/infraternal/bin/current: Permission denied, status=203/EXEC). Merging dev resolved them, so setenforce 0 was not used.

verify fast on 1e3f15e8b:

  • First run (evidence/fast4.log): FAST_EXIT=1, one test failed, in code this branch doesn't touch:
test remote_runtime::tests::standby::liveness_doubt::one_missed_primary_reply_shows_reconnecting_and_only_a_second_retires_the_link ... FAILED
the link retired on its first miss: 366.070579ms
test result: FAILED. 960 passed; 1 failed; 57 ignored; 0 measured; 0 filtered out; finished in 835.71s

On the same head it passed 5/5 in isolation at load ~30 (retire_ms 522–548, evidence/liveness-rerun.log). It's a wall-clock timing test that's flaky under load.

  • Re-run (evidence/fast5.log):
+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| privileged integration    | PASS         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
FAST_EXIT=0

Static: x86 28176912 bytes, aarch64 25310432 bytes (report-only deltas +5758656 / +4857152 against the row's baseline; +4928 / +5120 against 5f84474d0, from the merged #401 code).

  • An earlier gate attempt (evidence/fast3-enospc.log) died from compute01 running out of disk (comment 88314).

Cleanup: every server and guest of mine stopped by PID (none remain), 54G target deleted, worktree clean at the head. Scratch evidence: 584M in ~/.cache/codex-scratch/j395/. Kept small overlays in /root/j395-vm/ as repros: run-kayg-19, diag-kayg-006, diag-kayg-018 (the root-cause catch), enf-root-001 and run-root-01 (#401), enospc-final-kayg-034, harness-trust-final-kayg-039. Production and minipc01 untouched.

Open item for the orchestrator: a double-click on "trust host key" may produce "Something went wrong" in a real browser too. Not investigated here.

J395b finish (round 2). **Not merged, not deployed.** **Head:** `1e3f15e8bea5a81357c95dbed409bacb3a71ae16` on `fix-setup-lock-race` - `b079bb74f`: J395's unreaped-lease retirement fix - `5f84474d0`: doc comment for the D345 row - `17750ccd4`: **setup guard waits up to 5 s instead of refusing a starting agent**, plus a deterministic regression test - `1e3f15e8b`: merge of origin/dev `6b60f1fb7` (brings the #401 SELinux fix) **Root cause** (comment 88104): systemd starts the new infraternal agent during activation. While admitting its runtime, the agent takes a brief *shared* lock on the install root's parent (`runtime_lock::acquire_bound`, called from `plugin_control.rs:567`). On a loaded host this overlaps the next guarded setup command, whose non-blocking exclusive `setup_guard` (`flock -n 9`) refuses with 73, so a finished install is reported as "Setup is already running". Caught by the scratch diagnostic build: the refusing command was the post-activation linger check, and `/proc/locks` + `fuser` showed the holder gone milliseconds later. **Fix:** `flock -w 5` by default. Restore keeps 25 s, and the locked-state inspector keeps `-n`, so host checks stay fast. A guard holder that outlasts the wait still gets 73. One that releases sooner is still refused by the admission, transaction and stage-flock checks that follow. **Regression test** `commit_waits_for_a_brief_shared_holder_and_refuses_a_long_one`: red on the shipped guard (`kind: SetupRunning … exit_status: Some(73)`), green with the fix. `install` suite 109/0 and `lease_progress` 3/0. **Sweep** (comment 88459): **50/50 rootless, 50/50 root, enforcing SELinux, 0 spurious refusals**, every receipt checked by `report-final.py` (`evidence/sweep-final-report.json`). The concurrency check with the shipped guard gave 73 while held and 0 after release, in both modes. Three rootless attempts were set aside, none a refusal: a harness crash after a pass, compute01's disk filling up, and a triple-click on host-key trust. **Root failures from round 1 = #401**, confirmed on the enforcing guest (`Failed at step EXEC spawning /var/lib/infraternal/bin/current: Permission denied`, `status=203/EXEC`). Merging dev resolved them, so `setenforce 0` was not used. **verify fast** on `1e3f15e8b`: - First run (`evidence/fast4.log`): `FAST_EXIT=1`, one test failed, in code this branch doesn't touch: ```text test remote_runtime::tests::standby::liveness_doubt::one_missed_primary_reply_shows_reconnecting_and_only_a_second_retires_the_link ... FAILED the link retired on its first miss: 366.070579ms test result: FAILED. 960 passed; 1 failed; 57 ignored; 0 measured; 0 filtered out; finished in 835.71s ``` On the same head it passed 5/5 in isolation at load ~30 (`retire_ms` 522–548, `evidence/liveness-rerun.log`). It's a wall-clock timing test that's flaky under load. - Re-run (`evidence/fast5.log`): ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | Nix Cargo vendoring | PASS | | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | PASS | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | static builds | PASS | +---------------------------+--------------+ FAST_EXIT=0 ``` Static: x86 28176912 bytes, aarch64 25310432 bytes (report-only deltas +5758656 / +4857152 against the row's baseline; +4928 / +5120 against `5f84474d0`, from the merged #401 code). - An earlier gate attempt (`evidence/fast3-enospc.log`) died from compute01 running out of disk (comment 88314). **Cleanup:** every server and guest of mine stopped by PID (none remain), 54G target deleted, worktree clean at the head. Scratch evidence: 584M in `~/.cache/codex-scratch/j395/`. Kept small overlays in `/root/j395-vm/` as repros: `run-kayg-19`, `diag-kayg-006`, `diag-kayg-018` (the root-cause catch), `enf-root-001` and `run-root-01` (#401), `enospc-final-kayg-034`, `harness-trust-final-kayg-039`. Production and minipc01 untouched. **Open item for the orchestrator:** a double-click on "trust host key" may produce "Something went wrong" in a real browser too. Not investigated here.
kayg commented 2026-11-09 08:30:45 +00:00
Author
Owner
Copy link

J417 review of fix-setup-lock-race at 1e3f15e8bea5a81357c95dbed409bacb3a71ae16 (correctness and safety). Gate: queued at position 9 on compute01, in a scratch merge with origin/dev 4754b14a7 (~/worktrees/j417, merge 9597c06cf, log ~/.cache/codex-scratch/j417/evidence/fast.log). The result follows in a separate comment.

1. b079bb74f, retire_setup_lease. Correct. PID+birth and fd/7 -ef $stage now only decide whether we signal. Whether retirement is complete is decided by flock -w 2 7 on the stage directory, which was already the authority. Cases:

  • live lease holds the stage lock: kill, then the flock waits it out → 0 (as before)
  • exited, unreaped lease: no fds, no kill, the flock succeeds → 0 (the bug fixed)
  • a different, unrecorded holder of the stage lock: the flock fails → 73 (still refused; the test checks the held case)
  • PID reuse: the birth mismatch prevents a kill, and the flock decides
  • kill failing (lease exits in the race window) is ignored safely, for the same reason
    A stale holder can't be misread as gone: a holder with a live descriptor keeps the flock.

2. 17750ccd4, flock -w 5. No new double-admission path. The guard is still an exclusive flock, so two shells can never hold it at the same time. A waiting setup that gets in after a finishing one releases is the same state as a setup that arrived a moment after the release under the old -n, and the transaction/admission checks and the stage flock -n 7 already had to (and do) refuse there. The wait only changes when a contender looks, not what it is allowed past. Precedence is right: an explicit setup_guard_wait (restore 25 s) wins, the inspector keeps -n, and everything else gets -w 5. A real competitor that holds the guard past 5 s still exits 73 (your concurrency check: 73 held / 0 after release).

Tests. Both are meaningful and deterministic:

  • released_lease_owner_can_stay_unreaped… keeps a real zombie with WNOWAIT and probes unheld→0 and held→73, so it covers both the fix and the guard against over-admitting.
  • commit_waits_for_a_brief_shared_holder… releases only once /proc/locks shows the commit queued (->), so it proves the wait rather than a lucky ordering, plus the outlasting-holder→SetupRunning side.

Non-blocking notes:

  • (P3) A real refusal now takes 5 s longer to report. That is fine inside the 30 s/60 s command limits, but it is user-visible latency on the true "already running" path.
  • (P3) The fix assumes the infraternal agent's shared hold in runtime_lock::acquire_bound stays brief, which is how it behaves today. If a later change keeps that shared lock across the agent's lifetime, every post-activation command would wait 5 s and then refuse. A one-line invariant comment at runtime_lock.rs:168 pointing back here would keep that coupling visible.
  • (P3) The lease test spins on yield_now for up to 60 s while it waits for the child to exit. It is harmless, but a short sleep would be gentler on a loaded host.

Verdict pending the gate: APPROVE on the code.

J417 review of `fix-setup-lock-race` at `1e3f15e8bea5a81357c95dbed409bacb3a71ae16` (correctness and safety). Gate: queued at position 9 on compute01, in a scratch merge with origin/dev `4754b14a7` (`~/worktrees/j417`, merge `9597c06cf`, log `~/.cache/codex-scratch/j417/evidence/fast.log`). The result follows in a separate comment. **1. `b079bb74f`, retire_setup_lease.** Correct. PID+birth and `fd/7 -ef $stage` now only decide whether we *signal*. Whether retirement is complete is decided by `flock -w 2 7` on the stage directory, which was already the authority. Cases: - live lease holds the stage lock: kill, then the flock waits it out → 0 (as before) - exited, unreaped lease: no fds, no kill, the flock succeeds → 0 (the bug fixed) - a different, unrecorded holder of the stage lock: the flock fails → 73 (still refused; the test checks the held case) - PID reuse: the birth mismatch prevents a kill, and the flock decides - `kill` failing (lease exits in the race window) is ignored safely, for the same reason A stale holder can't be misread as gone: a holder with a live descriptor keeps the flock. **2. `17750ccd4`, `flock -w 5`.** No new double-admission path. The guard is still an exclusive flock, so two shells can never hold it at the same time. A waiting setup that gets in after a finishing one releases is the same state as a setup that arrived a moment after the release under the old `-n`, and the transaction/admission checks and the stage `flock -n 7` already had to (and do) refuse there. The wait only changes *when* a contender looks, not *what* it is allowed past. Precedence is right: an explicit `setup_guard_wait` (restore 25 s) wins, the inspector keeps `-n`, and everything else gets `-w 5`. A real competitor that holds the guard past 5 s still exits 73 (your concurrency check: 73 held / 0 after release). **Tests.** Both are meaningful and deterministic: - `released_lease_owner_can_stay_unreaped…` keeps a real zombie with `WNOWAIT` and probes unheld→0 and held→73, so it covers both the fix and the guard against over-admitting. - `commit_waits_for_a_brief_shared_holder…` releases only once `/proc/locks` shows the commit queued (`->`), so it proves the wait rather than a lucky ordering, plus the outlasting-holder→SetupRunning side. **Non-blocking notes:** - (P3) A real refusal now takes 5 s longer to report. That is fine inside the 30 s/60 s command limits, but it is user-visible latency on the true "already running" path. - (P3) The fix assumes the infraternal agent's shared hold in `runtime_lock::acquire_bound` stays brief, which is how it behaves today. If a later change keeps that shared lock across the agent's lifetime, every post-activation command would wait 5 s and then refuse. A one-line invariant comment at runtime_lock.rs:168 pointing back here would keep that coupling visible. - (P3) The lease test spins on `yield_now` for up to 60 s while it waits for the child to exit. It is harmless, but a short sleep would be gentler on a loaded host. Verdict pending the gate: **APPROVE** on the code.
kayg commented 2026-11-09 11:31:11 +00:00
Author
Owner
Copy link

J417 final. Verdict: APPROVE. Reviewed 1e3f15e8bea5a81357c95dbed409bacb3a71ae16; the review is in comment 88538, with no blocking findings and three P3 notes.

scripts/verify fast on compute01, scratch merge 9597c06cf = branch + origin/dev 4754b14a7 (log ~/.cache/codex-scratch/j417/evidence/fast.log), verbatim:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | FAIL (1)     |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| privileged integration    | PASS         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
VERIFY_EXIT=1

The one failing row is not caused by this branch. error[vulnerability]: libsqlite3-sys via C SQLite CVE-2026-119375, RUSTSEC-2026-0459, a new advisory-database entry. cargo deny check advisories on an unmodified origin/dev 4754b14a7 checkout gives the same RUSTSEC-2026-0459 … advisories FAILED. The merge differs from dev only by the branch's 5 files (crates/plugins/ssh/...), and none of them touches a manifest or Cargo.lock. That advisory needs its own dev fix (bump libsqlite3-sys/sqlx or a reasoned deny.toml ignore). It does not block this branch.

Scratch worktrees removed. No target dir was left behind (the gate built into the worktree's target, which was removed with it).

J417 final. **Verdict: APPROVE.** Reviewed `1e3f15e8bea5a81357c95dbed409bacb3a71ae16`; the review is in comment 88538, with no blocking findings and three P3 notes. `scripts/verify fast` on compute01, scratch merge `9597c06cf` = branch + origin/dev `4754b14a7` (log `~/.cache/codex-scratch/j417/evidence/fast.log`), verbatim: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | Nix Cargo vendoring | PASS | | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | FAIL (1) | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | PASS | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | static builds | PASS | +---------------------------+--------------+ VERIFY_EXIT=1 ``` The one failing row is not caused by this branch. `error[vulnerability]: libsqlite3-sys via C SQLite CVE-2026-119375`, `RUSTSEC-2026-0459`, a new advisory-database entry. `cargo deny check advisories` on an unmodified origin/dev `4754b14a7` checkout gives the same `RUSTSEC-2026-0459 … advisories FAILED`. The merge differs from dev only by the branch's 5 files (`crates/plugins/ssh/...`), and none of them touches a manifest or Cargo.lock. That advisory needs its own dev fix (bump libsqlite3-sys/sqlx or a reasoned deny.toml ignore). It does not block this branch. Scratch worktrees removed. No target dir was left behind (the gate built into the worktree's target, which was removed with it).
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
j470b-server-mesh-rejoin
mesh-compat-382
j444b-d381-key-wins-flood
j461b-macos-utun-reader-thread
j475b-remove-host-motion
fix-reconnect-new-link
j481-d380
dev
j479-mesh-soak
fix/j476-host-facts
j475-motion-pill-split
j470-mac-node-join
root-upgrade-d376
j474-review-merge
latency-targets-120hz
nightly-suite-shards
gotatun-pin-j469
macos-utun-reader-thread
fix-flood-echo-stall
wip/j416
motion-fast-ease-out
mesh-addhost-j259
takeover-close-terminals-d368
kbd-instant-followups
fix-trust-key-double-click
nightly-suite-partial-report
appwin-icons-j207
rusqlite-slice1
swipe-content-first-d374
mesh-mtu-381
addhost-instant-open
glass-contrast-d365-mock
glass-contrast-d365
signout-followups-442
fix-startup-toast-red
research-gotatun-matrix-j317
themes-sync-j304
fix-load-flakes
fix-nix-zig-patch
fix-compute01-browser-baseline
ratchet-size-baseline-419
predict-auto-d130
revoke-coverage-d367
nightly-suite-queue-timeout
nightly-browser-suite
fix-rustsec-2026-0459
fonts-ioskeley-obel
log-retention-d375
fix-kbd-no-anim-427
fix-dev-red-specs-2
takeover-proof-305-j205
wip/j410-server-pump
ui-consistency-d372
fix-osc52-clipboard
libghostty-vt-1.0
forget-retired-host-d369
j404-merge-426
fix-setup-lock-race
fix-touch-select-426
revoke-access-report-ui
fix-predictive-echo-specs
fix-selinux-agent-exec
wip/j395-checkpoint2
wip/j395
j391-review-scratch
ui-consistency-j195-deferred-rebased
fix-rustc-paths-421
decisions-wording-341
fix-install-suite-red
fix-dev-rtt-specs
fix-pending-removal-align
fix-privileged-skip
fix-install-suite-404
review-j368
mesh-dns-j254
revoke-access-report-d356b
revoke-privilege-round5
fix-removal-confirm
review-j360
gotatun-rotation-367
deps-refresh-340
review-j354
review-J351
review-j350
addhost-takeover-d359
revoke-report-items-347
revoke-end-sessions-d362
review-j348
size-mesh-ssh-346
size-trim-337
revoke-access-diff-d356
j259-pre-rebase-b06c5a6
j305-ended-at-start-spec
addhost-sudo-consent
fix-layout-real-specs-411
fix-shell-open-411
fix-console-hosts-373
fix-shell-exit
fuzz-host-name-353
fix-host-details-410
macos-filter-j197
research-gotatun-tricks
revoke-p3-j216l
stale-state-pill
revoke-followups-j216k
revoke-progress-j216e
backup/macos01-20261106/lume/j197
backup/macos01-20261106/lume/appwin-icons-j207-stash0
backup/macos01-20261106/claude/stash-0
backup/macos01-20261106/claude/stash-1
backup/macos01-20261106/claude/J80-macos-s1
backup/macos01-20261106/claude/J80
backup/macos01-20261106/claude/J80-slice1-20261102
backup/macos01-20261106/claude/J16b-appwin-on-dev
backup/macos01-20261106/claude/appwin-launch-idle-146
backup/macos01-20261106/claude/J80-s1-run
wip/j259
trust-285-p3c
runtime-lock-j216f
mesh-ssh-p3-j175
tun-proof-p3b
trust-285-p3
terminal-polish-j271
tun-proof-p3
ssh-install-flakes-j216i
fuzz-fixes-j264
trust-285-a
trust-285-b
restart-oversized-fixture-j198
backup/macos01-20261106/lume/appwin-icons-j207
restart-bound-j216h
up-preflight-p3-j198
backup/macos01-20261106/lume/j251
up-preflight-j198
wip/up-preflight-j198-j245-p2
wip/up-preflight-j198-j245-p1
agent-fsync-offloop-j196
wip/up-preflight-j198-j245
settings-offscreen-j231
wip/up-preflight-j198-j226-inspection
wip/j103f
themes-j206
wip/up-preflight-j198-j226
flakes-j216c
backup/macos01-20261106/claude/j197
flakes-j216b
backup/macos01-20261106/claude/j197-red
wip/up-preflight-j198-review
flakes-j216
backup/macos01-20261106/claude/appwin-icons-j207
engineering-j208b
appwin-mock-j209
toast-stack-j165
engineering-j208
decisions-d351
decisions-d350
decisions-d349
appearance-presets-mock-j129
decisions-d348
docs-audit-spec
doc-coverage-baseline-raise
decisions-d347
decisions-d346b
decisions-d346
j88h-profile
ui-consistency-j195-deferred
ui-consistency-j195
engineering-practices-j194
preferred-login-ui-j166
signin-pools-263-j174
macos-node-265-s1
decisions-d345
decisions-d344
proxy-cgroup-j164
verify-quick-fuzz-329
proxy-start-io-j168
timing-flakes-j163
up-263-s1
mcp-events-j147
verify-quick-fuzz-j158
proxy-p3-j159
agent-resource-j157-detect
wip/j103d-final
remote-cancel-hang-j155
proxy-traefik-281-j62d
j149-verify-quick
proxy-traefik-281-j62b
j150-drop-unnamed
wip/j79d-red-r4
wip/j103d-d332
rootless-toast-j142
wip/j103d-head
wip/j103d
j125c-mcp
macos-build-j136
backup/macos01-20261106/claude/j136
wip/j79d-red-cli
wip/j79d-red-server
j121-link-liveness
j118c-tracer-failures
wip/j79c-head
release-desktop-j131
j124c-signed-macos
wip/j79c-red-p3
wip/j79c-red-p23
wip/j79c-red-p22
merge-j128-check
perf-deferred-j128
sidebar-toggle-260
j127-shortcut-changed-dot
j125-mcp
j118-tracer-failures
j126-account-summary
settings-sections-mock-j92
j120-pinch-lock
sidebar-292
backup/macos01-20261106/claude/j120
taste-2026-11-03
row-actions-242-215
backup/macos01-20261106/claude/stash-0-untracked
backup/macos01-20261106/claude/J80-265-s1-run
backup/macos01-20261106/claude/J80-macos-s1-20261102
j112-canvas-commands
ui-no-overshoot-scroll
fix-recovery-restart
backup/macos01-20261106/claude/stash-1-untracked
taste-2026-11-02
size-baseline-ratchet
machine-cards-structure-j108
j109-boot-copy-mockups
fix-dev-red
wip/up-263-s1
j95b-settings-rows
proxy-traefik-281
j95-option-rows
docs-human-j91
docs-keys-palette
fix-flood-1013-270-j38d
mesh-ssh-s1
mesh-ssh-s0
addhost-d291-tunnel
test-stability-load
mesh-relay-220
repro-296
shim-thp-off-24
mesh-pins-258-v2
mesh-dns
signin-d311-merged
docs-pages
ui-size-283
fix-ui-batch-specs
wip/j12b-edge-update
gate-on-arm
fix-rename-sync-293
trust-285-j60b
fix-focus-ring-clip
integ-ui-1101
edge-update-v2
signin-d311
taste-2026-11-01
layout-per-tab-250
ui-approved-splitopt-224
ui-approved-splitopt-224-on-dev
icon-text-centre-282
ui-approved-189-212-216
research-reverse-proxy
transfer-budget-320
review-mesh-pins-258-r2
sidebar-pill-259
review-mesh-relay-220
wip/j12-edge-update-v2
nix-package
status-crash-loop-237
fix-dev-test-failures
fix-fixture-socket-path
wip/mesh-relay-220-pre-dev-advance-rebase
fix-canvas-side-clip
research-head-to-head
frosted-real-256
review-mesh-pins-258
wip/mesh-relay-220-pre-final-rebase
review-layout-per-tab-250-r2
broad-suite-triage
nix-devshell
review-status-crash-loop-237-r3
wip/nix-devshell-pre-p3
taste-frosted-real
wip/code-serve-demo-2026-10-30
wip/code-serve-glass-2026-10-30
wip/code-serve-clusters-2026-10-30
wip/code-merge-release94-dev-2026-10-30
wip/rproxy-delta-scratch-2026-10-30
wip/stale-note-257-staged-2026-10-30
wip/d241-size-staged-2026-10-30
review-broad-suite-207
wip/mesh-pins-258-pre-split
wip/mesh-relay-220-pre-rebase
mesh-measure-220
plan-211-update
review-edge-autoupdate-r3
perf24-reconnect
review-stale-note-r2
release-prod-live
d241-shell-marker
review-mesh-pins-server-258
review-revocation-r3
review-setup-token-r2
review-sync-journal-r3
wip/review-mesh-pins-258-prune-review-1995fee46
frosted-scroll-256
sync-journal-r3
revocation-per-account-wip
review-tokio-asyncfd-0386
review-sync-p3s
wip/stash-8by64-stash0-2026-10-30
mesh-drag-r2
setup-token-data-wip
release-9-4
auth-busy-1013
mesh-view-overlap
sync-p3s
review-auth-busy-wake
review-readd-r8-wip
mesh-drag-bounds
audit-auth-93
app-windows-stream-start
review-authority-cap-r3
app-windows-r7-measure
readd-minimal-r6
readd-minimal-r5
app-windows-r7-soak
readd-minimal-r4
app-windows-r7
app-windows-r6
readd-minimal-r3
readd-minimal-r2
readd-minimal
merge-release8-dev
review-busy-page-liveness-r2
review-busy-page-liveness
mesh-view-trim
review-authority-read-retry-r2
review-remove-uninstall-r2
release-retry-reattach
release-retry
review-remove-uninstall
review-authority-read-retry
app-windows-ship
mesh-autoswitch-p2
mesh-ssh
mesh-relay
mesh-macos-node
screen-and-control
session-labels
review-real-logins-r5
review-real-logins-r4
review-real-logins-r3
shim-memory-r2
review-real-logins-r2
shim-memory
shadow-addhost-instant
shadow-real-logins
shadow-same-session-flood
shadow-prod-perf-fix
hosts-from-server
app-windows-real
app-windows-transport
release-prod-6w
mesh-policy-verify
quiet-close-toast
dev-merge-r4
review-mesh-agent-transport
review-app-windows-transport
settings-roundtrip
review-release-pipeline-r4
mesh-agent-transport
ended-strip-fix
review-audit-ca-update-d244
settings-rows
phone-width
signout-clear
fix-restart-tests
dev-merge-rp
signer-hardening
review-audit-ca-update
terminal-fidelity
review-audit-ca-update-wip
mesh-integrate
review-mesh-agent-transport-515d3712
review-mesh-dataplane-feec1988
mesh-dataplane
signout
research-canvas-node
wip/review-renderer-cost
wip/review-terminal-fidelity
padlock-glyph
review-release-pipeline-2e95b689
phone-polish
sidebar-polish
inbox-generic
review-mesh-dataplane-378d4055
prod-batch
ui-inbox
ui-ca-unlock
ui-machine-sections
review-mesh-dataplane-a5f744fb
tls-session-prod
session-bounds-prod
fix-dead-session
session-bounds
tls-session
palette-polish
shortcuts-d222
research-appmeta
phone-gutter
ui-centre-button
wip/ui-centre-button
wip/tls-session
review-addhost-breakit
ui-platform
renderer-cost
remote-agents
mesh-api
ssh-bootstrap
prod-hotfix2
prod-fixes
review-remote-agents
review-mesh-dataplane
session-life
research-browser
review-mesh-api
research-capture
review-mesh-api-round3
review-mesh-api-round1
gate-stability
review-gate-stability
review-relay-direct
review-mesh-api-findings
proto-sessions
font-picker
proto-addhost
proto-windows
mesh-core
latency-gate
research-bsd
research-discovery
keytray-mods
mock-core
ui-tweaks
fix-motion-sel
fix-motion
integrate2
fix-robust
t-web
t-server
vt-ghostty
fix-snap-echo
t-web-auth
review-vt2
t-agent
lane-protocol
lane-shim
lane-auth
s1-base
s1-contracts
s1-workspace
No results found.
Labels
Clear labels   
launch-qa
Findings from launch QA.

No labels
launch-qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/infraternal#439
No description provided.