kayg/infraternal
Watch
1
0
Fork
You've already forked infraternal
 0
Code Issues 356 Pull requests Projects Releases Packages Wiki Activity Actions

Agent: durable writes off the event loop (follow-up to #334, #344) #347

New issue
Closed
opened 2026-11-04 16:15:29 +00:00 by kayg · 142 comments
kayg commented 2026-11-04 16:15:29 +00:00
Owner
Copy link

J196: preserve durable save completion, errors and ordering while moving runtime-thread fsync work to the blocking pool. Add causal regression tests, prove under real disk load, run quick and fast, push atomic commits. No merge or deploy.

J196: preserve durable save completion, errors and ordering while moving runtime-thread fsync work to the blocking pool. Add causal regression tests, prove under real disk load, run quick and fast, push atomic commits. No merge or deploy.
kayg commented 2026-11-04 16:15:54 +00:00
Author
Owner
Copy link

J196 started on branch agent-fsync-offloop-j196, base 2198148d7 (origin/dev). Plan: audit all durable writes and J163 pattern; add causal failing tests; move saves off-loop with locks retained through completion; run loaded-disk proof and quick/fast; push and remove targets. Estimate: 90 minutes including gates, queue time uncertain. Dedicated target: VM worktrees/j196/target. Both VM slot queues are being checked before selecting the build host.

J196 started on branch agent-fsync-offloop-j196, base 2198148d7 (origin/dev). Plan: audit all durable writes and J163 pattern; add causal failing tests; move saves off-loop with locks retained through completion; run loaded-disk proof and quick/fast; push and remove targets. Estimate: 90 minutes including gates, queue time uncertain. Dedicated target: VM worktrees/j196/target. Both VM slot queues are being checked before selecting the build host.
kayg commented 2026-11-04 16:16:38 +00:00
Author
Owner
Copy link

Audit: logins::restore_at directory fsync; logins::revoke_at file+directory fsync; rootless_sessions::reserve file+directory fsync (called by async launch_configured); rootless_sessions::revoke directory fsync; login_binary::accessible -> publish/prune file+directory fsync (called by async ensure). All currently execute inline on runtime. automation::operations::save is already offloaded by J163. startup_guard::change runs only in the synchronous __startup-guard helper process. Preserve person fences through offloaded completion, including cancellation. dev-02 currently has two occupied slots and two queued jobs; compute01 has four occupied slots and two queued jobs, so loaded proof will use compute01 unless dev-02 becomes idle.

Audit: logins::restore_at directory fsync; logins::revoke_at file+directory fsync; rootless_sessions::reserve file+directory fsync (called by async launch_configured); rootless_sessions::revoke directory fsync; login_binary::accessible -> publish/prune file+directory fsync (called by async ensure). All currently execute inline on runtime. automation::operations::save is already offloaded by J163. startup_guard::change runs only in the synchronous __startup-guard helper process. Preserve person fences through offloaded completion, including cancellation. dev-02 currently has two occupied slots and two queued jobs; compute01 has four occupied slots and two queued jobs, so loaded proof will use compute01 unless dev-02 becomes idle.
kayg commented 2026-11-04 16:19:02 +00:00
Author
Owner
Copy link

Causal regression tests are written for all five paths. A cfg(test) path-scoped gate holds the write; the current-thread reactor must answer unrelated work before it releases the gate. A watchdog only bounds a deadlock, with no stopwatch assertion. Red run queued on compute01 at position 5; log /home/claude/j196-evidence/red.log. No implementation changes yet.

Causal regression tests are written for all five paths. A cfg(test) path-scoped gate holds the write; the current-thread reactor must answer unrelated work before it releases the gate. A watchdog only bounds a deadlock, with no stopwatch assertion. Red run queued on compute01 at position 5; log /home/claude/j196-evidence/red.log. No implementation changes yet.
kayg commented 2026-11-04 16:33:00 +00:00
Author
Owner
Copy link

Queue progress: compute01 advanced J196 from fifth to second waiting position; dev-02 remains occupied with two queued jobs. Added a causal cancellation test: aborting a held authority write must not free the person fence before disk completion. Lock ownership will be transferred into blocking writes, avoiding new shared state. No implementation changes before the red run completes.

Queue progress: compute01 advanced J196 from fifth to second waiting position; dev-02 remains occupied with two queued jobs. Added a causal cancellation test: aborting a held authority write must not free the person fence before disk completion. Lock ownership will be transferred into blocking writes, avoiding new shared state. No implementation changes before the red run completes.
kayg commented 2026-11-04 16:35:28 +00:00
Author
Owner
Copy link

Crash-safety checkpoint pushed: e8d593b4b on agent-fsync-offloop-j196. Tests only, deliberately retaining inline writes for the red run, not a completed implementation. J196 is now first in compute01 queue. Red evidence: /home/claude/j196-evidence/red.log; target /home/claude/worktrees/j196/target. WIP checkpoint will be replaced by one verified atomic fix before handoff.

Crash-safety checkpoint pushed: e8d593b4b on agent-fsync-offloop-j196. Tests only, deliberately retaining inline writes for the red run, not a completed implementation. J196 is now first in compute01 queue. Red evidence: /home/claude/j196-evidence/red.log; target /home/claude/worktrees/j196/target. WIP checkpoint will be replaced by one verified atomic fix before handoff.
kayg commented 2026-11-04 16:44:25 +00:00
Author
Owner
Copy link

Red reproduced on compute01: cargo test -j4 -p infraternal-agent --lib durable_write -- --nocapture exited 101. All 6 tests failed with "durable write held the reactor until the watchdog released it" (0 passed; 6 failed). Log /home/claude/j196-evidence/red.log. The only compilation warning is existing plugin_control::Service::with_registry dead_code under this narrow feature set. Implemented offloading for all five paths, with owned person fences and rootless directory sync outside the cleanup deadline. Focused green + quick pre-check queued next.

Red reproduced on compute01: cargo test -j4 -p infraternal-agent --lib durable_write -- --nocapture exited 101. All 6 tests failed with "durable write held the reactor until the watchdog released it" (0 passed; 6 failed). Log /home/claude/j196-evidence/red.log. The only compilation warning is existing plugin_control::Service::with_registry dead_code under this narrow feature set. Implemented offloading for all five paths, with owned person fences and rootless directory sync outside the cleanup deadline. Focused green + quick pre-check queued next.
kayg commented 2026-11-04 16:49:43 +00:00
Author
Owner
Copy link

Green focused run on compute01: same durable_write filter now passes 6/6, FOCUSED_EXIT=0. This cached focused unit run is the light-command exception in the brief; the heavy quick/clippy check remains queued through build-slot. Existing narrow-feature with_registry dead_code warning remains unchanged. Evidence /home/claude/j196-evidence/focused.log. The reservation owns its person fence so cancellation keeps both durable IO and reservation cleanup serialized; restore/revoke transfer owned fences into blocking tasks.

Green focused run on compute01: same durable_write filter now passes 6/6, FOCUSED_EXIT=0. This cached focused unit run is the light-command exception in the brief; the heavy quick/clippy check remains queued through build-slot. Existing narrow-feature with_registry dead_code warning remains unchanged. Evidence /home/claude/j196-evidence/focused.log. The reservation owns its person fence so cancellation keeps both durable IO and reservation cleanup serialized; restore/revoke transfer owned fences into blocking tasks.
kayg commented 2026-11-04 16:50:31 +00:00
Author
Owner
Copy link

Atomic fix pushed: 069b38c80, replacing the explicitly temporary tests-only checkpoint. Six focused causal tests pass. Full audit changes: logins::restore_at; logins::revoke_at/revoke_marker; rootless_sessions::reserve; rootless_sessions::revoke final directory sync; login_binary::accessible covering publish/prune. File and directory fsync calls remain unchanged, with owned lock handoffs and errors propagated. quick is queued through build-slot on the same target; fast and root/disk-load proof remain pending.

Atomic fix pushed: 069b38c80, replacing the explicitly temporary tests-only checkpoint. Six focused causal tests pass. Full audit changes: logins::restore_at; logins::revoke_at/revoke_marker; rootless_sessions::reserve; rootless_sessions::revoke final directory sync; login_binary::accessible covering publish/prune. File and directory fsync calls remain unchanged, with owned lock handoffs and errors propagated. quick is queued through build-slot on the same target; fast and root/disk-load proof remain pending.
kayg commented 2026-11-04 16:53:38 +00:00
Author
Owner
Copy link

Current atomic head is ae5f08029 (amended the checkpoint fix to avoid a used-underscore-binding lint in the ownership handoff; same behavior). Focused regression tests passed 6/6. quick is still queued, and a foreground SSH session will start fast only if quick exits 0. Root proof is staged in /root/j196-proof, using a throwaway clone and the actual test executable; loaded proof will use compute01 because dev-02 remains busy.

Current atomic head is ae5f08029 (amended the checkpoint fix to avoid a used-underscore-binding lint in the ownership handoff; same behavior). Focused regression tests passed 6/6. quick is still queued, and a foreground SSH session will start fast only if quick exits 0. Root proof is staged in /root/j196-proof, using a throwaway clone and the actual test executable; loaded proof will use compute01 because dev-02 remains busy.
kayg commented 2026-11-04 16:58:20 +00:00
Author
Owner
Copy link

Head 984b24301: tightened the cancellation test to join disk completion via a blocking flock instead of the production 5 s acquisition deadline, so busy-disk latency cannot determine the assertion. Focused tests pass 6/6 again. quick remains queued on compute01; fast foreground session waits for quick success. Root/disk proof is also queued through build-slot, in a public-HTTPS throwaway clone under /root/j196-proof; it uses one 512 MiB stress-ng writer at idle IO priority and starts tests only after write_bytes is positive. The proof checks all six causal cases as root, including actual binary publication.

Head 984b24301: tightened the cancellation test to join disk completion via a blocking flock instead of the production 5 s acquisition deadline, so busy-disk latency cannot determine the assertion. Focused tests pass 6/6 again. quick remains queued on compute01; fast foreground session waits for quick success. Root/disk proof is also queued through build-slot, in a public-HTTPS throwaway clone under /root/j196-proof; it uses one 512 MiB stress-ng writer at idle IO priority and starts tests only after write_bytes is positive. The proof checks all six causal cases as root, including actual binary publication.
kayg commented 2026-11-04 17:13:52 +00:00
Author
Owner
Copy link

Head 984b24301f: scripts/verify quick passed on compute01, with 146 unit tests passed, 0 failed, 9 existing ignored. Verbatim summary:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| scope guard               | PASS         |
| fmt                       | PASS         |
| doc coverage ratchet      | PASS         |
| clippy (workspace)        | PASS         |
| unit tests (changed)      | PASS         |
+---------------------------+--------------+

QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done.
QUICK_EXIT=0

Full gate foreground session has advanced to build-slot after quick success. Root/disk-load proof remains queued. Logs: /home/claude/j196-evidence/{focused,green,quick,fast}.log and /root/j196-proof/proof.log. No merge or deploy.

Head 984b24301fabd6eaa6b196127661f4875e92a964: scripts/verify quick passed on compute01, with 146 unit tests passed, 0 failed, 9 existing ignored. Verbatim summary: ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | scope guard | PASS | | fmt | PASS | | doc coverage ratchet | PASS | | clippy (workspace) | PASS | | unit tests (changed) | PASS | +---------------------------+--------------+ QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done. QUICK_EXIT=0 ``` Full gate foreground session has advanced to build-slot after quick success. Root/disk-load proof remains queued. Logs: /home/claude/j196-evidence/{focused,green,quick,fast}.log and /root/j196-proof/proof.log. No merge or deploy.
kayg commented 2026-11-04 17:19:51 +00:00
Author
Owner
Copy link

Real-environment proof passed on compute01, head 984b24301f, uid 0, throwaway clone under /root. Ran the actual test executable while ionice -c3 stress-ng --hdd 1 --hdd-bytes 512M --hdd-opts wr-seq,fsync was active. LOAD_STARTED observed worker write_bytes=524288 before tests began. All 6 causal cases passed, including actual root-only binary publication; PROOF_EXIT=0 and PROOF_COMMAND_EXIT=0. The writer reported 45 HDD operations, 0 failed; whole-host sda counters rose by 125546496 written bytes / 910 write requests during the proof (includes tests and other host work, not a benchmark).

Evidence retained on the control host: /home/kayg/.cache/codex-scratch/j196/{proof.log,stress.log,root-evidence.tar.gz}. The writer was reaped, and the root clone, binary and fixtures were removed: ROOT_PROOF_CLEANUP=PASS. Remaining target: /home/claude/worktrees/j196/target, retained only for the pending fast gate and to be deleted when it finishes.

Real-environment proof passed on compute01, head 984b24301fabd6eaa6b196127661f4875e92a964, uid 0, throwaway clone under /root. Ran the actual test executable while ionice -c3 stress-ng --hdd 1 --hdd-bytes 512M --hdd-opts wr-seq,fsync was active. LOAD_STARTED observed worker write_bytes=524288 before tests began. All 6 causal cases passed, including actual root-only binary publication; PROOF_EXIT=0 and PROOF_COMMAND_EXIT=0. The writer reported 45 HDD operations, 0 failed; whole-host sda counters rose by 125546496 written bytes / 910 write requests during the proof (includes tests and other host work, not a benchmark). Evidence retained on the control host: /home/kayg/.cache/codex-scratch/j196/{proof.log,stress.log,root-evidence.tar.gz}. The writer was reaped, and the root clone, binary and fixtures were removed: ROOT_PROOF_CLEANUP=PASS. Remaining target: /home/claude/worktrees/j196/target, retained only for the pending fast gate and to be deleted when it finishes.
kayg commented 2026-11-04 17:27:48 +00:00
Author
Owner
Copy link

J196 verification evidence for head 984b24301fabd6eaa6b196127661f4875e92a964, branch agent-fsync-offloop-j196.

  • Red: six causal tests failed on the original inline-write implementation with durable write held the reactor until the watchdog released it.
  • Green: six causal tests pass after offloading, including cancellation fencing.
  • scripts/verify quick: PASS, 146 passed / 0 failed / 9 existing ignored; workspace clippy and doc coverage passed.
  • Real disk proof: PASS as uid 0 on compute01 in a throwaway clone, with actual root-only binary publication and a live stress-ng --hdd writer. All six cases passed; writer reported 45 HDD operations and no failures. The root proof clone, binary, fixtures and writer have been cleaned up.
  • scripts/verify fast: still queued in a foreground SSH session. This is a progress receipt, not completion.

Attached are the original red output, complete quick output, root proof output, and the root proof archive containing the command script, stress log and kernel disk counters. These are raw evidence, not latency benchmarks. The sole remaining target directory is /home/claude/worktrees/j196/target on compute01, retained for fast verification and scheduled for deletion afterward.

J196 verification evidence for head `984b24301fabd6eaa6b196127661f4875e92a964`, branch `agent-fsync-offloop-j196`. - Red: six causal tests failed on the original inline-write implementation with `durable write held the reactor until the watchdog released it`. - Green: six causal tests pass after offloading, including cancellation fencing. - `scripts/verify quick`: PASS, 146 passed / 0 failed / 9 existing ignored; workspace clippy and doc coverage passed. - Real disk proof: PASS as uid 0 on compute01 in a throwaway clone, with actual root-only binary publication and a live `stress-ng --hdd` writer. All six cases passed; writer reported 45 HDD operations and no failures. The root proof clone, binary, fixtures and writer have been cleaned up. - `scripts/verify fast`: still queued in a foreground SSH session. This is a progress receipt, not completion. Attached are the original red output, complete quick output, root proof output, and the root proof archive containing the command script, stress log and kernel disk counters. These are raw evidence, not latency benchmarks. The sole remaining target directory is `/home/claude/worktrees/j196/target` on compute01, retained for fast verification and scheduled for deletion afterward.
quick.log
37 KiB
proof.log
2.9 KiB
red.log
6.3 KiB
root-evidence.tar.gz
2.6 KiB
kayg commented 2026-11-04 18:03:28 +00:00
Author
Owner
Copy link

j196 progress: head 984b24301fabd6eaa6b196127661f4875e92a964 remains pushed and unchanged. quick and root disk-load proof passed as recorded above. the foreground scripts/verify fast run acquired compute01 slot 3 after roughly 24 minutes in queue; it is still executing workspace tests, with no failure reported so far. full log: /home/claude/j196-evidence/fast.log; sole target: /home/claude/worktrees/j196/target. continuing foreground polls; cleanup and final gate receipt remain pending.

j196 progress: head `984b24301fabd6eaa6b196127661f4875e92a964` remains pushed and unchanged. quick and root disk-load proof passed as recorded above. the foreground `scripts/verify fast` run acquired compute01 slot 3 after roughly 24 minutes in queue; it is still executing workspace tests, with no failure reported so far. full log: `/home/claude/j196-evidence/fast.log`; sole target: `/home/claude/worktrees/j196/target`. continuing foreground polls; cleanup and final gate receipt remain pending.
kayg commented 2026-11-04 18:35:36 +00:00
Author
Owner
Copy link

j196 finished. branch agent-fsync-offloop-j196, pushed head 984b24301fabd6eaa6b196127661f4875e92a964 (base 2198148d788c32b39d9ac604fc24c9cee6e79968). local and VM checkouts are clean and match the pushed head. no merge or deployment; issue remains open.

all ten previously inline fsync calls now execute in awaited blocking tasks across these five paths:

  • access restoration: logins.rs:519, registry directory sync.
  • revocation marker: logins.rs:610,611, marker file then registry directory sync.
  • rootless reservation: rootless_sessions.rs:71,72, record file then directory sync.
  • rootless cleanup: rootless_sessions.rs:113, final directory sync outside the shell-cleanup timeout.
  • login binary publication/pruning: login_binary.rs:313,314,393,400, output file and publication/pruning directories.

the audit's four other calls were already safe: automation operations.rs:74,76 uses J163's blocking-pool save; startup_guard.rs:286,297 runs in the synchronous single-use helper process. no sync was removed. callers still await durability and receive IO errors. owned person fences prevent cancelled writes from being overtaken; reservations retain their fence through launch readiness.

six event-ordering tests failed first on the inline implementation, then passed after the fix. unrelated work on the same current-thread runtime must answer before a held save is released; no elapsed-time assertion decides success. cancellation also retains the person fence until the write finishes.

scripts/verify quick passed on this head before fast. verbatim summary:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| scope guard               | PASS         |
| fmt                       | PASS         |
| doc coverage ratchet      | PASS         |
| clippy (workspace)        | PASS         |
| unit tests (changed)      | PASS         |
+---------------------------+--------------+

QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done.
QUICK_EXIT=0

scripts/verify fast passed on compute01 through build-slot, on the attached full head SHA. verbatim Rust summary:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+

the subsequent compiler, tracer production build, transfer budget and TypeScript protocol steps also completed successfully. final output, verbatim:

 48 pass
 0 fail
 77248 expect() calls
Ran 48 tests across 12 files. [13.70s]
FAST_EXIT=0

static gate measurements, verbatim, x86_64 then aarch64:

PT_INTERP: absent; size: 23411248 bytes; baseline 22418256; delta +992992 bytes (report only, D243 amended 2026-11-01)
PT_INTERP: absent; size: 21271744 bytes; baseline 20453280; delta +818464 bytes (report only, D243 amended 2026-11-01)

these deltas are against the gate's recorded references, not isolated measurements of this commit's contribution. both ELF checks passed; the arm64 key-memory execution probe reports its expected native-execution SKIP on the x86 host.

real-environment proof and original red/quick logs: #347 (comment) . the six causal cases passed as uid 0 in a throwaway root clone while a live stress-ng --hdd 1 --hdd-bytes 512M --hdd-opts wr-seq,fsync worker wrote to disk. this exercised actual root binary publication. the worker completed 45 HDD operations with no failures. dev-02 was busy, so the authorized compute01 fallback was used. this is correctness evidence under load, not a latency benchmark.

cleanup confirmed: J196_ORPHANS=[], TARGET_AND_STATE_CLEANUP=PASS. deleted /home/claude/worktrees/j196/target and /home/claude/j196-state; root proof clone/binary/fixtures were already removed and its writer reaped. no pending J196 build or gate remains. small logs remain under /home/claude/j196-evidence and are attached here. no blockers or owner decision needed.

j196 finished. branch `agent-fsync-offloop-j196`, pushed head `984b24301fabd6eaa6b196127661f4875e92a964` (base `2198148d788c32b39d9ac604fc24c9cee6e79968`). local and VM checkouts are clean and match the pushed head. no merge or deployment; issue remains open. all ten previously inline fsync calls now execute in awaited blocking tasks across these five paths: - access restoration: `logins.rs:519`, registry directory sync. - revocation marker: `logins.rs:610,611`, marker file then registry directory sync. - rootless reservation: `rootless_sessions.rs:71,72`, record file then directory sync. - rootless cleanup: `rootless_sessions.rs:113`, final directory sync outside the shell-cleanup timeout. - login binary publication/pruning: `login_binary.rs:313,314,393,400`, output file and publication/pruning directories. the audit's four other calls were already safe: automation `operations.rs:74,76` uses J163's blocking-pool save; `startup_guard.rs:286,297` runs in the synchronous single-use helper process. no sync was removed. callers still await durability and receive IO errors. owned person fences prevent cancelled writes from being overtaken; reservations retain their fence through launch readiness. six event-ordering tests failed first on the inline implementation, then passed after the fix. unrelated work on the same current-thread runtime must answer before a held save is released; no elapsed-time assertion decides success. cancellation also retains the person fence until the write finishes. `scripts/verify quick` passed on this head before fast. verbatim summary: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | scope guard | PASS | | fmt | PASS | | doc coverage ratchet | PASS | | clippy (workspace) | PASS | | unit tests (changed) | PASS | +---------------------------+--------------+ QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done. QUICK_EXIT=0 ``` `scripts/verify fast` passed on compute01 through build-slot, on the attached full head SHA. verbatim Rust summary: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | static builds | PASS | +---------------------------+--------------+ ``` the subsequent compiler, tracer production build, transfer budget and TypeScript protocol steps also completed successfully. final output, verbatim: ```text 48 pass 0 fail 77248 expect() calls Ran 48 tests across 12 files. [13.70s] FAST_EXIT=0 ``` static gate measurements, verbatim, x86_64 then aarch64: ```text PT_INTERP: absent; size: 23411248 bytes; baseline 22418256; delta +992992 bytes (report only, D243 amended 2026-11-01) PT_INTERP: absent; size: 21271744 bytes; baseline 20453280; delta +818464 bytes (report only, D243 amended 2026-11-01) ``` these deltas are against the gate's recorded references, not isolated measurements of this commit's contribution. both ELF checks passed; the arm64 key-memory execution probe reports its expected native-execution SKIP on the x86 host. real-environment proof and original red/quick logs: https://git.kayg.org/kayg/infraternal/issues/347#issuecomment-78561 . the six causal cases passed as uid 0 in a throwaway root clone while a live `stress-ng --hdd 1 --hdd-bytes 512M --hdd-opts wr-seq,fsync` worker wrote to disk. this exercised actual root binary publication. the worker completed 45 HDD operations with no failures. dev-02 was busy, so the authorized compute01 fallback was used. this is correctness evidence under load, not a latency benchmark. cleanup confirmed: `J196_ORPHANS=[]`, `TARGET_AND_STATE_CLEANUP=PASS`. deleted `/home/claude/worktrees/j196/target` and `/home/claude/j196-state`; root proof clone/binary/fixtures were already removed and its writer reaped. no pending J196 build or gate remains. small logs remain under `/home/claude/j196-evidence` and are attached here. no blockers or owner decision needed.
fast.log
835 KiB
fast-head.txt
41 B
kayg commented 2026-11-04 18:39:49 +00:00
Author
Owner
Copy link

J212 starting review: branch agent-fsync-offloop-j196, head 984b24301f, base 984b243^. Plan (~45 minutes): compare J163, inspect five durability paths and cancellation/admission bounds, reproduce findings on compute01, post verdict. No changes to reviewed branch.

J212 starting review: branch agent-fsync-offloop-j196, head 984b24301fabd6eaa6b196127661f4875e92a964, base 984b243^. Plan (~45 minutes): compare J163, inspect five durability paths and cancellation/admission bounds, reproduce findings on compute01, post verdict. No changes to reviewed branch.
kayg commented 2026-11-04 18:43:07 +00:00
Author
Owner
Copy link

J212 progress: all ten syncs remain awaited and IO errors propagate. The new per-person fences cover queued/running marker and reservation writes; rootless cleanup moves its final directory sync outside the shell timeout. Investigating a P2 resource-bound concern: unrestricted distinct-person restore/revoke calls enqueue spawn_blocking closures, and publication waits on a global flock inside that pool. Unlike J163, ingress is not serialized. Review-only reproduction is staged in my separate j212 worktree on compute01, currently waiting for a build slot (queue position 5 at start). Reviewed branch remains unchanged.

J212 progress: all ten syncs remain awaited and IO errors propagate. The new per-person fences cover queued/running marker and reservation writes; rootless cleanup moves its final directory sync outside the shell timeout. Investigating a P2 resource-bound concern: unrestricted distinct-person restore/revoke calls enqueue spawn_blocking closures, and publication waits on a global flock inside that pool. Unlike J163, ingress is not serialized. Review-only reproduction is staged in my separate j212 worktree on compute01, currently waiting for a build slot (queue position 5 at start). Reviewed branch remains unchanged.
kayg commented 2026-11-04 18:53:15 +00:00
Author
Owner
Copy link

J212 review remains active in foreground. Compute01 build-slot wait is now about 12 minutes; root paths will be exercised there as requested. Static review completed: no new durability/error/fence-ordering defect identified; outstanding reproduction concerns admission of blocking work (queue ownership retains FDs, and global publication flock can occupy workers). No branch edits or full-gate rerun yet.

J212 review remains active in foreground. Compute01 build-slot wait is now about 12 minutes; root paths will be exercised there as requested. Static review completed: no new durability/error/fence-ordering defect identified; outstanding reproduction concerns admission of blocking work (queue ownership retains FDs, and global publication flock can occupy workers). No branch edits or full-gate rerun yet.
kayg commented 2026-11-04 19:03:25 +00:00
Author
Owner
Copy link

J212 independent current-thread test run on compute01 completed: 6 passed, 0 failed, TEST_EXIT=0. Both review probes compiled in my detached worktree; the root proof is now queued through build-slot. The reviewed branch/head is unchanged. Full test output will be attached with the verdict.

J212 independent current-thread test run on compute01 completed: 6 passed, 0 failed, TEST_EXIT=0. Both review probes compiled in my detached worktree; the root proof is now queued through build-slot. The reviewed branch/head is unchanged. Full test output will be attached with the verdict.
kayg commented 2026-11-04 19:12:51 +00:00
Author
Owner
Copy link

J212 progress: independent six causal current-thread tests passed on compute01, TEST_EXIT=0 (full output attached). Attached review-only patch contains the two resource-bound probes; those probes have compiled but have not run yet. The root run is queued again after git rejected a cross-owner local clone. A shallow public-HTTPS throwaway clone under /root/j212-proof now matches 984b24301fabd6eaa6b196127661f4875e92a964. Root proof script and pending output are under /root/j212-evidence/. My build target is /home/claude/worktrees/j212/target and will be removed after the proof. Reviewed branch remains unchanged. No verdict until reproduction completes. Still waiting in foreground as instructed.

J212 progress: independent six causal current-thread tests passed on compute01, `TEST_EXIT=0` (full output attached). Attached review-only patch contains the two resource-bound probes; **those probes have compiled but have not run yet**. The root run is queued again after git rejected a cross-owner local clone. A shallow public-HTTPS throwaway clone under `/root/j212-proof` now matches `984b24301fabd6eaa6b196127661f4875e92a964`. Root proof script and pending output are under `/root/j212-evidence/`. My build target is `/home/claude/worktrees/j212/target` and will be removed after the proof. Reviewed branch remains unchanged. No verdict until reproduction completes. Still waiting in foreground as instructed.
repro.patch
5.4 KiB
tests.log
4.2 KiB
kayg commented 2026-11-04 19:16:57 +00:00
Author
Owner
Copy link

J212 confirmed P2 on compute01 (single quick unit probe, no rebuild): after saturating a two-worker blocking pool, polling 1,024 real restore_at calls for distinct people and dropping all their async futures leaves all 1,024 person flocks retained by queued blocking closures. Verbatim: J212 submitted=1024 cancelled=1024 retained_fences=1024 blocking_workers=2. FLOOD_EXIT=0; 1 passed, 0 failed. This proves the queue/FD growth concern; it does not dispute the intentional cancellation fencing. Production has a 512-worker cap but no blocking-queue admission bound. The root-only real-publication flock probe remains queued through build-slot. Probe source is attached at comment 78851.

J212 confirmed P2 on compute01 (single quick unit probe, no rebuild): after saturating a two-worker blocking pool, polling 1,024 real restore_at calls for distinct people and dropping all their async futures leaves all 1,024 person flocks retained by queued blocking closures. Verbatim: J212 submitted=1024 cancelled=1024 retained_fences=1024 blocking_workers=2. FLOOD_EXIT=0; 1 passed, 0 failed. This proves the queue/FD growth concern; it does not dispute the intentional cancellation fencing. Production has a 512-worker cap but no blocking-queue admission bound. The root-only real-publication flock probe remains queued through build-slot. Probe source is attached at comment 78851.
kayg commented 2026-11-04 19:22:05 +00:00
Author
Owner
Copy link

J212 update: the review has exceeded the initial 45-minute estimate because of compute01 slot waits and the rejected cross-owner local-clone setup. The P2 queue/FD finding is now reproduced (1,024 queued owned fences retained after caller cancellation), and all six causal tests pass independently. Remaining: queued root publication/causal run, attach raw output, final verdict, remove my target and root clone. Continuing foreground waits; no branch changes.

J212 update: the review has exceeded the initial 45-minute estimate because of compute01 slot waits and the rejected cross-owner local-clone setup. The P2 queue/FD finding is now reproduced (1,024 queued owned fences retained after caller cancellation), and all six causal tests pass independently. Remaining: queued root publication/causal run, attach raw output, final verdict, remove my target and root clone. Continuing foreground waits; no branch changes.
kayg commented 2026-11-04 19:43:14 +00:00
Author
Owner
Copy link

J212 correctness review of agent-fsync-offloop-j196 at 984b24301fabd6eaa6b196127661f4875e92a964 (base 2198148d788c32b39d9ac604fc24c9cee6e79968).

REQUEST CHANGES: one P2. No P1 or P3 findings.

P2: bound admission to the blocking pool; its worker cap does not bound queued durable operations. logins.rs:486-493,545-553 submits a blocking closure after acquiring an owned person flock. daemon.rs:931-947 accepts distinct-person restore/revoke requests without an in-flight bound. Every queued closure retains its flock/file descriptor, including after its async caller is dropped. Tokio limits workers (the shipping runtime keeps the default 512), but its blocking task queue has no bound here. A stalled pool therefore turns a control-call flood into retained tasks/FDs until process resources are exhausted. This also contradicts the existing contention rationale in logins.rs:793-798 (flock waiters must not starve unrelated cleanup). Source inspection additionally shows publication (login_binary.rs:21-37,351-353) can consume workers waiting for the same blocking exclusive flock; unrelated marker writes share that pool. This is a consequence of the same missing admission bound, not a separately reproduced root-only finding.

This differs from J163/J176 on #334: that path accepts one automation call at a time, so it can have only one pending blocking save. Moving these concurrent paths to the same pool needs its own resource admission bound. Keep person fences owned until disk completion; add bounded admission before enqueuing work and avoid occupying shared workers merely waiting for another publisher. Add causal flood coverage, including revocation progress behind a held publisher.

Reproduction on compute01: the review-only probe fills a two-worker pool with held work, polls 1,024 actual restore_at calls for distinct people, then drops every async caller. All 1,024 queued closures retain their person flocks until the pool is released. No elapsed-time assertion is used. The two-worker setting exposes the same queue behavior without creating 512 workers on the shared VM.

J212 submitted=1024 cancelled=1024 retained_fences=1024 blocking_workers=2
FLOOD_EXIT=0

The full probe source and independent six-test output are attached at #347 (comment) . The additional root-only publication-starvation probe compiled, but was cancelled before execution after a long shared build-slot wait. No independent root-starvation result is claimed. I inspected J196’s original root proof.log: exact reviewed head, UID=0, six tests passed under a live disk writer, PROOF_EXIT=0. That is author evidence.

Other review results:

  • Restoration still unlinks the validated marker then fsyncs the registry before success.
  • Revocation still fsyncs the marker file and then the registry before cleanup; its owned person fence transfers into the task and back through cleanup.
  • Reservation still fsyncs its record and registry before returning; the returned reservation retains the person fence through spawn/readiness, and cancellation before return drops the unconsumed record after the disk work.
  • Rootless cleanup still bounds shell cleanup but awaits the final registry fsync outside that deadline. The person fence moves into that sync task.
  • Publication/pruning retains the synchronous algorithm inside the awaited closure: output fsync, atomic link, publication directory fsync, and pruning directory fsyncs. Existing locks still protect image lifetime.
  • IO errors and JoinErrors reach the caller. No new success-before-durability or revocation-ordering defect found in those paths.
  • The six new tests use default current-thread runtimes and held-write/event ordering. The 30 s watchdog is a hang guard rather than an elapsed-time performance assertion. They cover reactor responsiveness and one cancelled revocation, but not pool flooding.

No product code changed; the reviewed branch remains at the exact requested SHA (also verified with git ls-remote). Changes are test-only probes in detached review checkouts, retained as a patch attachment. Both detached review checkouts have been restored clean; no repository code change is proposed or retained. Under BRIEF-COMMON’s review/research guidance, I did not rerun the full fast gate. I inspected J196's retained fast-head.txt and fast.log; that is author evidence, not an independent full-gate run. The head receipt matches the reviewed SHA. Its gate output, verbatim:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
FAST_EXIT=0

Independent runs on compute01:

claude, build-slot:
  nix develop -c cargo test -j4 -p infraternal-agent --lib durable_tests -- --nocapture
  6 passed; 0 failed; 0 ignored; 151 filtered out
  TEST_EXIT=0
claude, one quick unit probe, ulimit -n 4096:
  INFRATERNAL_TEST_STATE_ROOT=/tmp/j212-state target/debug/deps/infraternal_agent-c976c94f3e887ef4 j212_restore_flood --nocapture
  J212 submitted=1024 cancelled=1024 retained_fences=1024 blocking_workers=2
  1 passed; 0 failed; 0 ignored; 156 filtered out
  FLOOD_EXIT=0

Cleanup was verified:

ROOT_EXTRA_DID_NOT_EXECUTE
ROOT_QUEUE_PROCESS_EXITED
ROOT_CLONE_CLEANUP=PASS
TARGET_CLEANUP=PASS

The extra root run was stopped while queued (ROOT_COMMAND_EXIT=143). Its initial local-clone setup had failed Git’s cross-owner check before any test; the subsequent public HTTPS clone matched the requested head and has been removed. My VM target and temporary state are also removed; no J212 run remains active. Both detached review checkouts are clean, and the remote branch still names 984b24301fabd6eaa6b196127661f4875e92a964. No push, deployment or issue closure. The attached patch preserves the probes; its root-publication test is explicitly unexecuted.

J212 correctness review of `agent-fsync-offloop-j196` at `984b24301fabd6eaa6b196127661f4875e92a964` (base `2198148d788c32b39d9ac604fc24c9cee6e79968`). **REQUEST CHANGES: one P2. No P1 or P3 findings.** **P2: bound admission to the blocking pool; its worker cap does not bound queued durable operations.** `logins.rs:486-493,545-553` submits a blocking closure after acquiring an owned person flock. `daemon.rs:931-947` accepts distinct-person restore/revoke requests without an in-flight bound. Every queued closure retains its flock/file descriptor, including after its async caller is dropped. Tokio limits workers (the shipping runtime keeps the default 512), but its blocking task queue has no bound here. A stalled pool therefore turns a control-call flood into retained tasks/FDs until process resources are exhausted. This also contradicts the existing contention rationale in `logins.rs:793-798` (flock waiters must not starve unrelated cleanup). Source inspection additionally shows publication (`login_binary.rs:21-37,351-353`) can consume workers waiting for the same blocking exclusive flock; unrelated marker writes share that pool. This is a consequence of the same missing admission bound, not a separately reproduced root-only finding. This differs from J163/J176 on #334: that path accepts one automation call at a time, so it can have only one pending blocking save. Moving these concurrent paths to the same pool needs its own resource admission bound. Keep person fences owned until disk completion; add bounded admission before enqueuing work and avoid occupying shared workers merely waiting for another publisher. Add causal flood coverage, including revocation progress behind a held publisher. Reproduction on compute01: the review-only probe fills a two-worker pool with held work, polls 1,024 actual `restore_at` calls for distinct people, then drops every async caller. All 1,024 queued closures retain their person flocks until the pool is released. No elapsed-time assertion is used. The two-worker setting exposes the same queue behavior without creating 512 workers on the shared VM. ```text J212 submitted=1024 cancelled=1024 retained_fences=1024 blocking_workers=2 FLOOD_EXIT=0 ``` The full probe source and independent six-test output are attached at https://git.kayg.org/kayg/infraternal/issues/347#issuecomment-78851 . The additional root-only publication-starvation probe compiled, but was cancelled before execution after a long shared build-slot wait. No independent root-starvation result is claimed. I inspected J196’s original root `proof.log`: exact reviewed head, `UID=0`, six tests passed under a live disk writer, `PROOF_EXIT=0`. That is author evidence. Other review results: - Restoration still unlinks the validated marker then fsyncs the registry before success. - Revocation still fsyncs the marker file and then the registry before cleanup; its owned person fence transfers into the task and back through cleanup. - Reservation still fsyncs its record and registry before returning; the returned reservation retains the person fence through spawn/readiness, and cancellation before return drops the unconsumed record after the disk work. - Rootless cleanup still bounds shell cleanup but awaits the final registry fsync outside that deadline. The person fence moves into that sync task. - Publication/pruning retains the synchronous algorithm inside the awaited closure: output fsync, atomic link, publication directory fsync, and pruning directory fsyncs. Existing locks still protect image lifetime. - IO errors and JoinErrors reach the caller. No new success-before-durability or revocation-ordering defect found in those paths. - The six new tests use default current-thread runtimes and held-write/event ordering. The 30 s watchdog is a hang guard rather than an elapsed-time performance assertion. They cover reactor responsiveness and one cancelled revocation, but not pool flooding. No product code changed; the reviewed branch remains at the exact requested SHA (also verified with `git ls-remote`). Changes are test-only probes in detached review checkouts, retained as a patch attachment. Both detached review checkouts have been restored clean; no repository code change is proposed or retained. Under BRIEF-COMMON’s review/research guidance, I did not rerun the full fast gate. I inspected J196's retained `fast-head.txt` and `fast.log`; that is author evidence, not an independent full-gate run. The head receipt matches the reviewed SHA. Its gate output, verbatim: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | static builds | PASS | +---------------------------+--------------+ FAST_EXIT=0 ``` Independent runs on compute01: ```text claude, build-slot: nix develop -c cargo test -j4 -p infraternal-agent --lib durable_tests -- --nocapture 6 passed; 0 failed; 0 ignored; 151 filtered out TEST_EXIT=0 claude, one quick unit probe, ulimit -n 4096: INFRATERNAL_TEST_STATE_ROOT=/tmp/j212-state target/debug/deps/infraternal_agent-c976c94f3e887ef4 j212_restore_flood --nocapture J212 submitted=1024 cancelled=1024 retained_fences=1024 blocking_workers=2 1 passed; 0 failed; 0 ignored; 156 filtered out FLOOD_EXIT=0 ``` Cleanup was verified: ```text ROOT_EXTRA_DID_NOT_EXECUTE ROOT_QUEUE_PROCESS_EXITED ROOT_CLONE_CLEANUP=PASS TARGET_CLEANUP=PASS ``` The extra root run was stopped while queued (`ROOT_COMMAND_EXIT=143`). Its initial local-clone setup had failed Git’s cross-owner check before any test; the subsequent public HTTPS clone matched the requested head and has been removed. My VM target and temporary state are also removed; no J212 run remains active. Both detached review checkouts are clean, and the remote branch still names `984b24301fabd6eaa6b196127661f4875e92a964`. No push, deployment or issue closure. The attached patch preserves the probes; its root-publication test is explicitly unexecuted.
repro.patch
5.4 KiB
tests.log
4.2 KiB
flood.log
268 B
kayg commented 2026-11-04 19:46:18 +00:00
Author
Owner
Copy link

j196 review follow-up started from 984b24301fabd6eaa6b196127661f4875e92a964, same branch agent-fsync-offloop-j196. addressing J212 comment 78874: reproduce the 1,024-caller cancellation flood, bound both execution and queued ownership, avoid publisher-lock starvation, and keep already-started durable work fenced through completion. merge current origin/dev (e8e8d2ff79b328a71faa62e7f764d705a133eeb4), rerun the six ordering tests and flood regression, then quick and full fast in the foreground. estimate: 150 minutes including builds and uncertain shared queues. one target will be retained only for this follow-up and deleted at finish.

j196 review follow-up started from `984b24301fabd6eaa6b196127661f4875e92a964`, same branch `agent-fsync-offloop-j196`. addressing J212 comment 78874: reproduce the 1,024-caller cancellation flood, bound both execution and queued ownership, avoid publisher-lock starvation, and keep already-started durable work fenced through completion. merge current origin/dev (`e8e8d2ff79b328a71faa62e7f764d705a133eeb4`), rerun the six ordering tests and flood regression, then quick and full fast in the foreground. estimate: 150 minutes including builds and uncertain shared queues. one target will be retained only for this follow-up and deleted at finish.
kayg commented 2026-11-04 19:58:06 +00:00
Author
Owner
Copy link

verified J212 P2 against source: restore/revoke acquire person FDs before submitting uncapped closures; cancelled JoinHandles do not remove queued payload ownership. merged origin/dev into this branch as 300529570. tests are written; the original implementation is queued for the red run on compute01 (position 8 initially; both VMs busy). proposed scoped fix: at most two blocking submissions and sixteen cooperative waiters, busy before fence acquisition, synchronous release of cancelled unstarted payloads, and continued fencing of already-started durable writes. publisher flock contention will retry outside the worker pool. additional probes count owned queue bytes and check root revocation progress behind a held publisher. no production behavior changed yet.

verified J212 P2 against source: restore/revoke acquire person FDs before submitting uncapped closures; cancelled JoinHandles do not remove queued payload ownership. merged origin/dev into this branch as `300529570`. tests are written; the original implementation is queued for the red run on compute01 (position 8 initially; both VMs busy). proposed scoped fix: at most two blocking submissions and sixteen cooperative waiters, busy before fence acquisition, synchronous release of cancelled unstarted payloads, and continued fencing of already-started durable writes. publisher flock contention will retry outside the worker pool. additional probes count owned queue bytes and check root revocation progress behind a held publisher. no production behavior changed yet.
kayg commented 2026-11-04 20:11:23 +00:00
Author
Owner
Copy link

checkpoint: tests pushed as a WIP on agent-fsync-offloop-j196; no production fix yet. the 1,024-caller regression remains in a foreground build-slot queue on compute01, now with one waiting job ahead (both VMs busy). the queued worktree contains the same test and merged source. the sole target will be /home/claude/worktrees/j196/target. resource bounds and the root publisher-contention proof are prepared; red confirmation, implementation, quick and fast remain pending.

checkpoint: tests pushed as a WIP on `agent-fsync-offloop-j196`; no production fix yet. the 1,024-caller regression remains in a foreground build-slot queue on compute01, now with one waiting job ahead (both VMs busy). the queued worktree contains the same test and merged source. the sole target will be `/home/claude/worktrees/j196/target`. resource bounds and the root publisher-contention proof are prepared; red confirmation, implementation, quick and fast remain pending.
kayg commented 2026-11-04 20:22:06 +00:00
Author
Owner
Copy link

red confirmed on the merged source before the production change:

flood=1024 submitted_fds=1024 cancelled_fds=1024 retained_fences=1024
unbounded descriptor admission: 1024
RED_EXIT=101

the original implementation fails the descriptor/fence bound while both blocking workers are still held. attached is the complete failing test output. the bounded implementation is checkpointed as WIP 015fd621b; focused tests (the original six plus resource regressions) and quick are queued in the foreground. no passing result claimed yet.

red confirmed on the merged source before the production change: ```text flood=1024 submitted_fds=1024 cancelled_fds=1024 retained_fences=1024 unbounded descriptor admission: 1024 RED_EXIT=101 ``` the original implementation fails the descriptor/fence bound while both blocking workers are still held. attached is the complete failing test output. the bounded implementation is checkpointed as WIP `015fd621b`; focused tests (the original six plus resource regressions) and quick are queued in the foreground. no passing result claimed yet.
review-red.log
4.7 KiB
kayg commented 2026-11-04 20:30:23 +00:00
Author
Owner
Copy link

focused green on WIP 7e07892ab: all six original ordering/cancellation cases plus both resource regressions pass.

flood=4096 peak_owned_bytes=1179648 retained_owned_bytes=0
flood=1024 submitted_fds=18 cancelled_fds=0 retained_fences=0
test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 150 filtered out
GREEN_EXIT=0

owned bytes are actual deliberately sized payload allocations counted by lifetime, not an RSS/allocator-cache estimate. the queue admits at most eighteen operations, submits at most two to Tokio, and returns WouldBlock before overload opens another fence. unstarted payload cancellation is synchronous; started writes retain their fences. publication uses nonblocking flock attempts with cooperative contention retries. revocation carries its admission through cleanup and directory sync. quick is currently running in the same target; the root publisher-contention proof and final fast gate remain pending.

focused green on WIP `7e07892ab`: all six original ordering/cancellation cases plus both resource regressions pass. ```text flood=4096 peak_owned_bytes=1179648 retained_owned_bytes=0 flood=1024 submitted_fds=18 cancelled_fds=0 retained_fences=0 test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 150 filtered out GREEN_EXIT=0 ``` owned bytes are actual deliberately sized payload allocations counted by lifetime, not an RSS/allocator-cache estimate. the queue admits at most eighteen operations, submits at most two to Tokio, and returns WouldBlock before overload opens another fence. unstarted payload cancellation is synchronous; started writes retain their fences. publication uses nonblocking flock attempts with cooperative contention retries. revocation carries its admission through cleanup and directory sync. quick is currently running in the same target; the root publisher-contention proof and final fast gate remain pending.
review-green.log
2 KiB
kayg commented 2026-11-04 20:38:30 +00:00
Author
Owner
Copy link

atomic follow-up pushed: e591ef0ad1fce1d84f5ffa64e426e550805e71e0, after merge 300529570 of origin/dev e8e8d2ff79b328a71faa62e7f764d705a133eeb4. the two WIP commits were consolidated into one logical resource-bound fix. its tree a3041539690aa51a83cadc02fcd83b0d09dbb4b2 is identical to the quick-verified tree (148 passed, 0 failed, 10 ignored; workspace clippy and doc coverage passed). the final quick/fast run is queued on compute01, with head recorded before execution. the root disk-load and publisher-contention proof is separately queued through build-slot against the same head in /root/j196-proof; it will clean its clone, binary and fixtures on completion. both are being polled in the foreground. target remains /home/claude/worktrees/j196/target; no other target exists.

atomic follow-up pushed: `e591ef0ad1fce1d84f5ffa64e426e550805e71e0`, after merge `300529570` of origin/dev `e8e8d2ff79b328a71faa62e7f764d705a133eeb4`. the two WIP commits were consolidated into one logical resource-bound fix. its tree `a3041539690aa51a83cadc02fcd83b0d09dbb4b2` is identical to the quick-verified tree (148 passed, 0 failed, 10 ignored; workspace clippy and doc coverage passed). the final quick/fast run is queued on compute01, with head recorded before execution. the root disk-load and publisher-contention proof is separately queued through build-slot against the same head in `/root/j196-proof`; it will clean its clone, binary and fixtures on completion. both are being polled in the foreground. target remains `/home/claude/worktrees/j196/target`; no other target exists.
kayg commented 2026-11-04 20:50:40 +00:00
Author
Owner
Copy link

documentation-only amendment: final head is now 77ae647dd59d4b3a3235223f0e6ef0a74899ca7d. the comments explicitly distinguish cancelled unstarted payloads from started writes and document WouldBlock on queue overload. no executable behavior changed. both still-queued final runs have been updated to that head before execution; a single cached unit probe refreshes the copied test executable for the root proof. previous green/quick evidence remains attached; final quick/fast and root results remain pending.

documentation-only amendment: final head is now `77ae647dd59d4b3a3235223f0e6ef0a74899ca7d`. the comments explicitly distinguish cancelled unstarted payloads from started writes and document WouldBlock on queue overload. no executable behavior changed. both still-queued final runs have been updated to that head before execution; a single cached unit probe refreshes the copied test executable for the root proof. previous green/quick evidence remains attached; final quick/fast and root results remain pending.
kayg commented 2026-11-04 21:08:42 +00:00
Author
Owner
Copy link

foreground verification progress: pushed head remains 77ae647dd59d4b3a3235223f0e6ef0a74899ca7d, clean locally and on compute01. the final quick/fast run has waited about thirty minutes for a slot and is now first in the waiting queue; the root proof is immediately behind it. both VMs remain fully occupied, with further queues. continuing foreground polls. no pending code changes and no passing final gate claimed. prior quick passed 148 enabled tests, workspace clippy and doc coverage; the final-head cached flood probe again reports 18 peak descriptors and zero retained descriptors/fences. root proof clone and binary are pinned to this head and will be removed on completion. sole target: /home/claude/worktrees/j196/target; logs: /home/claude/j196-evidence/review/.

foreground verification progress: pushed head remains `77ae647dd59d4b3a3235223f0e6ef0a74899ca7d`, clean locally and on compute01. the final quick/fast run has waited about thirty minutes for a slot and is now first in the waiting queue; the root proof is immediately behind it. both VMs remain fully occupied, with further queues. continuing foreground polls. no pending code changes and no passing final gate claimed. prior quick passed 148 enabled tests, workspace clippy and doc coverage; the final-head cached flood probe again reports 18 peak descriptors and zero retained descriptors/fences. root proof clone and binary are pinned to this head and will be removed on completion. sole target: `/home/claude/worktrees/j196/target`; logs: `/home/claude/j196-evidence/review/`.
kayg commented 2026-11-04 21:27:17 +00:00
Author
Owner
Copy link

foreground continuation recovered the clean pushed head 77ae647dd59d4b3a3235223f0e6ef0a74899ca7d and read J212 comment 78874. final quick has completed successfully (QUICK_EXIT=0, all four rows PASS); fast is actively progressing through workspace integration tests on the recorded same head, without a reported failure so far. root loaded proof remains queued through build-slot. no duplicate build started. continuing foreground polls; final gate, loaded proof, attachments and target cleanup remain pending.

foreground continuation recovered the clean pushed head `77ae647dd59d4b3a3235223f0e6ef0a74899ca7d` and read J212 comment 78874. final quick has completed successfully (`QUICK_EXIT=0`, all four rows PASS); fast is actively progressing through workspace integration tests on the recorded same head, without a reported failure so far. root loaded proof remains queued through build-slot. no duplicate build started. continuing foreground polls; final gate, loaded proof, attachments and target cleanup remain pending.
kayg commented 2026-11-04 21:29:24 +00:00
Author
Owner
Copy link

root loaded proof completed on final head 77ae647dd59d4b3a3235223f0e6ef0a74899ca7d on compute01 through build-slot. live disk writes were confirmed through /proc before starting causal tests. all eight durable tests (six original ordering tests plus descriptor and memory floods) passed; the root-only held-publisher revocation probe also passed. temporary root clone, copied executable and fixtures removed. fast continues through workspace tests. complete root output:

build-slot: [21:38:30] all 4 slots busy; queue position 7, waiting...
build-slot: [22:28:03] got slot 3/4
HEAD=77ae647dd59d4b3a3235223f0e6ef0a74899ca7d
UID=0
Linux infraternal-dev-01-compute01 6.12.113+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.113-1 (2026-10-15) x86_64 GNU/Linux
b77ba6782de83c0641fb127cdd471bd2dc33dca4825d50c51b6119779bd329b8  /root/j196-proof/test-bin
7.55 14.21 18.47 4/480 2226490
LOAD_STARTED pid=2226499 write_bytes=262144

running 8 tests
test durable_io::tests::cancelled_flood_bounds_owned_memory ... 
running 1 test
test durable_io::tests::cancelled_flood_bounds_owned_memory ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 157 filtered out; finished in 0.01s

flood=4096 peak_owned_bytes=1179648 retained_owned_bytes=0
ok
test login_binary::durable_tests::image_durable_write_leaves_the_reactor_responsive ... ok
test logins::durable_tests::cancelled_durable_write_keeps_the_person_fenced ... ok
test logins::durable_tests::cancelled_restore_flood_bounds_descriptors ... 
running 1 test
test logins::durable_tests::cancelled_restore_flood_bounds_descriptors ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 157 filtered out; finished in 0.01s

flood=1024 submitted_fds=18 cancelled_fds=0 retained_fences=0
ok
test logins::durable_tests::login_durable_writes_leave_the_reactor_responsive ... ok
test logins::durable_tests::restore_durable_write_leaves_the_reactor_responsive ... ok
test rootless_sessions::durable_tests::cleanup_durable_write_leaves_the_reactor_responsive ... ok
test rootless_sessions::durable_tests::reservation_durable_write_leaves_the_reactor_responsive ... ok

test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 150 filtered out; finished in 0.54s


running 1 test
test logins::tests::publisher_contention_does_not_starve_revocation ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 157 filtered out; finished in 0.07s

ROOT_PUBLISHER_EXIT=0
stress-ng: info:  [2226492] setting to a 2 mins run per stressor
stress-ng: info:  [2226492] dispatching hogs: 1 hdd
stress-ng: info:  [2226499] hdd: using 512MB file system space per stressor instance (total 512MB of 132.98GB available file system space)
stress-ng: info:  [2226492] stopping 1 stressors
stress-ng: metrc: [2226492] stressor       bogo ops real time  usr time  sys time   bogo ops/s     bogo ops/s
stress-ng: metrc: [2226492]                           (secs)    (secs)    (secs)   (real time) (usr+sys time)
stress-ng: metrc: [2226492] hdd                  47      0.64      0.00      0.01        73.04        4305.61
stress-ng: info:  [2226492] skipped: 0
stress-ng: info:  [2226492] passed: 1: hdd (1)
stress-ng: info:  [2226492] failed: 0
stress-ng: info:  [2226492] metrics untrustworthy: 0
stress-ng: info:  [2226492] successful run completed in 0 secs
PROOF_EXIT=0
ROOT_COMMAND_EXIT=0
ROOT_PROOF_CLEANUP=PASS
root loaded proof completed on final head `77ae647dd59d4b3a3235223f0e6ef0a74899ca7d` on compute01 through build-slot. live disk writes were confirmed through /proc before starting causal tests. all eight durable tests (six original ordering tests plus descriptor and memory floods) passed; the root-only held-publisher revocation probe also passed. temporary root clone, copied executable and fixtures removed. fast continues through workspace tests. complete root output: ```text build-slot: [21:38:30] all 4 slots busy; queue position 7, waiting... build-slot: [22:28:03] got slot 3/4 HEAD=77ae647dd59d4b3a3235223f0e6ef0a74899ca7d UID=0 Linux infraternal-dev-01-compute01 6.12.113+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.113-1 (2026-10-15) x86_64 GNU/Linux b77ba6782de83c0641fb127cdd471bd2dc33dca4825d50c51b6119779bd329b8 /root/j196-proof/test-bin 7.55 14.21 18.47 4/480 2226490 LOAD_STARTED pid=2226499 write_bytes=262144 running 8 tests test durable_io::tests::cancelled_flood_bounds_owned_memory ... running 1 test test durable_io::tests::cancelled_flood_bounds_owned_memory ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 157 filtered out; finished in 0.01s flood=4096 peak_owned_bytes=1179648 retained_owned_bytes=0 ok test login_binary::durable_tests::image_durable_write_leaves_the_reactor_responsive ... ok test logins::durable_tests::cancelled_durable_write_keeps_the_person_fenced ... ok test logins::durable_tests::cancelled_restore_flood_bounds_descriptors ... running 1 test test logins::durable_tests::cancelled_restore_flood_bounds_descriptors ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 157 filtered out; finished in 0.01s flood=1024 submitted_fds=18 cancelled_fds=0 retained_fences=0 ok test logins::durable_tests::login_durable_writes_leave_the_reactor_responsive ... ok test logins::durable_tests::restore_durable_write_leaves_the_reactor_responsive ... ok test rootless_sessions::durable_tests::cleanup_durable_write_leaves_the_reactor_responsive ... ok test rootless_sessions::durable_tests::reservation_durable_write_leaves_the_reactor_responsive ... ok test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 150 filtered out; finished in 0.54s running 1 test test logins::tests::publisher_contention_does_not_starve_revocation ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 157 filtered out; finished in 0.07s ROOT_PUBLISHER_EXIT=0 stress-ng: info: [2226492] setting to a 2 mins run per stressor stress-ng: info: [2226492] dispatching hogs: 1 hdd stress-ng: info: [2226499] hdd: using 512MB file system space per stressor instance (total 512MB of 132.98GB available file system space) stress-ng: info: [2226492] stopping 1 stressors stress-ng: metrc: [2226492] stressor bogo ops real time usr time sys time bogo ops/s bogo ops/s stress-ng: metrc: [2226492] (secs) (secs) (secs) (real time) (usr+sys time) stress-ng: metrc: [2226492] hdd 47 0.64 0.00 0.01 73.04 4305.61 stress-ng: info: [2226492] skipped: 0 stress-ng: info: [2226492] passed: 1: hdd (1) stress-ng: info: [2226492] failed: 0 stress-ng: info: [2226492] metrics untrustworthy: 0 stress-ng: info: [2226492] successful run completed in 0 secs PROOF_EXIT=0 ROOT_COMMAND_EXIT=0 ROOT_PROOF_CLEANUP=PASS ```
root-proof.log
2.8 KiB
root-evidence.tar.gz
1.9 KiB
kayg commented 2026-11-04 21:55:17 +00:00
Author
Owner
Copy link

foreground fast progress on unchanged final head 77ae647dd59d4b3a3235223f0e6ef0a74899ca7d: workspace tests and the later SSH/shim fixtures, cargo-deny, gate architecture and doc coverage completed. optimized x86 static build completed; aarch64 static compilation is now active. root loaded proof already passed and is attached in comment 78987. continuing foreground polls; target deletion follows gate completion.

foreground fast progress on unchanged final head `77ae647dd59d4b3a3235223f0e6ef0a74899ca7d`: workspace tests and the later SSH/shim fixtures, cargo-deny, gate architecture and doc coverage completed. optimized x86 static build completed; aarch64 static compilation is now active. root loaded proof already passed and is attached in comment 78987. continuing foreground polls; target deletion follows gate completion.
kayg commented 2026-11-04 22:12:58 +00:00
Author
Owner
Copy link

final fast completed on 77ae647dd59d4b3a3235223f0e6ef0a74899ca7d: all rows pass except workspace test. The sole failure is desktop helper::startup_recovery_tests::delta_before_startup_keyframe_requests_repair_before_delivery (startup delta must trigger a repair keyframe: Disconnected, helper.rs:510). That desktop code is identical to the merged dev base. A focused unmodified-base reproduction is in progress; the initial default-feature probe selected zero tests and is not counted as evidence. The enabled-feature probe selects the failing helper test. No unrelated code change made. Gate table verbatim:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | FAIL (101)   |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
FAST_EXIT=1
final fast completed on `77ae647dd59d4b3a3235223f0e6ef0a74899ca7d`: all rows pass except workspace test. The sole failure is desktop `helper::startup_recovery_tests::delta_before_startup_keyframe_requests_repair_before_delivery` (`startup delta must trigger a repair keyframe: Disconnected`, helper.rs:510). That desktop code is identical to the merged dev base. A focused unmodified-base reproduction is in progress; the initial default-feature probe selected zero tests and is not counted as evidence. The enabled-feature probe selects the failing helper test. No unrelated code change made. Gate table verbatim: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | FAIL (101) | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | static builds | PASS | +---------------------------+--------------+ FAST_EXIT=1 ```
kayg commented 2026-11-04 22:18:17 +00:00
Author
Owner
Copy link

J196 follow-up implementation and requested verification finished; full gate is NOT green.

Branch agent-fsync-offloop-j196, pushed head 77ae647dd59d4b3a3235223f0e6ef0a74899ca7d. Merge 300529570 incorporates the then-current origin/dev e8e8d2ff79b328a71faa62e7f764d705a133eeb4; fix 77ae647dd is one atomic, documented change. Local and VM worktrees are clean, and remote branch SHA matches.

The durable queue now admits two submitted blocking jobs plus sixteen cooperative waiters. Overload returns WouldBlock before opening fences or copying request data. Cancelling unstarted work synchronously drops its owned payload/fences; submitted placeholders retain their two execution slots until consumed, preventing repeated cancellation from growing Tokio's internal queue. Started writes retain their fences through completion. Revocation carries admission through cleanup/final sync. Publisher flock contention retries cooperatively outside shared workers.

Causal regression evidence (no stopwatch assertions): original 1,024-caller flood failed first with 1,024 retained descriptors/fences. Fixed flood peaks at 18 descriptors, then retains zero descriptors/fences after caller cancellation. Four 1,024-call memory waves peak at 1,179,648 owned bytes and retain zero. The six original ordering tests plus both flood regressions pass. Root proof on this exact head passed all eight tests under a confirmed live stress-ng hdd writer and passed the held-publisher revocation-progress test. Full root evidence is attached in comment 78987.

scripts/verify quick passed on the final head. scripts/verify fast ran verbatim in the foreground on compute01; its sole failure was desktop helper::startup_recovery_tests::delta_before_startup_keyframe_requests_repair_before_delivery, helper.rs:510, startup delta must trigger a repair keyframe: Disconnected. Desktop source is unchanged from the merged base. A feature-enabled isolated baseline probe passed, and six full 39-test desktop suites on unmodified merged dev passed. The exact desktop binary from the failed final-head gate then passed all 39 tests on rerun. Therefore this is observed intermittence, not a proven baseline failure or an overall passing gate. No unrelated desktop code changed. The initial default-feature baseline probe selected zero tests and is explicitly excluded from evidence.

Static builds passed: x86_64 23,511,472 bytes (gate baseline delta +1,093,216); aarch64 21,369,104 bytes (+915,824); no PT_INTERP. These are the gate's recorded baseline deltas, not a separately rebuilt current-dev comparison.

Full gate output, verbatim:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | FAIL (101)   |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
FAST_EXIT=1

Cleanup: TARGET_CLEANUP=PASS; /home/claude/worktrees/j196/target and /home/claude/j196-state removed. Root clone, executable and load fixtures were removed (ROOT_PROOF_CLEANUP=PASS). No J196 run remains active, no local target exists, and small evidence logs are retained under /home/claude/j196-evidence/review/ and on this issue. No merge to dev, deployment or issue closure. Remaining limitation for the orchestrator: the full fast gate has one unresolved intermittent desktop test failure; it must not be represented as green.

J196 follow-up implementation and requested verification finished; full gate is NOT green. Branch `agent-fsync-offloop-j196`, pushed head `77ae647dd59d4b3a3235223f0e6ef0a74899ca7d`. Merge `300529570` incorporates the then-current origin/dev `e8e8d2ff79b328a71faa62e7f764d705a133eeb4`; fix `77ae647dd` is one atomic, documented change. Local and VM worktrees are clean, and remote branch SHA matches. The durable queue now admits two submitted blocking jobs plus sixteen cooperative waiters. Overload returns WouldBlock before opening fences or copying request data. Cancelling unstarted work synchronously drops its owned payload/fences; submitted placeholders retain their two execution slots until consumed, preventing repeated cancellation from growing Tokio's internal queue. Started writes retain their fences through completion. Revocation carries admission through cleanup/final sync. Publisher flock contention retries cooperatively outside shared workers. Causal regression evidence (no stopwatch assertions): original 1,024-caller flood failed first with 1,024 retained descriptors/fences. Fixed flood peaks at 18 descriptors, then retains zero descriptors/fences after caller cancellation. Four 1,024-call memory waves peak at 1,179,648 owned bytes and retain zero. The six original ordering tests plus both flood regressions pass. Root proof on this exact head passed all eight tests under a confirmed live stress-ng hdd writer and passed the held-publisher revocation-progress test. Full root evidence is attached in comment 78987. `scripts/verify quick` passed on the final head. `scripts/verify fast` ran verbatim in the foreground on compute01; its sole failure was desktop `helper::startup_recovery_tests::delta_before_startup_keyframe_requests_repair_before_delivery`, helper.rs:510, `startup delta must trigger a repair keyframe: Disconnected`. Desktop source is unchanged from the merged base. A feature-enabled isolated baseline probe passed, and six full 39-test desktop suites on unmodified merged dev passed. The exact desktop binary from the failed final-head gate then passed all 39 tests on rerun. Therefore this is observed intermittence, not a proven baseline failure or an overall passing gate. No unrelated desktop code changed. The initial default-feature baseline probe selected zero tests and is explicitly excluded from evidence. Static builds passed: x86_64 23,511,472 bytes (gate baseline delta +1,093,216); aarch64 21,369,104 bytes (+915,824); no PT_INTERP. These are the gate's recorded baseline deltas, not a separately rebuilt current-dev comparison. Full gate output, verbatim: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | FAIL (101) | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | static builds | PASS | +---------------------------+--------------+ FAST_EXIT=1 ``` Cleanup: `TARGET_CLEANUP=PASS`; `/home/claude/worktrees/j196/target` and `/home/claude/j196-state` removed. Root clone, executable and load fixtures were removed (`ROOT_PROOF_CLEANUP=PASS`). No J196 run remains active, no local target exists, and small evidence logs are retained under `/home/claude/j196-evidence/review/` and on this issue. No merge to dev, deployment or issue closure. Remaining limitation for the orchestrator: the full fast gate has one unresolved intermittent desktop test failure; it must not be represented as green.
fast.log
515 KiB
quick-final.log
25 KiB
green.log
2 KiB
red.log
4.7 KiB
final-focused.log
1.2 KiB
final-head.txt
41 B
baseline-desktop-enabled.log
2 KiB
baseline-desktop-suite.log
4.1 KiB
baseline-desktop-repeat.log
18 KiB
head-desktop-rerun.log
3.6 KiB
kayg commented 2026-11-04 22:19:54 +00:00
Author
Owner
Copy link

J212 delta re-review starting on agent-fsync-offloop-j196 at 77ae647dd5. Scope: repair commit only (77ae647dd^..77ae647dd), excluding the dev merge and previously reviewed 984b24301 code. Plan (~35 minutes plus slot waits): trace all admission sites and full-queue behavior, test revocation progress, rerun the original flood reproduction on compute01, post verdict. No branch changes.

J212 delta re-review starting on agent-fsync-offloop-j196 at 77ae647dd59d4b3a3235223f0e6ef0a74899ca7d. Scope: repair commit only (77ae647dd^..77ae647dd), excluding the dev merge and previously reviewed 984b24301 code. Plan (~35 minutes plus slot waits): trace all admission sites and full-queue behavior, test revocation progress, rerun the original flood reproduction on compute01, post verdict. No branch changes.
kayg commented 2026-11-04 22:30:14 +00:00
Author
Owner
Copy link

J212 delta source finding, pending compute reproduction: revoke_at now calls the same durable_io::admit() as ordinary saves (logins.rs:547). At 18 admitted ordinary operations, try_acquire returns WouldBlock before revoke_marker. daemon.rs maps this to ResourceUnavailable; remote_guard.rs:97-106 retries only three times, and remote_authority.rs:168-173 continues retirement with cleanup unconfirmed. Therefore the full-queue path can discard the host marker/shell cleanup request instead of retaining it for later completion. All ten offloaded syncs use bounded run() and cleanup reuses admission; no nested-permit deadlock found. Original flood and full-queue revocation probes are staged in my detached j212 checkout and queued on compute01 (both VMs busy). Reviewed branch unchanged; no verdict until run evidence.

J212 delta source finding, pending compute reproduction: revoke_at now calls the same durable_io::admit() as ordinary saves (logins.rs:547). At 18 admitted ordinary operations, try_acquire returns WouldBlock before revoke_marker. daemon.rs maps this to ResourceUnavailable; remote_guard.rs:97-106 retries only three times, and remote_authority.rs:168-173 continues retirement with cleanup unconfirmed. Therefore the full-queue path can discard the host marker/shell cleanup request instead of retaining it for later completion. All ten offloaded syncs use bounded run() and cleanup reuses admission; no nested-permit deadlock found. Original flood and full-queue revocation probes are staged in my detached j212 checkout and queued on compute01 (both VMs busy). Reviewed branch unchanged; no verdict until run evidence.
kayg commented 2026-11-04 22:43:10 +00:00
Author
Owner
Copy link

J212 delta re-review progress: source inspection of repair 300529570..77ae647dd is complete, excluding the dev merge. All ten offloaded fsyncs use the bounded runner; rootless cleanup reuses admission; no nested-permit deadlock identified. Full-queue revocation remains a candidate P1: revoke_at shares fallible ordinary admission, and the caller gives up after three failures and continues retirement with cleanup unconfirmed.

Attached are the two review-only probes. They have not run yet: one reruns the original 1,024-call cancellation pattern; the other fills all 18 ordinary slots with real marker restorations, polls three revocation requests, drains ordinary work, and fails if no refusal marker was published. Both are causal checks, with no elapsed-time assertion. The probe preserves pending revocations through draining, so a queued priority implementation can complete.

The compute01 build/test is queued through build-slot in a foreground SSH session. Both VM pools were busy; it has advanced from eighth to fifth. Log: /home/claude/j212-delta.log; source: detached /home/claude/worktrees/j212 at the exact reviewed head. No branch change or push. Final verdict will follow run evidence, cleanup and attachments.

J212 delta re-review progress: source inspection of repair `300529570..77ae647dd` is complete, excluding the dev merge. All ten offloaded fsyncs use the bounded runner; rootless cleanup reuses admission; no nested-permit deadlock identified. Full-queue revocation remains a candidate P1: `revoke_at` shares fallible ordinary admission, and the caller gives up after three failures and continues retirement with cleanup unconfirmed. Attached are the two review-only probes. They have **not run yet**: one reruns the original 1,024-call cancellation pattern; the other fills all 18 ordinary slots with real marker restorations, polls three revocation requests, drains ordinary work, and fails if no refusal marker was published. Both are causal checks, with no elapsed-time assertion. The probe preserves pending revocations through draining, so a queued priority implementation can complete. The compute01 build/test is queued through build-slot in a foreground SSH session. Both VM pools were busy; it has advanced from eighth to fifth. Log: `/home/claude/j212-delta.log`; source: detached `/home/claude/worktrees/j212` at the exact reviewed head. No branch change or push. Final verdict will follow run evidence, cleanup and attachments.
probes.patch
5.7 KiB
kayg commented 2026-11-04 22:58:48 +00:00
Author
Owner
Copy link

J212 delta re-review: REQUEST CHANGES, one P1. The original cancellation-resource P2 is fixed.

Scope: repair 30052957084230e7995b6f348c2ccd34dd5b0f51..77ae647dd59d4b3a3235223f0e6ef0a74899ca7d, excluding the dev merge and unchanged original implementation. Reviewed branch unchanged. All reproduction runs below were on compute01 in my detached review worktree, with test-only probes attached.

P1: ordinary admission saturation discards revocation before publishing its refusal marker. crates/infraternal-agent/src/logins.rs:547 uses the same fallible durable_io::admit() as restoration/publication. durable_io.rs:31-35 immediately returns WouldBlock when all 18 admissions are owned. Admission is acquired before waiting on person fences, so 18 ordinary restorations contending on a single person's fence can reject revocation for an unrelated person even with no disk job executing. There is no retained revocation intent or later wakeup.

The unchanged caller makes this consequential: remote_guard.rs:97-105 attempts host-login revocation only three times; remote_authority.rs:168-173 warns when unconfirmed and continues authority retirement. This does not mean server authority remains granted. It means the host-login refusal marker and cleanup can be omitted permanently after those attempts, contrary to the requirement that revocation never be dropped or delayed indefinitely behind ordinary saves.

Deterministic compute01 proof: hold person A's fence, poll 18 restorations until each is pending, then attempt person B's revocation three times. Release all ordinary work and check B again. No sleeps or elapsed-time assertions:

J212_DELTA ordinary_fence_waiters=18 revoke_errors=[WouldBlock, WouldBlock, WouldBlock] accepted_pending=0
J212_DELTA ordinary_queue_drained=true access_still_permitted=true
J212_DELTA explicit_retry_revoked=true

The regression assertion fails after fixture cleanup. A fourth, explicit request works, proving the earlier requests were rejected rather than retained. Please give revocation bounded ownership and progress independent of ordinary admission saturation, including through the actual finite-retry caller; simply sharing the ordinary busy response is insufficient. Keep the cancellation/resource bounds.

Other delta conclusions:

  • All ten moved syncs across restoration, revocation markers, rootless reservation/cleanup, and login-binary publication/pruning go through bounded admission/execution. Rootless revocation reuses its admission rather than acquiring a nested one. No circular semaphore deadlock found.
  • At most two submitted blocking jobs and sixteen additional admitted operations are retained. Cancelled unstarted payloads release their fences synchronously; submitted placeholders retain admission/execution until consumed, bounding repeated-cancellation queue nodes. Started writes retain their fences and still propagate file/directory sync errors. Cooperative publisher flock retries avoid parking blocking workers, but can occupy ordinary admission while contended.
  • Accepted execution waiters use the semaphore queue; rejected admissions have no fairness or progress guarantee. Thus the bound is effective, but ordinary sustained contention can repeatedly exclude revocation.

Re-ran the original 1,024-call cancellation reproduction with both blocking workers explicitly held, then checked every person fence:

J212_DELTA original_submitted=1024 cancelled=1024 retained_fences=0

Also re-ran the repair's exact flood tests from the newly compiled binary:

flood=1024 submitted_fds=18 cancelled_fds=0 retained_fences=0
flood=4096 peak_owned_bytes=1179648 retained_owned_bytes=0

Both author flood tests passed. The two independent delta probes report 1 passed; 1 failed (exit 101), with the failure being the P1 above. These probes use barriers, explicit polling and lock state, not elapsed-time thresholds. Full logs and the final probe patch are attached; this patch supersedes the pending probe attached in comment 79044.

No production code changed and no new full fast gate was run for this read-only review. Author evidence in comment 79024 reports quick/focused checks passed but full fast remained FAST_EXIT=1, with the desktop startup-recovery test failing intermittently. I have not established that failure on baseline and am not calling the full gate green.

J212 delta re-review: **REQUEST CHANGES**, one **P1**. The original cancellation-resource P2 is fixed. Scope: repair `30052957084230e7995b6f348c2ccd34dd5b0f51..77ae647dd59d4b3a3235223f0e6ef0a74899ca7d`, excluding the dev merge and unchanged original implementation. Reviewed branch unchanged. All reproduction runs below were on compute01 in my detached review worktree, with test-only probes attached. **P1: ordinary admission saturation discards revocation before publishing its refusal marker.** `crates/infraternal-agent/src/logins.rs:547` uses the same fallible `durable_io::admit()` as restoration/publication. `durable_io.rs:31-35` immediately returns `WouldBlock` when all 18 admissions are owned. Admission is acquired before waiting on person fences, so 18 ordinary restorations contending on a single person's fence can reject revocation for an unrelated person even with no disk job executing. There is no retained revocation intent or later wakeup. The unchanged caller makes this consequential: `remote_guard.rs:97-105` attempts host-login revocation only three times; `remote_authority.rs:168-173` warns when unconfirmed and continues authority retirement. This does not mean server authority remains granted. It means the host-login refusal marker and cleanup can be omitted permanently after those attempts, contrary to the requirement that revocation never be dropped or delayed indefinitely behind ordinary saves. Deterministic compute01 proof: hold person A's fence, poll 18 restorations until each is pending, then attempt person B's revocation three times. Release all ordinary work and check B again. No sleeps or elapsed-time assertions: ```text J212_DELTA ordinary_fence_waiters=18 revoke_errors=[WouldBlock, WouldBlock, WouldBlock] accepted_pending=0 J212_DELTA ordinary_queue_drained=true access_still_permitted=true J212_DELTA explicit_retry_revoked=true ``` The regression assertion fails after fixture cleanup. A fourth, explicit request works, proving the earlier requests were rejected rather than retained. Please give revocation bounded ownership and progress independent of ordinary admission saturation, including through the actual finite-retry caller; simply sharing the ordinary busy response is insufficient. Keep the cancellation/resource bounds. Other delta conclusions: - All ten moved syncs across restoration, revocation markers, rootless reservation/cleanup, and login-binary publication/pruning go through bounded admission/execution. Rootless revocation reuses its admission rather than acquiring a nested one. No circular semaphore deadlock found. - At most two submitted blocking jobs and sixteen additional admitted operations are retained. Cancelled unstarted payloads release their fences synchronously; submitted placeholders retain admission/execution until consumed, bounding repeated-cancellation queue nodes. Started writes retain their fences and still propagate file/directory sync errors. Cooperative publisher flock retries avoid parking blocking workers, but can occupy ordinary admission while contended. - Accepted execution waiters use the semaphore queue; rejected admissions have no fairness or progress guarantee. Thus the bound is effective, but ordinary sustained contention can repeatedly exclude revocation. Re-ran the original 1,024-call cancellation reproduction with both blocking workers explicitly held, then checked every person fence: ```text J212_DELTA original_submitted=1024 cancelled=1024 retained_fences=0 ``` Also re-ran the repair's exact flood tests from the newly compiled binary: ```text flood=1024 submitted_fds=18 cancelled_fds=0 retained_fences=0 flood=4096 peak_owned_bytes=1179648 retained_owned_bytes=0 ``` Both author flood tests passed. The two independent delta probes report `1 passed; 1 failed` (exit 101), with the failure being the P1 above. These probes use barriers, explicit polling and lock state, not elapsed-time thresholds. Full logs and the final probe patch are attached; this patch supersedes the pending probe attached in comment 79044. No production code changed and no new full fast gate was run for this read-only review. Author evidence in comment 79024 reports quick/focused checks passed but full fast remained **`FAST_EXIT=1`**, with the desktop startup-recovery test failing intermittently. I have not established that failure on baseline and am not calling the full gate green.
j212-delta.log
4.6 KiB
j212-author-flood.log
867 B
probes.patch
5 KiB
kayg commented 2026-11-04 23:01:37 +00:00
Author
Owner
Copy link

J196 third-round repair explicitly requested by the orchestrator, addressing J212 P1 comment 79060. Starting head 77ae647dd59d4b3a3235223f0e6ef0a74899ca7d, branch agent-fsync-offloop-j196; fetched origin/dev 01ee7103a4ea3221cdff4fcfd59195fcc1f77032 for the requested merge. Plan: reproduce the 18 ordinary fence-waiter/three revocation interleave first; reserve security progress independently of ordinary admission and deny before disk waits, preserve bounded ownership and cancellation; test persistence failure/retry as well as six ordering cases; quick then full fast in foreground; push atomic repair and delete target. Estimate 180 minutes including fresh builds/shared queues. This user-directed repair supersedes the common third-review stop rule. No merge to dev or deployment.

J196 third-round repair explicitly requested by the orchestrator, addressing J212 P1 comment 79060. Starting head `77ae647dd59d4b3a3235223f0e6ef0a74899ca7d`, branch `agent-fsync-offloop-j196`; fetched origin/dev `01ee7103a4ea3221cdff4fcfd59195fcc1f77032` for the requested merge. Plan: reproduce the 18 ordinary fence-waiter/three revocation interleave first; reserve security progress independently of ordinary admission and deny before disk waits, preserve bounded ownership and cancellation; test persistence failure/retry as well as six ordering cases; quick then full fast in foreground; push atomic repair and delete target. Estimate 180 minutes including fresh builds/shared queues. This user-directed repair supersedes the common third-review stop rule. No merge to dev or deployment.
kayg commented 2026-11-04 23:09:19 +00:00
Author
Owner
Copy link

P1 repair design: retain at most eighteen per-person revocation intents per configured login registry, coalesce duplicates, and own them in one retry task rather than in callers. Use separate security admission/execution unavailable to ordinary saves. Record in-memory denial before any await; report errors through shared receipts while rotating/retrying retained work. Security-only overflow coalesces into one host-wide refusal (memory first, then file+directory sync and host login cleanup), rather than evicting an intent or allocating an unbounded queue. Individual restoration cannot clear that conservative host latch; recovery requires an administrator with the daemon stopped. This fallback is reached by distinct-security-intent saturation, never by ordinary saves. Tests cover ordinary flood/interleaved revocations, persistence error after finite callers leave, and 1,024 cancelled security requests with at most nineteen retained intent records. The original red test remains queued on dev-02 at position two; old source pinned at 6d0dcee7b. Sole target will be /home/claude/worktrees/j196/target on dev-02; no compute01 target is retained.

P1 repair design: retain at most eighteen per-person revocation intents per configured login registry, coalesce duplicates, and own them in one retry task rather than in callers. Use separate security admission/execution unavailable to ordinary saves. Record in-memory denial before any await; report errors through shared receipts while rotating/retrying retained work. Security-only overflow coalesces into one host-wide refusal (memory first, then file+directory sync and host login cleanup), rather than evicting an intent or allocating an unbounded queue. Individual restoration cannot clear that conservative host latch; recovery requires an administrator with the daemon stopped. This fallback is reached by distinct-security-intent saturation, never by ordinary saves. Tests cover ordinary flood/interleaved revocations, persistence error after finite callers leave, and 1,024 cancelled security requests with at most nineteen retained intent records. The original red test remains queued on dev-02 at position two; old source pinned at `6d0dcee7b`. Sole target will be `/home/claude/worktrees/j196/target` on dev-02; no compute01 target is retained.
kayg commented 2026-11-04 23:20:59 +00:00
Author
Owner
Copy link

Red confirmed on the pre-fix merged snapshot 6d0dcee7b on dev-02:

ordinary_waiters=18 security_pending=[false, false, false] denied_before_disk=[false, false, false]
revocation was refused by ordinary admission
RED_EXIT=101

The original six ordering cases have not yet been rerun on this repair. Prepared bounded intent/security-lane checkpoint is pushed as a3a803704; applying it only after this old-source failure, then quick and focused checks. The red log is /home/claude/j196-evidence/r3/red.log; sole target remains dev-02 /home/claude/worktrees/j196/target.

Red confirmed on the pre-fix merged snapshot `6d0dcee7b` on dev-02: ```text ordinary_waiters=18 security_pending=[false, false, false] denied_before_disk=[false, false, false] revocation was refused by ordinary admission RED_EXIT=101 ``` The original six ordering cases have not yet been rerun on this repair. Prepared bounded intent/security-lane checkpoint is pushed as `a3a803704`; applying it only after this old-source failure, then quick and focused checks. The red log is `/home/claude/j196-evidence/r3/red.log`; sole target remains dev-02 `/home/claude/worktrees/j196/target`.
red.log
4.8 KiB
red-revocation-tests.txt
3.1 KiB
kayg commented 2026-11-04 23:39:51 +00:00
Author
Owner
Copy link

J196 progress: repair checkpoint 61f2babf93b185b926a4bab86fe388460732a68d is pushed. Failing-first proof is attached in comment 79074. Reserved security admission, synchronous in-memory denial, retained retries and bounded security overflow are implemented; compilation and green proof remain pending.

verify quick is first in dev-02's queue (PID 1530154, queued since 23:21 UTC / 04:51 IST); both dev-02 slots and all four compute01 slots are occupied, and compute01 has the longer queue. Foreground polling continues. Only target: /home/claude/worktrees/j196/target; logs: /home/claude/j196-evidence/r3/.

The strengthened regression requires all three revocations to become durable while all 18 ordinary admissions remain occupied. Additional tests exercise finite caller errors, repair-triggered durable retry, and 1,024 cancelled security requests coalescing to at most 19 retained intent records. No gate result claimed yet.

J196 progress: repair checkpoint `61f2babf93b185b926a4bab86fe388460732a68d` is pushed. Failing-first proof is attached in comment 79074. Reserved security admission, synchronous in-memory denial, retained retries and bounded security overflow are implemented; compilation and green proof remain pending. `verify quick` is first in dev-02's queue (PID 1530154, queued since 23:21 UTC / 04:51 IST); both dev-02 slots and all four compute01 slots are occupied, and compute01 has the longer queue. Foreground polling continues. Only target: `/home/claude/worktrees/j196/target`; logs: `/home/claude/j196-evidence/r3/`. The strengthened regression requires all three revocations to become durable while all 18 ordinary admissions remain occupied. Additional tests exercise finite caller errors, repair-triggered durable retry, and 1,024 cancelled security requests coalescing to at most 19 retained intent records. No gate result claimed yet.
kayg commented 2026-11-05 00:06:08 +00:00
Author
Owner
Copy link

J196 checkpoint: fetched and merged current origin/dev (36c971a65) cleanly; pushed head 5eec4f718b422d77bbb3bc79ffa517bf0886062f. The additional upstream files are tracer gesture/tests only. The held-person-fence test now requires immediate in-memory denial using explicit polling, replacing its old permitted-until-disk assertion and fixed sleep.

Quick remains first in dev-02's queue, now waiting more than 40 minutes. Both dev-02 slots remain occupied; compute01 remains occupied with a longer queue. Source-head receipt and VM checkout agree on 5eec4f718. Foreground polling continues; no green result claimed. Target and evidence paths remain as in comment 79102.

J196 checkpoint: fetched and merged current `origin/dev` (`36c971a65`) cleanly; pushed head `5eec4f718b422d77bbb3bc79ffa517bf0886062f`. The additional upstream files are tracer gesture/tests only. The held-person-fence test now requires immediate in-memory denial using explicit polling, replacing its old permitted-until-disk assertion and fixed sleep. Quick remains first in dev-02's queue, now waiting more than 40 minutes. Both dev-02 slots remain occupied; compute01 remains occupied with a longer queue. Source-head receipt and VM checkout agree on `5eec4f718`. Foreground polling continues; no green result claimed. Target and evidence paths remain as in comment 79102.
kayg commented 2026-11-05 00:42:21 +00:00
Author
Owner
Copy link

J196 finding and next qualification head: 639c36ef0ed0014bf261fdaa89574c7dd3b7adc4 (pushed), with documented dev merge 0e27bf603c57f780ea8eb3faf28b5ec8d95cca90 incorporating 36c971a65. WIP checkpoints were consolidated; the repair tree includes the final watchdog adjustment.

First quick on 5eec4f718 failed: 148 unit tests passed, 3 new security tests failed, 10 ignored. Scope, migration names, formatting, doc coverage and workspace clippy passed. All three new failures were the 30-second real-durability watchdogs. The ordinary flood had already observed security_pending=[true, true, true] denied_before_disk=[true, true, true], but disk completion exceeded its guard. The unchanged exact ordinary-flood and finite-retry tests passed in isolation; the security-overflow test reproduced its timeout alone.

Diagnostic evidence for the latter: its Tokio blocking worker was in kernel state D, waiting in jbd2_log_wait_commit. The 30-second test eventually finished after 66.20 seconds because runtime teardown also waited for that started durable write. This is real journal delay, not proof that ordinary admission starved the reserved lane. Attached receipts preserve the failure and observed worker state.

The only subsequent change was to use a five-minute deadlock watchdog for real durable completion, documented separately from the event-order assertions. Immediate denial, 18 held ordinary admissions, durable completion before ordinary release, retained retry and the 19-record security bound are unchanged. No disk-speed success threshold was added. Final qualification is not yet green.

The final run will execute quick, the ordering/resource tests, all three security tests, scripts/verify fast verbatim, and cross-build the root proof executable. dev-02 retains the sole target /home/claude/worktrees/j196/target. compute01 had a free slot but only 14 GB free, below build-slot's 25 GB floor; dev-02 is usable. Foreground polling continues.

J196 finding and next qualification head: `639c36ef0ed0014bf261fdaa89574c7dd3b7adc4` (pushed), with documented dev merge `0e27bf603c57f780ea8eb3faf28b5ec8d95cca90` incorporating `36c971a65`. WIP checkpoints were consolidated; the repair tree includes the final watchdog adjustment. First quick on `5eec4f718` **failed**: 148 unit tests passed, 3 new security tests failed, 10 ignored. Scope, migration names, formatting, doc coverage and workspace clippy passed. All three new failures were the 30-second real-durability watchdogs. The ordinary flood had already observed `security_pending=[true, true, true] denied_before_disk=[true, true, true]`, but disk completion exceeded its guard. The unchanged exact ordinary-flood and finite-retry tests passed in isolation; the security-overflow test reproduced its timeout alone. Diagnostic evidence for the latter: its Tokio blocking worker was in kernel state `D`, waiting in `jbd2_log_wait_commit`. The 30-second test eventually finished after 66.20 seconds because runtime teardown also waited for that started durable write. This is real journal delay, not proof that ordinary admission starved the reserved lane. Attached receipts preserve the failure and observed worker state. The only subsequent change was to use a five-minute deadlock watchdog for real durable completion, documented separately from the event-order assertions. Immediate denial, 18 held ordinary admissions, durable completion before ordinary release, retained retry and the 19-record security bound are unchanged. No disk-speed success threshold was added. Final qualification is not yet green. The final run will execute quick, the ordering/resource tests, all three security tests, `scripts/verify fast` verbatim, and cross-build the root proof executable. dev-02 retains the sole target `/home/claude/worktrees/j196/target`. compute01 had a free slot but only 14 GB free, below build-slot's 25 GB floor; dev-02 is usable. Foreground polling continues.
quick-first-failed.log
40 KiB
journal-wait.txt
304 B
trace-security.log
783 B
kayg commented 2026-11-05 01:03:24 +00:00
Author
Owner
Copy link

J196 host update, same pushed head 639c36ef0ed0014bf261fdaa89574c7dd3b7adc4: compute01's queue cleared while dev-02 still had four jobs ahead of qualification. I canceled only J196's still-queued dev-02 run (QUALIFICATION_EXIT=143, no checks started), then deleted its target in the foreground. Deletion completed with ARM_TARGET_DELETED; a separate check confirmed the path is absent. The deletion itself blocked in wait_on_buffer, consistent with the journal delay diagnosed earlier.

Final qualification is now queued on compute01 through build-slot, nice, nix and -j4. Sole target will be /home/claude/worktrees/j196/target on compute01. Current logs are /home/claude/j196-evidence/r3/ on compute01; top receipt qualification.log. Commands are in that directory's qualify.sh: quick, ordering/resource filter, security filter, scripts/verify fast verbatim, and x86-musl root-test executable build. Foreground polling continues. No green claim yet.

The throwaway root clone /root/j196-proof/repo on compute01 is prepared and verifies the same exact SHA; UID=0 and the stress-ng executable are available. Root proof and its cleanup are still pending, as are final gate results. The prior failed quick and journal diagnostics remain preserved on dev-02, locally and attached in comment 79141.

J196 host update, same pushed head `639c36ef0ed0014bf261fdaa89574c7dd3b7adc4`: compute01's queue cleared while dev-02 still had four jobs ahead of qualification. I canceled only J196's still-queued dev-02 run (`QUALIFICATION_EXIT=143`, no checks started), then deleted its target in the foreground. Deletion completed with `ARM_TARGET_DELETED`; a separate check confirmed the path is absent. The deletion itself blocked in `wait_on_buffer`, consistent with the journal delay diagnosed earlier. Final qualification is now queued on compute01 through build-slot, nice, nix and `-j4`. **Sole target** will be `/home/claude/worktrees/j196/target` on compute01. **Current logs** are `/home/claude/j196-evidence/r3/` on compute01; top receipt `qualification.log`. Commands are in that directory's `qualify.sh`: quick, ordering/resource filter, security filter, `scripts/verify fast` verbatim, and x86-musl root-test executable build. Foreground polling continues. No green claim yet. The throwaway root clone `/root/j196-proof/repo` on compute01 is prepared and verifies the same exact SHA; UID=0 and the stress-ng executable are available. Root proof and its cleanup are still pending, as are final gate results. The prior failed quick and journal diagnostics remain preserved on dev-02, locally and attached in comment 79141.
kayg commented 2026-11-05 01:33:00 +00:00
Author
Owner
Copy link

J196 focused qualification on 639c36ef0ed0014bf261fdaa89574c7dd3b7adc4, compute01, through build-slot/nice/nix with -j4: quick passed (151 passed, 0 failed, 10 ignored), then all six causal ordering tests plus the descriptor flood passed (7/7), and all three security tests passed (3/3). The final full scripts/verify fast is now running; no final gate claim yet. Root-under-load proof and cleanup remain pending.

Exact focused receipts:

flood=1024 submitted_fds=18 cancelled_fds=0 retained_fences=0
ordinary_waiters=18 security_pending=[true, true, true] denied_before_disk=[true, true, true] durable_before_ordinary_release=true
security_flood=1024 retained_intents_peak=19 retained_after_durability=0 host_denied=true
QUICK_EXIT=0
ORDERING_EXIT=0
SECURITY_EXIT=0

The failed-marker test also passed: three finite error-returning callers share one retained intent; after repair it becomes durable without another revocation call, and explicit restore succeeds only after completion. Security overflow still intentionally denies the whole host; ordinary saturation never invokes that fallback. The previous short-watchdog failures and journal evidence remain attached in comment 79141.

J196 focused qualification on `639c36ef0ed0014bf261fdaa89574c7dd3b7adc4`, compute01, through build-slot/nice/nix with `-j4`: quick passed (151 passed, 0 failed, 10 ignored), then all six causal ordering tests plus the descriptor flood passed (7/7), and all three security tests passed (3/3). The final full `scripts/verify fast` is now running; no final gate claim yet. Root-under-load proof and cleanup remain pending. Exact focused receipts: ```text flood=1024 submitted_fds=18 cancelled_fds=0 retained_fences=0 ordinary_waiters=18 security_pending=[true, true, true] denied_before_disk=[true, true, true] durable_before_ordinary_release=true security_flood=1024 retained_intents_peak=19 retained_after_durability=0 host_denied=true QUICK_EXIT=0 ORDERING_EXIT=0 SECURITY_EXIT=0 ``` The failed-marker test also passed: three finite error-returning callers share one retained intent; after repair it becomes durable without another revocation call, and explicit restore succeeds only after completion. Security overflow still intentionally denies the whole host; ordinary saturation never invokes that fallback. The previous short-watchdog failures and journal evidence remain attached in comment 79141.
quick-final.log
40 KiB
ordering.log
1.6 KiB
security.log
1.6 KiB
final-source-head.txt
41 B
kayg commented 2026-11-05 02:02:26 +00:00
Author
Owner
Copy link

J196 progress at the original 180-minute estimate: head remains 639c36ef0ed0014bf261fdaa89574c7dd3b7adc4, pushed and unchanged. Quick and focused checks are green (comment 79180). The full fast run has completed workspace tests without a reported failure and is now running its extra SSH/shim fixture rows; static builds and final gate output remain pending. The previously intermittent desktop startup-recovery test passed in this run.

Most elapsed time has been shared-slot waits, cold native builds after moving hosts, and the diagnosed real-journal watchdog failure. Foreground polling continues. The sole target remains compute01 /home/claude/worktrees/j196/target; the arm target is confirmed deleted. No deployment or integration to dev has occurred.

Since qualification moved to native x86 compute01, root proof can reuse the actual native test executable produced by this gate instead of compiling a second ABI solely to run on compute01. A root-owned mode-0555 copy has matching SHA-256 0914327ab576ec8ed8c5f35b17aaaee5c318a088a03e3d915c3483ba0554ac72; the throwaway root clone verifies the same source SHA. Its test inventory contains all required ordering, resource, security and publisher tests. Root execution under real disk load and cleanup still remain; neither is being claimed yet.

J196 progress at the original 180-minute estimate: head remains `639c36ef0ed0014bf261fdaa89574c7dd3b7adc4`, pushed and unchanged. Quick and focused checks are green (comment 79180). The full fast run has completed workspace tests without a reported failure and is now running its extra SSH/shim fixture rows; static builds and final gate output remain pending. The previously intermittent desktop startup-recovery test passed in this run. Most elapsed time has been shared-slot waits, cold native builds after moving hosts, and the diagnosed real-journal watchdog failure. Foreground polling continues. The sole target remains compute01 `/home/claude/worktrees/j196/target`; the arm target is confirmed deleted. No deployment or integration to dev has occurred. Since qualification moved to native x86 compute01, root proof can reuse the actual native test executable produced by this gate instead of compiling a second ABI solely to run on compute01. A root-owned mode-0555 copy has matching SHA-256 `0914327ab576ec8ed8c5f35b17aaaee5c318a088a03e3d915c3483ba0554ac72`; the throwaway root clone verifies the same source SHA. Its test inventory contains all required ordering, resource, security and publisher tests. Root execution under real disk load and cleanup still remain; neither is being claimed yet.
kayg commented 2026-11-05 02:36:25 +00:00
Author
Owner
Copy link

J196 foreground qualification on final head 639c36ef0ed0014bf261fdaa89574c7dd3b7adc4: scripts/verify quick passed, the six ordering tests plus cancellation descriptor flood passed, and three security regressions passed. scripts/verify fast ran verbatim and exited 0. Full raw output attached.

| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
FAST_EXIT=0

Static shipping artifacts, whole branch versus the gate's recorded baseline (not isolated repair attribution): x86_64 23,538,928 bytes versus 22,418,256, delta +1,120,672; arm64 21,393,872 versus 20,453,280, delta +940,592. Both static rows passed; the arm64 native key-memory execution probe was skipped on x86_64 as reported by the gate.

Stopped only the helper's redundant post-gate musl unit-test compilation after recording FAST_EXIT=0; its process group exited 143, with no surviving descendants. Root proof already has the final native test executable and remains queued. Target deletion is running; real disk-load proof and final cleanup receipts follow. No deployment or integration performed.

J196 foreground qualification on final head `639c36ef0ed0014bf261fdaa89574c7dd3b7adc4`: `scripts/verify quick` passed, the six ordering tests plus cancellation descriptor flood passed, and three security regressions passed. `scripts/verify fast` ran verbatim and exited 0. Full raw output attached. ```text | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | static builds | PASS | +---------------------------+--------------+ FAST_EXIT=0 ``` Static shipping artifacts, whole branch versus the gate's recorded baseline (not isolated repair attribution): x86_64 23,538,928 bytes versus 22,418,256, delta +1,120,672; arm64 21,393,872 versus 20,453,280, delta +940,592. Both static rows passed; the arm64 native key-memory execution probe was skipped on x86_64 as reported by the gate. Stopped only the helper's redundant post-gate musl unit-test compilation after recording FAST_EXIT=0; its process group exited 143, with no surviving descendants. Root proof already has the final native test executable and remains queued. Target deletion is running; real disk-load proof and final cleanup receipts follow. No deployment or integration performed.
fast.log
844 KiB
kayg commented 2026-11-05 02:44:32 +00:00
Author
Owner
Copy link

J196 completion for P1 review comment 79060. Pushed branch agent-fsync-offloop-j196, head 639c36ef0ed0014bf261fdaa89574c7dd3b7adc4. Merged origin/dev snapshot 36c971a65 in 0e27bf603c57f780ea8eb3faf28b5ec8d95cca90 before qualification. No integration, deployment, or issue closure.

Revocation now publishes denial before its first await, uses reserved security admission/execution that ordinary saves cannot consume, and retains a bounded owner that retries persistence after errors or caller cancellation. File and directory fsync and cleanup still precede success, and finite callers still receive persistence errors. Failed host-wide attempts alternate with individual attempts to prevent starvation. Ordinary bounds stay at two submitted jobs plus sixteen admissions.

Material recovery behavior: more than eighteen distinct retained security intents coalesce into one host-wide denial, capped at nineteen intents total. The persistent infraternal-login-locks/host.revoked latch denies everyone on that host; individual restore cannot clear it. Administrator recovery requires stopping the daemon and removing the latch before explicitly restoring people. Ordinary saturation never triggers this fallback. This is a conservative fail-closed policy, not a claim of separate owner approval.

Failing-first regression and original failing source are attached to comment 79074. With eighteen ordinary waiters, pre-fix output was security_pending=[false, false, false] denied_before_disk=[false, false, false] and exit 101. Final causal assertions show all three pending and immediately denied, and security durability completing before ordinary admissions are released. The six ordering tests, descriptor cancellation flood, and three security tests all passed on final head (comment 79180).

Real-environment proof: root-owned throwaway clone on compute01 at the same head, UID 0, verified native test executable SHA-256 0914327ab576ec8ed8c5f35b17aaaee5c318a088a03e3d915c3483ba0554ac72. Ran through build-slot with ionice -c3 stress-ng --hdd 1 --hdd-bytes 512M --hdd-opts wr-seq,fsync --timeout 600s; tests began only after the writer reported 655,360 actual write bytes. All twelve selected tests passed (six ordering, descriptor flood, owned-memory flood, three security regressions, publisher contention). Exact evidence:

flood=1024 submitted_fds=18 cancelled_fds=0 retained_fences=0
flood=4096 peak_owned_bytes=1179648 retained_owned_bytes=0
ordinary_waiters=18 security_pending=[true, true, true] denied_before_disk=[true, true, true] durable_before_ordinary_release=true
security_flood=1024 retained_intents_peak=19 retained_after_durability=0 host_denied=true
ROOT_MEMORY_EXIT=0
ROOT_SECURITY_EXIT=0
ROOT_PUBLISHER_EXIT=0
PROOF_EXIT=0
ROOT_PROOF_RUN_EXIT=0

Owned-memory bytes measure retained queued payload, not whole-process RSS. The disk writer completed 353 fsync-heavy operations and was stopped and reaped by the proof script. Attached raw log and archive contain the script, stress log and before/after disk counters; binary/source receipt attached separately.

scripts/verify quick passed; scripts/verify fast ran verbatim in the foreground and exited 0. Full raw fast log is attached to comment 79229. Verbatim Rust summary and final status:

| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
FAST_EXIT=0

Static sizes, whole branch versus recorded gate baseline: x86_64 23,538,928 bytes (+1,120,672); arm64 21,393,872 bytes (+940,592). Both static rows passed. The gate explicitly skipped arm64 native key-memory execution on the x86 host.

Cleanup complete: both compute01 and dev-02 targets and J196 test state are absent; /root/j196-proof is absent; no own queued/running job, disk writer, or compiler descendants remain. The redundant optional post-gate musl test compile was stopped after FAST_EXIT=0, not counted as a failed gate. Control and compute source worktrees are clean; origin branch matches the final head. Evidence remains under /home/claude/j196-evidence/r3 and control /home/kayg/.cache/codex-scratch/j196/r3.

J196 completion for P1 review comment 79060. Pushed branch `agent-fsync-offloop-j196`, head `639c36ef0ed0014bf261fdaa89574c7dd3b7adc4`. Merged origin/dev snapshot `36c971a65` in `0e27bf603c57f780ea8eb3faf28b5ec8d95cca90` before qualification. No integration, deployment, or issue closure. Revocation now publishes denial before its first await, uses reserved security admission/execution that ordinary saves cannot consume, and retains a bounded owner that retries persistence after errors or caller cancellation. File and directory fsync and cleanup still precede success, and finite callers still receive persistence errors. Failed host-wide attempts alternate with individual attempts to prevent starvation. Ordinary bounds stay at two submitted jobs plus sixteen admissions. Material recovery behavior: more than eighteen distinct retained security intents coalesce into one host-wide denial, capped at nineteen intents total. The persistent `infraternal-login-locks/host.revoked` latch denies everyone on that host; individual restore cannot clear it. Administrator recovery requires stopping the daemon and removing the latch before explicitly restoring people. Ordinary saturation never triggers this fallback. This is a conservative fail-closed policy, not a claim of separate owner approval. Failing-first regression and original failing source are attached to comment 79074. With eighteen ordinary waiters, pre-fix output was `security_pending=[false, false, false] denied_before_disk=[false, false, false]` and exit 101. Final causal assertions show all three pending and immediately denied, and security durability completing before ordinary admissions are released. The six ordering tests, descriptor cancellation flood, and three security tests all passed on final head (comment 79180). Real-environment proof: root-owned throwaway clone on compute01 at the same head, UID 0, verified native test executable SHA-256 `0914327ab576ec8ed8c5f35b17aaaee5c318a088a03e3d915c3483ba0554ac72`. Ran through build-slot with `ionice -c3 stress-ng --hdd 1 --hdd-bytes 512M --hdd-opts wr-seq,fsync --timeout 600s`; tests began only after the writer reported 655,360 actual write bytes. All twelve selected tests passed (six ordering, descriptor flood, owned-memory flood, three security regressions, publisher contention). Exact evidence: ```text flood=1024 submitted_fds=18 cancelled_fds=0 retained_fences=0 flood=4096 peak_owned_bytes=1179648 retained_owned_bytes=0 ordinary_waiters=18 security_pending=[true, true, true] denied_before_disk=[true, true, true] durable_before_ordinary_release=true security_flood=1024 retained_intents_peak=19 retained_after_durability=0 host_denied=true ROOT_MEMORY_EXIT=0 ROOT_SECURITY_EXIT=0 ROOT_PUBLISHER_EXIT=0 PROOF_EXIT=0 ROOT_PROOF_RUN_EXIT=0 ``` Owned-memory bytes measure retained queued payload, not whole-process RSS. The disk writer completed 353 fsync-heavy operations and was stopped and reaped by the proof script. Attached raw log and archive contain the script, stress log and before/after disk counters; binary/source receipt attached separately. `scripts/verify quick` passed; `scripts/verify fast` ran verbatim in the foreground and exited 0. Full raw fast log is attached to comment 79229. Verbatim Rust summary and final status: ```text | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | static builds | PASS | +---------------------------+--------------+ FAST_EXIT=0 ``` Static sizes, whole branch versus recorded gate baseline: x86_64 23,538,928 bytes (+1,120,672); arm64 21,393,872 bytes (+940,592). Both static rows passed. The gate explicitly skipped arm64 native key-memory execution on the x86 host. Cleanup complete: both compute01 and dev-02 targets and J196 test state are absent; `/root/j196-proof` is absent; no own queued/running job, disk writer, or compiler descendants remain. The redundant optional post-gate musl test compile was stopped after FAST_EXIT=0, not counted as a failed gate. Control and compute source worktrees are clean; origin branch matches the final head. Evidence remains under `/home/claude/j196-evidence/r3` and control `/home/kayg/.cache/codex-scratch/j196/r3`.
root-proof.log
3.8 KiB
root-binary-receipt.txt
295 B
root-proof-artifacts.tar.gz
2 KiB
kayg commented 2026-11-05 02:50:38 +00:00
Author
Owner
Copy link

J221 independent security re-review started. Review branch: review-j221 based on target head 639c36ef0e; comparison base origin/dev 2a4f29de4e. Plan (~60 minutes plus shared build-slot wait): trace revocation admission, access checks, retries and restart state; inspect recovery, authorization and D345 docs; reproduce crash/overflow/fairness and event-loop cases on compute01 through build-slot, adding review-only failing tests for reproduced findings; post verdict with evidence. No production code changes, push, merge, deploy or issue closure.

J221 independent security re-review started. Review branch: review-j221 based on target head 639c36ef0ed0014bf261fdaa89574c7dd3b7adc4; comparison base origin/dev 2a4f29de4ed888e203e2210780745f2070346d5a. Plan (~60 minutes plus shared build-slot wait): trace revocation admission, access checks, retries and restart state; inspect recovery, authorization and D345 docs; reproduce crash/overflow/fairness and event-loop cases on compute01 through build-slot, adding review-only failing tests for reproduced findings; post verdict with evidence. No production code changes, push, merge, deploy or issue closure.
kayg commented 2026-11-05 03:06:25 +00:00
Author
Owner
Copy link

J221 progress: the review-only crash/restart regression is queued on compute01 through build-slot (position 7). Both compute01 and dev-02 were saturated when checked; compute01 had 228–281 GB free and remains the requested reproduction host. Source trace shows the in-memory registry is a Weak-cached process-local map and the worker waits for reserved capacity before creating the persistent marker; I am waiting for the focused test to establish the restart result. No production changes.

J221 progress: the review-only crash/restart regression is queued on compute01 through build-slot (position 7). Both compute01 and dev-02 were saturated when checked; compute01 had 228–281 GB free and remains the requested reproduction host. Source trace shows the in-memory registry is a Weak-cached process-local map and the worker waits for reserved capacity before creating the persistent marker; I am waiting for the focused test to establish the restart result. No production changes.
kayg commented 2026-11-05 03:21:27 +00:00
Author
Owner
Copy link

J221 queue update: compute01 build-slot has held the focused restart probe at position 2 for about 25 minutes; no build has started. Current free disk is 269 GB, above the 25 GB floor. The control and compute worktrees still contain only the review test, and the branch under review remains unchanged at 639c36ef0e. Continuing foreground wait.

J221 queue update: compute01 build-slot has held the focused restart probe at position 2 for about 25 minutes; no build has started. Current free disk is 269 GB, above the 25 GB floor. The control and compute worktrees still contain only the review test, and the branch under review remains unchanged at 639c36ef0ed0014bf261fdaa89574c7dd3b7adc4. Continuing foreground wait.
kayg commented 2026-11-05 03:31:20 +00:00
Author
Owner
Copy link

J221 finding 1 — P1: a pending revocation is forgotten on daemon restart before marker creation.

crates/infraternal-agent/src/revocations.rs:61-79 keeps pending denials only in a process-local Weak<Registry> map; submit adds the intent and spawns its owner at lines 84-145. The durable marker is created later by logins.rs:575-657. After restart, logins.rs:472-490 permits access when neither the process-local registry nor an on-disk marker exists.

I added denied_intent_survives_restart_before_marker_write to the review worktree at crates/infraternal-agent/src/revocation_tests.rs:41-107. On compute01, the child held the reserved security permit, observed in-memory denial, verified the person marker was absent, and exited abruptly. The parent then checked the same state directory as a fresh process. Exact test result:

thread 'logins::revocation_tests::denied_intent_survives_restart_before_marker_write' panicked at crates/infraternal-agent/src/revocation_tests.rs:103:5:
restart lost the accepted in-memory denial before it became durable
test logins::revocation_tests::denied_intent_survives_restart_before_marker_write ... FAILED
test result: FAILED. 0 passed; 1 failed; 0 ignored; 161 filtered out
error: test failed, to rerun pass `-p infraternal-agent --lib`

This is reachable after the server's finite wait: remote_guard.rs:97-105 makes three 100 ms attempts, and remote_authority.rs:168-173 logs unconfirmed cleanup but continues retirement. If the daemon exits in the volatile window, a later local open can pass again. The caller-independent retry fixes cancellation and ordinary queue saturation, but cannot survive process loss without a durable intent record or a policy that does not treat it as accepted until persisted.

The focused test ran through compute01 build-slot against 639c36ef0ed0014bf261fdaa89574c7dd3b7adc4; only the review test file changed.

J221 finding 1 — **P1: a pending revocation is forgotten on daemon restart before marker creation.** `crates/infraternal-agent/src/revocations.rs:61-79` keeps pending denials only in a process-local `Weak<Registry>` map; `submit` adds the intent and spawns its owner at lines 84-145. The durable marker is created later by `logins.rs:575-657`. After restart, `logins.rs:472-490` permits access when neither the process-local registry nor an on-disk marker exists. I added `denied_intent_survives_restart_before_marker_write` to the review worktree at `crates/infraternal-agent/src/revocation_tests.rs:41-107`. On compute01, the child held the reserved security permit, observed in-memory denial, verified the person marker was absent, and exited abruptly. The parent then checked the same state directory as a fresh process. Exact test result: ```text thread 'logins::revocation_tests::denied_intent_survives_restart_before_marker_write' panicked at crates/infraternal-agent/src/revocation_tests.rs:103:5: restart lost the accepted in-memory denial before it became durable test logins::revocation_tests::denied_intent_survives_restart_before_marker_write ... FAILED test result: FAILED. 0 passed; 1 failed; 0 ignored; 161 filtered out error: test failed, to rerun pass `-p infraternal-agent --lib` ``` This is reachable after the server's finite wait: `remote_guard.rs:97-105` makes three 100 ms attempts, and `remote_authority.rs:168-173` logs unconfirmed cleanup but continues retirement. If the daemon exits in the volatile window, a later local open can pass again. The caller-independent retry fixes cancellation and ordinary queue saturation, but cannot survive process loss without a durable intent record or a policy that does not treat it as accepted until persisted. The focused test ran through compute01 build-slot against `639c36ef0ed0014bf261fdaa89574c7dd3b7adc4`; only the review test file changed.
kayg commented 2026-11-05 03:57:24 +00:00
Author
Owner
Copy link

J221 final independent security re-review — REQUEST CHANGES

Target agent-fsync-offloop-j196 at 639c36ef0ed0014bf261fdaa89574c7dd3b7adc4, compared with origin/dev 2a4f29de4ed888e203e2210780745f2070346d5a.

P1 — accepted in-memory revocation can be lost on daemon restart. crates/infraternal-agent/src/revocations.rs:61-79,82-145 stores pending denial only in a process-local weak registry, then starts the durable owner asynchronously. The persistent person marker is written later in crates/infraternal-agent/src/logins.rs:622-657; a fresh process permits access at logins.rs:471-489 when neither that marker nor a host marker exists. The server makes three 100 ms cleanup attempts and continues authority retirement after an unconfirmed result (crates/infraternal-server/src/remote_guard.rs:97-105, remote_authority.rs:167-173).

The review regression denied_intent_survives_restart_before_marker_write (currently in the compute01 review worktree at crates/infraternal-agent/src/revocation_tests.rs:41-107) held the reserved permit, observed the in-memory denial, verified the marker was absent, then exited the child. The parent reopened the same state as a fresh process and failed because access was permitted: restart lost the accepted in-memory denial before it became durable (exit 101; 0 passed, 1 failed). This is also recorded in comment 79290. Caller cancellation and finite caller retries are handled while the daemon stays alive: the detached owner retains and retries failed intents. Process loss is not handled, so the accepted denial can disappear after restart.

J212's original ordinary-queue saturation failure is fixed: the reserved security admission and execution slots in durable_io.rs:23-26,48-58 cannot be consumed by ordinary saves. The compute01 ordering evidence showed 18 ordinary waiters with three pending denials, all denied before disk completion, and durable security work finishing before ordinary admissions were released. The restart gap keeps the P1 open.

P2 — owner decision on overflow blast radius and recovery. At revocations.rs:19,122-125, the 19th distinct pending intent becomes a host-wide lockdown; logins.rs:660-674 persists infraternal-login-locks/host.revoked, which denies all people. The production request path is the trusted server control stream (crates/infraternal-agent/src/daemon.rs:931-942); the public /remote/revocations route requires the configured origin and an owner session cookie before calling revoke_for_session (crates/infraternal-server/src/remote_http.rs:187-205). I found no direct route for an unauthenticated remote peer. The 1,024-UUID flood test exercises the agent boundary, not an unprivileged end-to-end request.

Recovery is described in code (revocations.rs:1-10, logins.rs:660-663): stop the daemon, remove the validated host marker, then explicitly restore people. There is no operator runbook under docs/ and no test of that recovery sequence; the overflow test checks denial only. This policy should be decided by kayg. Options: approve the host-wide latch and add a tested operator procedure; keep a bounded durable per-person overflow record; or apply upstream backpressure and avoid retiring authority until durable admission is available. Narrower fail-closed can preserve the guarantee if per-person denials are durable or the caller remains backpressured; an in-memory-only overflow set would have the same restart gap.

Other checks. The ordinary and security lanes have separate admission/execution semaphores (durable_io.rs:20-26); the worker alternates host-lock work with queued people (revocations.rs:157-169). I found no cross-lane starvation path. Fsyncs on the login, revocation, rootless reservation/cleanup, and login-binary publication paths execute inside bounded Admission::run / spawn_blocking work; startup-guard syncs run in its spawned helper process. No fsync remains on the reviewed agent dispatch path. Added code has D345 documentation comments; I found no secrets in logs or unrelated product names in the changed code.

I checked / before compute01 test work (248 GB free). I could not independently repeat the root-owned run: the supplied meshtest key did not authenticate to compute01's local SSH endpoints, and I did not read or print the key. J196's root proof and scripts/verify fast FAST_EXIT=0 are recorded in comment 79233; I did not independently rerun the full gate. The focused restart regression is the only failing reproduction; the other focused saturation/retry/overflow checks passed. No production files were changed by this review.

J221 final independent security re-review — **REQUEST CHANGES** Target `agent-fsync-offloop-j196` at `639c36ef0ed0014bf261fdaa89574c7dd3b7adc4`, compared with `origin/dev` `2a4f29de4ed888e203e2210780745f2070346d5a`. **P1 — accepted in-memory revocation can be lost on daemon restart.** `crates/infraternal-agent/src/revocations.rs:61-79,82-145` stores pending denial only in a process-local weak registry, then starts the durable owner asynchronously. The persistent person marker is written later in `crates/infraternal-agent/src/logins.rs:622-657`; a fresh process permits access at `logins.rs:471-489` when neither that marker nor a host marker exists. The server makes three 100 ms cleanup attempts and continues authority retirement after an unconfirmed result (`crates/infraternal-server/src/remote_guard.rs:97-105`, `remote_authority.rs:167-173`). The review regression `denied_intent_survives_restart_before_marker_write` (currently in the compute01 review worktree at `crates/infraternal-agent/src/revocation_tests.rs:41-107`) held the reserved permit, observed the in-memory denial, verified the marker was absent, then exited the child. The parent reopened the same state as a fresh process and failed because access was permitted: `restart lost the accepted in-memory denial before it became durable` (exit 101; 0 passed, 1 failed). This is also recorded in comment 79290. Caller cancellation and finite caller retries are handled while the daemon stays alive: the detached owner retains and retries failed intents. Process loss is not handled, so the accepted denial can disappear after restart. J212's original ordinary-queue saturation failure is fixed: the reserved security admission and execution slots in `durable_io.rs:23-26,48-58` cannot be consumed by ordinary saves. The compute01 ordering evidence showed 18 ordinary waiters with three pending denials, all denied before disk completion, and durable security work finishing before ordinary admissions were released. The restart gap keeps the P1 open. **P2 — owner decision on overflow blast radius and recovery.** At `revocations.rs:19,122-125`, the 19th distinct pending intent becomes a host-wide lockdown; `logins.rs:660-674` persists `infraternal-login-locks/host.revoked`, which denies all people. The production request path is the trusted server control stream (`crates/infraternal-agent/src/daemon.rs:931-942`); the public `/remote/revocations` route requires the configured origin and an owner session cookie before calling `revoke_for_session` (`crates/infraternal-server/src/remote_http.rs:187-205`). I found no direct route for an unauthenticated remote peer. The 1,024-UUID flood test exercises the agent boundary, not an unprivileged end-to-end request. Recovery is described in code (`revocations.rs:1-10`, `logins.rs:660-663`): stop the daemon, remove the validated host marker, then explicitly restore people. There is no operator runbook under `docs/` and no test of that recovery sequence; the overflow test checks denial only. This policy should be decided by kayg. Options: approve the host-wide latch and add a tested operator procedure; keep a bounded durable per-person overflow record; or apply upstream backpressure and avoid retiring authority until durable admission is available. Narrower fail-closed can preserve the guarantee if per-person denials are durable or the caller remains backpressured; an in-memory-only overflow set would have the same restart gap. **Other checks.** The ordinary and security lanes have separate admission/execution semaphores (`durable_io.rs:20-26`); the worker alternates host-lock work with queued people (`revocations.rs:157-169`). I found no cross-lane starvation path. Fsyncs on the login, revocation, rootless reservation/cleanup, and login-binary publication paths execute inside bounded `Admission::run` / `spawn_blocking` work; startup-guard syncs run in its spawned helper process. No fsync remains on the reviewed agent dispatch path. Added code has D345 documentation comments; I found no secrets in logs or unrelated product names in the changed code. I checked `/` before compute01 test work (248 GB free). I could not independently repeat the root-owned run: the supplied meshtest key did not authenticate to compute01's local SSH endpoints, and I did not read or print the key. J196's root proof and `scripts/verify fast` `FAST_EXIT=0` are recorded in comment 79233; I did not independently rerun the full gate. The focused restart regression is the only failing reproduction; the other focused saturation/retry/overflow checks passed. No production files were changed by this review.
kayg commented 2026-11-05 04:03:50 +00:00
Author
Owner
Copy link

J196 restart-safety follow-up to J221 comment 79308, continuing branch agent-fsync-offloop-j196 from 639c36ef0ed0014bf261fdaa89574c7dd3b7adc4. Read the complete review; copied the exact restart regression first (68 added lines, no implementation changes).

Plan, estimate 240 minutes: (1) confirm the review test fails on the reviewed head; (2) trace authority/reconnect admission and repair the crash window while keeping fsync on bounded blocking work, preferring server authority; (3) retain current host overflow policy and add a recovery runbook plus stop/remove-validated-marker/restore test; (4) merge current origin/dev, run quick, causal ordering/security tests, root disk-load proof, and scripts/verify fast in the foreground; (5) push final atomic commits, attach exact logs and remove targets/test state. New round-three P1 has been reviewed and this plan records the orchestrator's explicit instruction to continue its repair.

Both slot queues checked: compute01 has five queued for four slots, dev-02 three for two. Using compute01 and exactly one target /home/claude/worktrees/j196/target; red evidence under /home/claude/j196-evidence/r4. Overflow policy remains unchanged pending the owner's decision.

J196 restart-safety follow-up to J221 comment 79308, continuing branch `agent-fsync-offloop-j196` from `639c36ef0ed0014bf261fdaa89574c7dd3b7adc4`. Read the complete review; copied the exact restart regression first (68 added lines, no implementation changes). Plan, estimate 240 minutes: (1) confirm the review test fails on the reviewed head; (2) trace authority/reconnect admission and repair the crash window while keeping fsync on bounded blocking work, preferring server authority; (3) retain current host overflow policy and add a recovery runbook plus stop/remove-validated-marker/restore test; (4) merge current origin/dev, run quick, causal ordering/security tests, root disk-load proof, and `scripts/verify fast` in the foreground; (5) push final atomic commits, attach exact logs and remove targets/test state. New round-three P1 has been reviewed and this plan records the orchestrator's explicit instruction to continue its repair. Both slot queues checked: compute01 has five queued for four slots, dev-02 three for two. Using compute01 and exactly one target `/home/claude/worktrees/j196/target`; red evidence under `/home/claude/j196-evidence/r4`. Overflow policy remains unchanged pending the owner's decision.
kayg commented 2026-11-05 04:20:08 +00:00
Author
Owner
Copy link

Confirmed J221's exact copied restart regression fails on the reviewed source: restart lost the accepted in-memory denial before it became durable, RED_EXIT=101 (raw log and exact source attached). Ran the review's already-built test executable against isolated J196 state; cancelled the redundant queued red rebuild before it acquired a slot.

Chosen repair is (a), server authority: a durable outbox row is written in the same transaction as every host tombstone, including bootstrap-recovery paths. Normal frame writes still cut/drain before COMMIT. No host revocation is delivered before the server intent commits. Fresh daemon login checks require an affirmative trusted-server grant; absence of a marker is never sufficient. Existing trusted server open/prepare requests supply current positive authority without clearing a refusal. Revoked certificates can reconnect only to receive their persisted cleanup; they cannot enter inventory or perform ordinary operations, restore, terminal or plugin writes. Only an actual durable daemon reply confirms the outbox. Failed/lost replies retain it for reconnect replay. This avoids requiring host disk progress before the server can preserve the authoritative revocation.

P2 policy is unchanged. Added docs/operations/host-login-lockdown-recovery.md and a test executing its literal operator snippet after the child fixture exits, including refusal of a symlink and explicit per-person restoration.

Merged origin/dev 2a4f29de4 in fd4202b30 and pushed that merge. Implementation is in progress; quick is queued on compute01. One target only: /home/claude/worktrees/j196/target. No deployment or integration.

Confirmed J221's exact copied restart regression fails on the reviewed source: `restart lost the accepted in-memory denial before it became durable`, RED_EXIT=101 (raw log and exact source attached). Ran the review's already-built test executable against isolated J196 state; cancelled the redundant queued red rebuild before it acquired a slot. Chosen repair is (a), server authority: a durable outbox row is written in the same transaction as every host tombstone, including bootstrap-recovery paths. Normal frame writes still cut/drain before COMMIT. No host revocation is delivered before the server intent commits. Fresh daemon login checks require an affirmative trusted-server grant; absence of a marker is never sufficient. Existing trusted server open/prepare requests supply current positive authority without clearing a refusal. Revoked certificates can reconnect only to receive their persisted cleanup; they cannot enter inventory or perform ordinary operations, restore, terminal or plugin writes. Only an actual durable daemon reply confirms the outbox. Failed/lost replies retain it for reconnect replay. This avoids requiring host disk progress before the server can preserve the authoritative revocation. P2 policy is unchanged. Added `docs/operations/host-login-lockdown-recovery.md` and a test executing its literal operator snippet after the child fixture exits, including refusal of a symlink and explicit per-person restoration. Merged origin/dev `2a4f29de4` in `fd4202b30` and pushed that merge. Implementation is in progress; quick is queued on compute01. One target only: `/home/claude/worktrees/j196/target`. No deployment or integration.
red.log
638 B
kayg commented 2026-11-05 04:23:01 +00:00
Author
Owner
Copy link

Restart repair checkpoint pushed as 6adb69c94, on merged base fd4202b30. Qualification is pending; this is a WIP, not a passing head. compute01 quick is queued against this exact checkout; one target /home/claude/worktrees/j196/target.

The durable cleanup trigger covers every host-retirement writer, including startup bootstrap recovery. Tombstone/outbox commit before delivery; cleanup-only reconnects keep admission permits and never enter host inventory or plugin routing. The host still saves every refusal through the existing bounded blocking lane. Positive grant resynchronization is incremental per trusted server open/prepare, with all other people closed in a new process. Explicit restoration remains the only way to clear a persisted person marker.

The source attachment on comment 79375 was rejected by Forgejo for its .rs extension (the raw failing log attached successfully). Reattaching the exact copied review source as plain text here.

Restart repair checkpoint pushed as `6adb69c94`, on merged base `fd4202b30`. Qualification is pending; this is a WIP, not a passing head. compute01 quick is queued against this exact checkout; one target `/home/claude/worktrees/j196/target`. The durable cleanup trigger covers every host-retirement writer, including startup bootstrap recovery. Tombstone/outbox commit before delivery; cleanup-only reconnects keep admission permits and never enter host inventory or plugin routing. The host still saves every refusal through the existing bounded blocking lane. Positive grant resynchronization is incremental per trusted server open/prepare, with all other people closed in a new process. Explicit restoration remains the only way to clear a persisted person marker. The source attachment on comment 79375 was rejected by Forgejo for its `.rs` extension (the raw failing log attached successfully). Reattaching the exact copied review source as plain text here.
red-source.txt
12 KiB
kayg commented 2026-11-05 04:42:41 +00:00
Author
Owner
Copy link

First quick on the restart repair stopped at clippy (exit 101), before unit tests. fmt, migration names, scope and doc coverage passed. Raw output attached. The unreadable literal was the generated D347 UTC migration version in compact_migrations.rs; scoped that readability lint allowance to the generated static migration table, preserving the exact numeric version, SQL and checksum validation. Also extracted private cleanup delivery from admit_observed to keep its ordinary admission function below the line limit.

Checkpoint c85fb1c40 is pushed and the second quick is queued on compute01. Source is fixed while checks run; target remains only /home/claude/worktrees/j196/target. No fsync was added to the event loop.

First quick on the restart repair stopped at clippy (exit 101), before unit tests. fmt, migration names, scope and doc coverage passed. Raw output attached. The unreadable literal was the generated D347 UTC migration version in `compact_migrations.rs`; scoped that readability lint allowance to the generated static migration table, preserving the exact numeric version, SQL and checksum validation. Also extracted private cleanup delivery from `admit_observed` to keep its ordinary admission function below the line limit. Checkpoint `c85fb1c40` is pushed and the second quick is queued on compute01. Source is fixed while checks run; target remains only `/home/claude/worktrees/j196/target`. No fsync was added to the event loop.
quick-first-failed.log
23 KiB
kayg commented 2026-11-05 04:50:59 +00:00
Author
Owner
Copy link

J196 resumed qualification from pushed checkpoint c85fb1c40. Read J221 comment 79308 in full and checked the recorded exact failing regression (RED_EXIT=101). The second quick has passed workspace clippy and is compiling changed-package unit tests on compute01. No implementation success claim yet.

Merged latest origin/dev 130546edc into branch as dc1572585 and pushed. The sole conflict was the equivalent generated-migration literal allowance; kept upstream verbatim. Will rerun quick on this merge result, focused ordering/security/replay tests, root disk-load proof and scripts/verify fast, in the foreground. One target remains /home/claude/worktrees/j196/target. No deployment or integration.

J196 resumed qualification from pushed checkpoint c85fb1c40. Read J221 comment 79308 in full and checked the recorded exact failing regression (RED_EXIT=101). The second quick has passed workspace clippy and is compiling changed-package unit tests on compute01. No implementation success claim yet. Merged latest origin/dev 130546edc into branch as dc1572585 and pushed. The sole conflict was the equivalent generated-migration literal allowance; kept upstream verbatim. Will rerun quick on this merge result, focused ordering/security/replay tests, root disk-load proof and scripts/verify fast, in the foreground. One target remains /home/claude/worktrees/j196/target. No deployment or integration.
kayg commented 2026-11-05 04:58:16 +00:00
Author
Owner
Copy link

Second quick passed workspace clippy and ran 242 infraternal-agent tests: 241 passed, one fixture failed, 23 ignored. The unchanged rootless_self_account_login_does_not_depend_on_su_authentication fixture called ensure directly without the server grant now required after startup (PermissionDenied: host login authority has not been synchronized with the server). Updated that fixture to synchronize the authorized person first, matching daemon dispatch; production denial remains intact. Restart and operator recovery tests passed in that run.

Finalized the repair checkpoint into atomic commit dcf73da33, merged latest origin/dev 130546edc in 7eb84c139, and pushed branch agent-fsync-offloop-j196. The branch tree is unchanged from 0aa91d991; only checkpoint history was consolidated. Full qualification is queued via foreground SSH on compute01: quick, ordering, security/restart/recovery, real mTLS cleanup replay, scripts/verify fast, then root-owned disk-load proof. Logs: /home/claude/j196-evidence/r4; exactly one target: /home/claude/worktrees/j196/target. Both machine queues were checked; neither has a free slot and compute01 has the warm target.

Second quick passed workspace clippy and ran 242 infraternal-agent tests: 241 passed, one fixture failed, 23 ignored. The unchanged rootless_self_account_login_does_not_depend_on_su_authentication fixture called ensure directly without the server grant now required after startup (PermissionDenied: host login authority has not been synchronized with the server). Updated that fixture to synchronize the authorized person first, matching daemon dispatch; production denial remains intact. Restart and operator recovery tests passed in that run. Finalized the repair checkpoint into atomic commit dcf73da33, merged latest origin/dev 130546edc in 7eb84c139, and pushed branch agent-fsync-offloop-j196. The branch tree is unchanged from 0aa91d991; only checkpoint history was consolidated. Full qualification is queued via foreground SSH on compute01: quick, ordering, security/restart/recovery, real mTLS cleanup replay, scripts/verify fast, then root-owned disk-load proof. Logs: /home/claude/j196-evidence/r4; exactly one target: /home/claude/worktrees/j196/target. Both machine queues were checked; neither has a free slot and compute01 has the warm target.
quick-r2-failed.log
50 KiB
kayg commented 2026-11-05 05:15:17 +00:00
Author
Owner
Copy link

Qualification is running in a foreground SSH session at exact head 68ebc9e73f. The added test-only commit begins the retry/overflow checks with positive server grants, including an unrelated authorized person, so startup default denial cannot hide an overflow regression. J221's copied restart test remains unchanged.

Quick has passed clippy and its infraternal-agent portion: 242 passed, 0 failed, 23 ignored. The exact restart test, fresh-authority test, and literal runbook recovery test passed. Server unit tests are still running with no failures so far. Focused ordering/replay, fast and root disk-load proof remain pending. Root clone /root/j196-proof/repo is ready at the same head; it has no target. The one build target remains /home/claude/worktrees/j196/target.

Qualification is running in a foreground SSH session at exact head 68ebc9e73fc074038be300941edd13fe13d5f4b1. The added test-only commit begins the retry/overflow checks with positive server grants, including an unrelated authorized person, so startup default denial cannot hide an overflow regression. J221's copied restart test remains unchanged. Quick has passed clippy and its infraternal-agent portion: 242 passed, 0 failed, 23 ignored. The exact restart test, fresh-authority test, and literal runbook recovery test passed. Server unit tests are still running with no failures so far. Focused ordering/replay, fast and root disk-load proof remain pending. Root clone /root/j196-proof/repo is ready at the same head; it has no target. The one build target remains /home/claude/worktrees/j196/target.
kayg commented 2026-11-05 05:25:11 +00:00
Author
Owner
Copy link

Owner decision received through orchestrator: option (c). Remove the host-wide overflow latch; block only affected people. The server remains authoritative, persists an unconfirmed revocation and resends it on reconnect, and does not finish retirement until the infraternal agent acknowledges durable acceptance. Full reserved security capacity backpressures the server instead of creating a host-wide refusal. Fresh infraternal daemons remain closed until authoritative resync. Replace the latch recovery runbook with backpressure/resync documentation. Each behavioral part gets a failing test first and its own commit.

Stopped the superseded qualification process tree by its own PIDs, exit 143; no background qualification remains. Previous head 68ebc9e73 had quick, seven ordering/cancellation checks and six security checks passing. Its fast gate was interrupted, so it is not a passing final gate. The earlier focused replay command matched zero tests and is not evidence; the replacement will use the actual module filter and remote feature.

Updated plan, estimate 180 minutes from this decision: (1) red bounded security flood that never denies an unrelated authorized person; remove the host latch while keeping accepted intents bounded and durable, and replace latch documentation; (2) red server retirement/backpressure tests, implement persisted pending intent and durable receipt/reconnect completion; (3) red resync/abrupt-restart coverage combined with the retained restart fix; (4) merge origin/dev, rerun quick, ordering, explicit remote replay, root disk-load proof and fast in the foreground; push each part and attach results. One target remains /home/claude/worktrees/j196/target.

Owner decision received through orchestrator: option (c). Remove the host-wide overflow latch; block only affected people. The server remains authoritative, persists an unconfirmed revocation and resends it on reconnect, and does not finish retirement until the infraternal agent acknowledges durable acceptance. Full reserved security capacity backpressures the server instead of creating a host-wide refusal. Fresh infraternal daemons remain closed until authoritative resync. Replace the latch recovery runbook with backpressure/resync documentation. Each behavioral part gets a failing test first and its own commit. Stopped the superseded qualification process tree by its own PIDs, exit 143; no background qualification remains. Previous head 68ebc9e73 had quick, seven ordering/cancellation checks and six security checks passing. Its fast gate was interrupted, so it is not a passing final gate. The earlier focused replay command matched zero tests and is not evidence; the replacement will use the actual module filter and remote feature. Updated plan, estimate 180 minutes from this decision: (1) red bounded security flood that never denies an unrelated authorized person; remove the host latch while keeping accepted intents bounded and durable, and replace latch documentation; (2) red server retirement/backpressure tests, implement persisted pending intent and durable receipt/reconnect completion; (3) red resync/abrupt-restart coverage combined with the retained restart fix; (4) merge origin/dev, rerun quick, ordering, explicit remote replay, root disk-load proof and fast in the foreground; push each part and attach results. One target remains /home/claude/worktrees/j196/target.
kayg commented 2026-11-05 05:29:49 +00:00
Author
Owner
Copy link

Option (c), part 1 committed and pushed: 9711159e0. The new 1024-person saturation test failed first with “security saturation denied an unrelated authorized person” (RED_EXIT=101). Removed the host-wide latch and its recovery procedure. Per-person admission retains at most 18 accepted intents; duplicate callers share receipts, and excess distinct people receive WouldBlock before intent/fence/descriptor retention. The authoritative server must retain those busy requests, which is part 2. All five focused security tests now pass (GREEN_EXIT=0), including the unchanged copied restart regression and ordinary-queue ordering. Replaced the runbook with docs/operations/login-revocation-backpressure.md.

Part 2 red tests are ready: real mTLS retirement must remain pending until durable acceptance, including reconnect, and busy replies beyond the former three-attempt limit must not abandon the cleanup. compute01 has a long slot queue; its red rebuild is queued in a foreground SSH session. I am tracing all retirement writers, including bootstrap recovery, so confirmed retirement and intent persistence remain distinct. No fsync is being moved onto the event loop.

Option (c), part 1 committed and pushed: 9711159e0. The new 1024-person saturation test failed first with “security saturation denied an unrelated authorized person” (RED_EXIT=101). Removed the host-wide latch and its recovery procedure. Per-person admission retains at most 18 accepted intents; duplicate callers share receipts, and excess distinct people receive WouldBlock before intent/fence/descriptor retention. The authoritative server must retain those busy requests, which is part 2. All five focused security tests now pass (GREEN_EXIT=0), including the unchanged copied restart regression and ordinary-queue ordering. Replaced the runbook with docs/operations/login-revocation-backpressure.md. Part 2 red tests are ready: real mTLS retirement must remain pending until durable acceptance, including reconnect, and busy replies beyond the former three-attempt limit must not abandon the cleanup. compute01 has a long slot queue; its red rebuild is queued in a foreground SSH session. I am tracing all retirement writers, including bootstrap recovery, so confirmed retirement and intent persistence remain distinct. No fsync is being moved onto the event loop.
option-c-agent-red.log
1.4 KiB
option-c-agent-green.log
1.6 KiB
kayg commented 2026-11-05 05:55:17 +00:00
Author
Owner
Copy link

J196 progress: option-c part 1 is pushed at 9711159e0; test-first server regressions checkpointed at 2f07450b2. The regression compile has waited about 28 minutes for compute01 capacity. Both VM queues were checked; dev02 also has both slots occupied. No implementation has been written ahead of the red tests. Next: prove busy retry exhaustion and premature retirement, then fix each in its own passing commit. Target remains /home/claude/worktrees/j196/target; evidence /home/claude/j196-evidence/r4.

J196 progress: option-c part 1 is pushed at `9711159e0`; test-first server regressions checkpointed at `2f07450b2`. The regression compile has waited about 28 minutes for compute01 capacity. Both VM queues were checked; dev02 also has both slots occupied. No implementation has been written ahead of the red tests. Next: prove busy retry exhaustion and premature retirement, then fix each in its own passing commit. Target remains `/home/claude/worktrees/j196/target`; evidence `/home/claude/j196-evidence/r4`.
kayg commented 2026-11-05 06:04:43 +00:00
Author
Owner
Copy link

J196 option-c server regressions fail first (all exit 101):

  • retirement_waits_for_durable_acceptance_on_reconnect: retirement completed without a durable agent receipt: Ok(Ok(())).
  • host_removal_keeps_backpressure_beyond_three_busy_replies: host removal abandoned an unconfirmed login cleanup.
  • pending_revocation_survives_database_reopen_before_acceptance: restart found retirement without durable host acceptance.

Implementation uses the existing outbox as a pending denial before the final tombstone. Pending denial blocks ordinary certificate admission, renewals and provisional SSH; cleanup-only reconnect retries busy responses and the durable receipt commits retirement. No new migration is needed. Busy-retry green test is queued; implementation remains local until verified. Full red logs attached.

J196 option-c server regressions fail first (all exit 101): - `retirement_waits_for_durable_acceptance_on_reconnect`: `retirement completed without a durable agent receipt: Ok(Ok(()))`. - `host_removal_keeps_backpressure_beyond_three_busy_replies`: `host removal abandoned an unconfirmed login cleanup`. - `pending_revocation_survives_database_reopen_before_acceptance`: `restart found retirement without durable host acceptance`. Implementation uses the existing outbox as a pending denial before the final tombstone. Pending denial blocks ordinary certificate admission, renewals and provisional SSH; cleanup-only reconnect retries busy responses and the durable receipt commits retirement. No new migration is needed. Busy-retry green test is queued; implementation remains local until verified. Full red logs attached.
option-c-server-red.log
3 KiB
option-c-busy-red.log
685 B
option-c-reopen-red.log
1,012 B
kayg commented 2026-11-05 06:31:38 +00:00
Author
Owner
Copy link

J196 option-c implementation is pushed, merged with current dev (a8b4ee720) at e89da66bf81af1cc3dfd936c42e89141b6b3341c.

  • 9711159e0: remove host-wide latch; bounded per-person admission (red and green evidence in comment 79452).
  • b6e1c8e94: cooperate across repeated busy replies; four-refusal red test now passes.
  • 24d92da75: durable server pending denial; retirement waits for durable daemon receipt; normal certificate/renewal/provisional SSH admission is refused while pending. Cancellation releases only the waiter. Bootstrap and passkey removal share this path. The new mTLS ordering and database-reopen regressions pass, as does the updated passkey removal test.

The focused server run passed 11/12; the older replay fixture selected an expired enrollment certificate rather than the exact current TLS leaf. This reproduced individually, and the fixture now retains the exact fingerprint. That correction is awaiting final qualification, so the attached focused log is not an all-green claim.

Moving final qualification to dev02 because its queue is shorter. The compute01 target has been deleted; J196 will keep only /home/claude/worktrees/j196/target on dev02. Root disk-load execution remains on compute01, using the x86 static test binary cross-built on dev02. Quick, focused checks, root proof and foreground fast gate remain pending.

J196 option-c implementation is pushed, merged with current dev (`a8b4ee720`) at **`e89da66bf81af1cc3dfd936c42e89141b6b3341c`**. - `9711159e0`: remove host-wide latch; bounded per-person admission (red and green evidence in comment 79452). - `b6e1c8e94`: cooperate across repeated busy replies; four-refusal red test now passes. - `24d92da75`: durable server pending denial; retirement waits for durable daemon receipt; normal certificate/renewal/provisional SSH admission is refused while pending. Cancellation releases only the waiter. Bootstrap and passkey removal share this path. The new mTLS ordering and database-reopen regressions pass, as does the updated passkey removal test. The focused server run passed 11/12; the older replay fixture selected an expired enrollment certificate rather than the exact current TLS leaf. This reproduced individually, and the fixture now retains the exact fingerprint. That correction is awaiting final qualification, so the attached focused log is not an all-green claim. Moving final qualification to dev02 because its queue is shorter. The compute01 target has been deleted; J196 will keep only `/home/claude/worktrees/j196/target` on dev02. Root disk-load execution remains on compute01, using the x86 static test binary cross-built on dev02. Quick, focused checks, root proof and foreground fast gate remain pending.
option-c-busy-green.log
2 KiB
option-c-server-green.log
2.2 KiB
option-c-removal-green.log
237 B
kayg commented 2026-11-05 06:55:01 +00:00
Author
Owner
Copy link

J196 quick precheck found two strict lint errors in the new saturation regression: a wildcard matching only Poll::Ready and a missing unit-statement semicolon. Fixed without changing assertions in adf7640e7, pushed, and restarted qualification. Formatting and doc coverage passed before the Clippy failure. The prior quick log is preserved and attached.

The current source and root-proof checkout are both adf7640e7. Qualification is queued on dev02; its only target is /home/claude/worktrees/j196/target there. Compute01 has no J196 target. Final root disk-load proof and fast gate remain pending.

J196 quick precheck found two strict lint errors in the new saturation regression: a wildcard matching only `Poll::Ready` and a missing unit-statement semicolon. Fixed without changing assertions in **`adf7640e7`**, pushed, and restarted qualification. Formatting and doc coverage passed before the Clippy failure. The prior quick log is preserved and attached. The current source and root-proof checkout are both `adf7640e7`. Qualification is queued on dev02; its only target is `/home/claude/worktrees/j196/target` there. Compute01 has no J196 target. Final root disk-load proof and fast gate remain pending.
option-c-quick-attempt1.log
23 KiB
kayg commented 2026-11-05 07:25:16 +00:00
Author
Owner
Copy link

J196 foreground progress: current pushed head remains adf7640e7; no uncommitted source changes. Quick rerun has waited just over 30 minutes on dev02, with two jobs ahead; both slots are occupied. Compute01 is also full and has a longer queue. No final success claim.

The only J196 target remains /home/claude/worktrees/j196/target on dev02. Evidence and the foreground qualification helper are under /home/claude/j196-evidence/r4. It runs quick, ordering/cancellation, security, explicit remote replay and passkey-removal tests, builds the static x86 test executable, then runs scripts/verify fast verbatim in the same slot. Root disk-load proof will use that executable in the already updated /root/j196-proof/repo on compute01, in parallel with fast once the binary is ready. All waits remain foreground.

The preserved first quick failure and its two corrected test-only lint errors are in comment 79568. No own compute01 target remains. Final root proof, fast output and cleanup are still pending.

J196 foreground progress: current pushed head remains **`adf7640e7`**; no uncommitted source changes. Quick rerun has waited just over 30 minutes on dev02, with two jobs ahead; both slots are occupied. Compute01 is also full and has a longer queue. No final success claim. The only J196 target remains `/home/claude/worktrees/j196/target` on dev02. Evidence and the foreground qualification helper are under `/home/claude/j196-evidence/r4`. It runs quick, ordering/cancellation, security, explicit remote replay and passkey-removal tests, builds the static x86 test executable, then runs `scripts/verify fast` verbatim in the same slot. Root disk-load proof will use that executable in the already updated `/root/j196-proof/repo` on compute01, in parallel with fast once the binary is ready. All waits remain foreground. The preserved first quick failure and its two corrected test-only lint errors are in comment 79568. No own compute01 target remains. Final root proof, fast output and cleanup are still pending.
kayg commented 2026-11-05 07:52:58 +00:00
Author
Owner
Copy link

J196 head adf7640e7ab794d4277426ffbf7ddf17959871ea passed scripts/verify quick on dev02: 243 agent tests and 518 server tests passed, zero failures. The explicit ordering/cancellation run passed all 7 checks; security passed all 5. Cancellation flood: flood=1024 submitted_fds=18 cancelled_fds=0 retained_fences=0. Full logs attached.

Quick output excerpt, verbatim:

test result: ok. 518 passed; 0 failed; 26 ignored; 0 measured; 0 filtered out; finished in 234.69s

(unit tests (changed): 840s)

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| scope guard               | PASS         |
| migration names           | PASS         |
| fmt                       | PASS         |
| doc coverage ratchet      | PASS         |
| clippy (workspace)        | PASS         |
| unit tests (changed)      | PASS         |
+---------------------------+--------------+

QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done.
option-c-quick_EXIT=0

This is a precheck, not the final gate. Explicit remote replay tests are compiling/running next, followed by the static x86 test build and foreground scripts/verify fast in the same slot. Root disk-load proof will run on compute01 with that exact binary. No further source changes.

J196 head **`adf7640e7ab794d4277426ffbf7ddf17959871ea`** passed `scripts/verify quick` on dev02: 243 agent tests and 518 server tests passed, zero failures. The explicit ordering/cancellation run passed all 7 checks; security passed all 5. Cancellation flood: `flood=1024 submitted_fds=18 cancelled_fds=0 retained_fences=0`. Full logs attached. Quick output excerpt, verbatim: ``` test result: ok. 518 passed; 0 failed; 26 ignored; 0 measured; 0 filtered out; finished in 234.69s (unit tests (changed): 840s) +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | scope guard | PASS | | migration names | PASS | | fmt | PASS | | doc coverage ratchet | PASS | | clippy (workspace) | PASS | | unit tests (changed) | PASS | +---------------------------+--------------+ QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done. option-c-quick_EXIT=0 ``` This is a precheck, not the final gate. Explicit remote replay tests are compiling/running next, followed by the static x86 test build and foreground `scripts/verify fast` in the same slot. Root disk-load proof will run on compute01 with that exact binary. No further source changes.
option-c-quick.log
105 KiB
option-c-ordering.log
3.3 KiB
option-c-security.log
1.5 KiB
kayg commented 2026-11-05 07:59:58 +00:00
Author
Owner
Copy link

J196 foreground qualification update on adf7640e7ab794d4277426ffbf7ddf17959871ea: all 12 remote-feature revocation tests passed, including the durable-acknowledgement/reconnect and database-reopen regressions. The passkey-removal durable-receipt test passed too. Quick, seven ordering/cancellation tests and five security tests remain green on this head (previous comment).

scripts/verify fast is running in the existing dev02 slot and sole target /home/claude/worktrees/j196/target. The x86 static root test binary built successfully. Root disk-load proof is queued through compute01 build-slot, using the compute01 fallback specified in the task; dev02 has another gate and queued work. Its writer uses idle I/O priority and is bounded to 512 MiB. No merge or deploy. Raw replay/removal/build logs attached.

J196 foreground qualification update on `adf7640e7ab794d4277426ffbf7ddf17959871ea`: all **12 remote-feature revocation tests passed**, including the durable-acknowledgement/reconnect and database-reopen regressions. The passkey-removal durable-receipt test passed too. Quick, seven ordering/cancellation tests and five security tests remain green on this head (previous comment). `scripts/verify fast` is running in the existing dev02 slot and sole target `/home/claude/worktrees/j196/target`. The x86 static root test binary built successfully. Root disk-load proof is queued through compute01 build-slot, using the compute01 fallback specified in the task; dev02 has another gate and queued work. Its writer uses idle I/O priority and is bounded to 512 MiB. No merge or deploy. Raw replay/removal/build logs attached.
option-c-replay.log
3.8 KiB
option-c-removal.log
1.8 KiB
option-c-root-build.log
3.5 KiB
kayg commented 2026-11-05 08:24:22 +00:00
Author
Owner
Copy link

J196 qualification is still foreground-supervised on immutable head adf7640e7. The fast gate passed the initial repository/release-helper checks and is now linking workspace test targets on dev02. Host memory is healthy (24 GiB available), but /proc/pressure/io reports about 90% pressure; active linker processes are progressing. No stress-ng/fio runs there. Compute01 root proof has been queued about 25 minutes with four jobs ahead; no J196 writer has started. Both own runs remain in foreground sessions. No code changed, no merge/deploy. Current focused results are attached to #79608 and #79619.

J196 qualification is still foreground-supervised on immutable head `adf7640e7`. The fast gate passed the initial repository/release-helper checks and is now linking workspace test targets on dev02. Host memory is healthy (24 GiB available), but `/proc/pressure/io` reports about 90% pressure; active linker processes are progressing. No stress-ng/fio runs there. Compute01 root proof has been queued about 25 minutes with four jobs ahead; no J196 writer has started. Both own runs remain in foreground sessions. No code changed, no merge/deploy. Current focused results are attached to #79608 and #79619.
kayg commented 2026-11-05 08:49:10 +00:00
Author
Owner
Copy link

Root disk-load proof PASS on adf7640e7ab794d4277426ffbf7ddf17959871ea, UID 0, throwaway compute01 clone, build-slot, bounded 512 MiB stress-ng HDD writer with fsync and idle I/O priority. The tests started after write_bytes=1114112; all 7 ordering/responsiveness tests, 5 security tests, the cancellation memory test, and publisher contention passed. PROOF_EXIT=0, ROOT_RUN_EXIT=0.

Measured flood evidence: flood=1024 submitted_fds=18 cancelled_fds=0 retained_fences=0; flood=4096 peak_owned_bytes=1179648 retained_owned_bytes=0. All three revocations were denied before disk completion with 18 ordinary waiters. Full root/stress logs and diskstats attached. The writer was stopped and reaped.

The fast gate is still running on dev02. It reported a failure outside the changed files: restart::tests::deleted_path_review::relative_recovery_after_atomic_install_keeps_live_install_path, panic replaced-path child did not complete. That test binary had 156 pass, 1 fail, 3 ignored under severe disk pressure. I will investigate/reproduce on unmodified origin/dev once qualification releases the sole target. No green fast claim.

Root disk-load proof PASS on `adf7640e7ab794d4277426ffbf7ddf17959871ea`, UID 0, throwaway compute01 clone, build-slot, bounded 512 MiB stress-ng HDD writer with fsync and idle I/O priority. The tests started after `write_bytes=1114112`; all 7 ordering/responsiveness tests, 5 security tests, the cancellation memory test, and publisher contention passed. `PROOF_EXIT=0`, `ROOT_RUN_EXIT=0`. Measured flood evidence: `flood=1024 submitted_fds=18 cancelled_fds=0 retained_fences=0`; `flood=4096 peak_owned_bytes=1179648 retained_owned_bytes=0`. All three revocations were denied before disk completion with 18 ordinary waiters. Full root/stress logs and diskstats attached. The writer was stopped and reaped. The fast gate is still running on dev02. It reported a failure outside the changed files: `restart::tests::deleted_path_review::relative_recovery_after_atomic_install_keeps_live_install_path`, panic `replaced-path child did not complete`. That test binary had 156 pass, 1 fail, 3 ignored under severe disk pressure. I will investigate/reproduce on unmodified origin/dev once qualification releases the sole target. No green fast claim.
root-proof.log
3.8 KiB
root-stress.log
876 B
diskstats-before.txt
972 B
diskstats-after.txt
972 B
kayg commented 2026-11-05 08:56:43 +00:00
Author
Owner
Copy link

J196 first complete fast gate on adf7640e7ab794d4277426ffbf7ddf17959871ea, dev02. Every row passed except the workspace test row: one unchanged CLI restart fixture timed out (replaced-path child did not complete, 156 pass / 1 fail / 3 ignored in that binary). Root disk-load proof and focused regressions are green, attached above.

Static D243: x86 23,716,784 bytes, baseline 22,418,256, delta +1,298,528; ARM 21,564,896 bytes, baseline 20,453,280, delta +1,111,616 (report-only).

Next foreground slot on dev02 uses the same sole target: rerun the exact branch fixture, run the CLI suite on unmodified merge-base/origin-dev a8b4ee7206d3d72f4e04aeb432d86869a3ae532c, restore the branch, and rerun scripts/verify fast verbatim. No source fix is warranted until this is distinguished from host-pressure flakiness.

Verbatim first gate result:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | FAIL (101)   |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
option-c-fast_EXIT=1
J196 first complete fast gate on `adf7640e7ab794d4277426ffbf7ddf17959871ea`, dev02. Every row passed except the workspace `test` row: one unchanged CLI restart fixture timed out (`replaced-path child did not complete`, 156 pass / 1 fail / 3 ignored in that binary). Root disk-load proof and focused regressions are green, attached above. Static D243: x86 **23,716,784 bytes**, baseline 22,418,256, delta **+1,298,528**; ARM **21,564,896 bytes**, baseline 20,453,280, delta **+1,111,616** (report-only). Next foreground slot on dev02 uses the same sole target: rerun the exact branch fixture, run the CLI suite on unmodified merge-base/origin-dev `a8b4ee7206d3d72f4e04aeb432d86869a3ae532c`, restore the branch, and rerun `scripts/verify fast` verbatim. No source fix is warranted until this is distinguished from host-pressure flakiness. Verbatim first gate result: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | FAIL (101) | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | static builds | PASS | +---------------------------+--------------+ option-c-fast_EXIT=1 ```
option-c-fast.log
454 KiB
kayg commented 2026-11-05 09:25:53 +00:00
Author
Owner
Copy link

J196 checkpoint on unchanged adf7640e7: the first-gate restart timeout did not reproduce. The exact same branch test passed alone (HEAD_RESTART_EXIT=0). The complete CLI suite on unmodified origin/dev / merge-base a8b4ee7206d3d72f4e04aeb432d86869a3ae532c passed twice: 157 passed, 0 failed, 3 ignored, each time. Logs attached; this is not evidence of a baseline failure, and the first failed gate remains disclosed above.

Branch restored and scripts/verify fast rerun verbatim, foreground, same sole target and build-slot on dev02. Early test fixtures, Clippy and publication checks passed; workspace test executables are now linking under substantial host disk pressure. No source changes. Root proof remains green (#79658).

J196 checkpoint on unchanged `adf7640e7`: the first-gate restart timeout did **not** reproduce. The exact same branch test passed alone (`HEAD_RESTART_EXIT=0`). The complete CLI suite on unmodified `origin/dev` / merge-base `a8b4ee7206d3d72f4e04aeb432d86869a3ae532c` passed twice: **157 passed, 0 failed, 3 ignored**, each time. Logs attached; this is not evidence of a baseline failure, and the first failed gate remains disclosed above. Branch restored and `scripts/verify fast` rerun verbatim, foreground, same sole target and build-slot on dev02. Early test fixtures, Clippy and publication checks passed; workspace test executables are now linking under substantial host disk pressure. No source changes. Root proof remains green (#79658).
restart-head-alone.log
246 B
restart-baseline-1.log
18 KiB
restart-baseline-2.log
18 KiB
kayg commented 2026-11-05 09:55:49 +00:00
Author
Owner
Copy link

J196 foreground checkpoint: unchanged pushed head adf7640e7ab794d4277426ffbf7ddf17959871ea. The second verbatim fast gate is still linking workspace test targets on dev02, using its single target directory. Repeated process checks show fresh compiler/linker children, so the run is progressing despite severe disk I/O pressure. No new test failure has been reported. The original restart timeout remains unreproduced: exact head rerun and two unmodified-dev CLI suites passed (#79709). Root disk-load proof and all focused revocation/ordering tests remain green (#79658, #79619). No own work is detached; still foreground-polling. No merge/deploy.

J196 foreground checkpoint: unchanged pushed head `adf7640e7ab794d4277426ffbf7ddf17959871ea`. The second verbatim fast gate is still linking workspace test targets on dev02, using its single target directory. Repeated process checks show fresh compiler/linker children, so the run is progressing despite severe disk I/O pressure. No new test failure has been reported. The original restart timeout remains unreproduced: exact head rerun and two unmodified-dev CLI suites passed (#79709). Root disk-load proof and all focused revocation/ordering tests remain green (#79658, #79619). No own work is detached; still foreground-polling. No merge/deploy.
kayg commented 2026-11-05 10:29:48 +00:00
Author
Owner
Copy link

J196 second-gate update, unchanged adf7640e7: CLI suite and real-binary infraternal-agent/session checks passed; the infraternal-agent unit suite (including all changed durable/revocation regressions) passed. The server suite is still running and has reported two failures so far: our fresh_removal_waits_for_durable_host_receipt_and_cannot_replay and unchanged socket_tests::plugin_lifecycle::disabling_mesh_joins_pending_writer_and_refuses_nonowner_switches. It has not printed buffered failure diagnostics yet. Many socket tests have >60 s notices under severe I/O pressure.

The passkey fixture has a 10 s timeout while polling the pending DB row, which may be scheduling-sensitive (it passed isolated earlier). I am waiting for the actual panic before replacing any assertion; if that is the failure, event ordering can replace the short timer without weakening premature-success detection. No production change is indicated yet. Continuing foreground supervision. The common brief requires stop/report at 2x estimate, about 16:54 IST, if still unresolved.

J196 second-gate update, unchanged `adf7640e7`: CLI suite and real-binary infraternal-agent/session checks passed; the infraternal-agent unit suite (including all changed durable/revocation regressions) passed. The server suite is still running and has reported two failures so far: our `fresh_removal_waits_for_durable_host_receipt_and_cannot_replay` and unchanged `socket_tests::plugin_lifecycle::disabling_mesh_joins_pending_writer_and_refuses_nonowner_switches`. It has not printed buffered failure diagnostics yet. Many socket tests have >60 s notices under severe I/O pressure. The passkey fixture has a 10 s timeout while polling the pending DB row, which may be scheduling-sensitive (it passed isolated earlier). I am waiting for the actual panic before replacing any assertion; if that is the failure, event ordering can replace the short timer without weakening premature-success detection. No production change is indicated yet. Continuing foreground supervision. The common brief requires stop/report at 2x estimate, about 16:54 IST, if still unresolved.
kayg commented 2026-11-05 10:49:51 +00:00
Author
Owner
Copy link

Second complete fast invocation on adf7640e7 exited 1. The restarted CLI suite passed, as did real-binary infraternal-agent checks and its unit suite. The server suite reported the removal regression and unchanged plugin-lifecycle test as failed, then stopped advancing under ~95% I/O pressure; I terminated exactly its test PID with SIGTERM so the gate could finish. This means buffered server panic diagnostics were unavailable; I do not claim the removal failure was conclusively diagnosed as its short timeout.

The shim row also failed in unchanged session::user_host::tests::exhausted_pool_drains_live_ptys_during_random_session_churn at user_host.rs:1545, left 2 / right 1 (214 passed, 1 failed, 20 ignored). All other rows, including static builds, passed. Raw log attached.

A test-only event-order replacement for the removal fixture is now being qualified: it waits for the pending row while failing immediately if removal completes first, instead of imposing a 10 s row-creation timer. Its focused remote-feature regression passed. Quick is running; the next verbatim fast gate uses RUST_TEST_THREADS=3 and uncaptured diagnostics. Production code is unchanged.

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | FAIL (101)   |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | FAIL (101)   |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
FAST_RERUN_EXIT=1
Second complete fast invocation on `adf7640e7` exited 1. The restarted CLI suite passed, as did real-binary infraternal-agent checks and its unit suite. The server suite reported the removal regression and unchanged plugin-lifecycle test as failed, then stopped advancing under ~95% I/O pressure; I terminated exactly its test PID with SIGTERM so the gate could finish. This means buffered server panic diagnostics were unavailable; I do not claim the removal failure was conclusively diagnosed as its short timeout. The shim row also failed in unchanged `session::user_host::tests::exhausted_pool_drains_live_ptys_during_random_session_churn` at `user_host.rs:1545`, left 2 / right 1 (214 passed, 1 failed, 20 ignored). All other rows, including static builds, passed. Raw log attached. A test-only event-order replacement for the removal fixture is now being qualified: it waits for the pending row while failing immediately if removal completes first, instead of imposing a 10 s row-creation timer. Its focused remote-feature regression passed. Quick is running; the next verbatim fast gate uses RUST_TEST_THREADS=3 and uncaptured diagnostics. Production code is unchanged. ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | FAIL (101) | | SSH inode reuse fixture | PASS | | shim process-per-user | FAIL (101) | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | static builds | PASS | +---------------------------+--------------+ FAST_RERUN_EXIT=1 ```
option-c-fast-rerun.log
612 KiB
kayg commented 2026-11-05 11:08:48 +00:00
Author
Owner
Copy link

Pushed ed4178aef6ea795a15fdef2763cd7a6140a80bca on agent-fsync-offloop-j196: test-only event-order observation for the removal regression. It selects the committed pending row against request completion, so a premature acknowledgement fails directly without a ten-second row-creation deadline. No production behavior changed from root-proven adf7640e7. The remote-feature focused regression passes; log attached. Strict Clippy also accepted these bytes.

Quick is still running, but has already failed in unchanged socket_tests::review_control::pending_mesh_writers_preserve_authority_pool_capacity: initial SQLite pool creation at review_control.rs:175 returned PoolTimedOut. The authority-retry tests subsequently passed. No baseline reproduction of this server fixture or the shim churn failure is established, so these are qualification blockers, not waived failures. The script stops after a failing quick; a new-head fast result is not yet available. Prior complete fast outputs are #79684 and #79794. Continuing foreground until the brief's 16:54 IST scope cutoff, then stopping/cleaning up any remaining own work and reporting exactly what remains.

Pushed **`ed4178aef6ea795a15fdef2763cd7a6140a80bca`** on `agent-fsync-offloop-j196`: test-only event-order observation for the removal regression. It selects the committed pending row against request completion, so a premature acknowledgement fails directly without a ten-second row-creation deadline. No production behavior changed from root-proven `adf7640e7`. The remote-feature focused regression passes; log attached. Strict Clippy also accepted these bytes. Quick is still running, but has already failed in unchanged `socket_tests::review_control::pending_mesh_writers_preserve_authority_pool_capacity`: initial SQLite pool creation at `review_control.rs:175` returned `PoolTimedOut`. The authority-retry tests subsequently passed. No baseline reproduction of this server fixture or the shim churn failure is established, so these are qualification blockers, not waived failures. The script stops after a failing quick; a new-head fast result is not yet available. Prior complete fast outputs are #79684 and #79794. Continuing foreground until the brief's 16:54 IST scope cutoff, then stopping/cleaning up any remaining own work and reporting exactly what remains.
event-order-focused.log
2 KiB
kayg commented 2026-11-05 11:20:15 +00:00
Author
Owner
Copy link

J196 stop report: implementation pushed; full qualification remains blocked.

Branch agent-fsync-offloop-j196, head ed4178aef6ea795a15fdef2763cd7a6140a80bca. Merged origin/dev at e89da66bf (base a8b4ee720). No merge to dev and no deployment.

Implemented owner D352 option (c): no host-wide latch; bounded per-person security admission independently of ordinary saves; durable server outbox retains denial until the infraternal agent durably acknowledges; busy replies backpressure without a finite retry budget; reconnect replays cleanup; a fresh daemon begins without positive login grants. Pending denial prevents ordinary admission, renewal and provisional access until durable receipt. Documentation is docs/operations/login-revocation-backpressure.md; the latch runbook was removed.

Atomic owner-option commits: 9711159e0 (latch removal/bounded person admission), b6e1c8e94 (busy backpressure), 24d92da75 (durable server acceptance/replay). Latest ed4178aef only changes the removal regression to observe pending-row/request ordering without a short row-creation timer. Its focused remote-feature test and strict Clippy pass. No production code changed after root-proven adf7640e7.

Evidence: production regressions failed first (comment 79452, comment 79532). Final production head passed quick (243 infraternal-agent, 518 server), 7 ordering/cancellation tests, 5 security tests, all 12 remote-feature replay/retirement tests, and passkey removal (comment 79608, comment 79619). Root disk-load proof passed on compute01 as UID 0 with a real bounded fsync writer (comment 79658): 7 ordering tests, 5 security tests, memory flood and publisher contention. flood=1024 submitted_fds=18 cancelled_fds=0 retained_fences=0; flood=4096 peak_owned_bytes=1179648 retained_owned_bytes=0. The writer had issued 1,114,112 bytes before the tests began. Root source, binary and state were removed.

All five original durable paths remain off the async runtime: login restoration directory sync; refusal-marker file/directory sync; terminal-reservation file/directory sync; reservation cleanup directory sync; login-binary publication/pruning file/directory sync. The follow-up scan found no additional runtime fsync sites; automation names already use the background-save pattern and startup edits run in a separate helper.

Qualification blockers are NOT waived. First fast failed a CLI restart child timeout; exact head rerun and two unmodified-dev CLI suites passed (comment 79709), so that timeout was not reproduced on baseline. The second fast reported removal and unchanged plugin-lifecycle failures; its server suite then stopped advancing and was terminated by exact PID with SIGTERM. The shim row failed unchanged exhausted_pool_drains_live_ptys_during_random_session_churn (left 2 / right 1). A third pre-check with three test threads passed the infraternal-agent suite, then reported SQLite fixture PoolTimedOut at review_control.rs:175 and snapshot-fixture timeout before quick's 1,500 s unit timeout. There is no baseline proof for these server/shim failures. Latest-head fast did not start because quick failed. Full task qualification is therefore incomplete.

D243 static builds passed: x86 23,716,784 bytes, baseline 22,418,256, delta +1,298,528; ARM 21,564,896 bytes, baseline 20,453,280, delta +1,111,616 (report-only).

Last complete scripts/verify fast invocation, production head adf7640e7, verbatim:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | FAIL (101)   |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | FAIL (101)   |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
FAST_RERUN_EXIT=1

Latest quick, exact test bytes committed as ed4178aef, verbatim:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| scope guard               | PASS         |
| migration names           | PASS         |
| fmt                       | PASS         |
| doc coverage ratchet      | PASS         |
| clippy (workspace)        | PASS         |
| unit tests (changed)      | FAIL (124)   |
+---------------------------+--------------+

Remaining: reproduce the server fixture and shim failures on unmodified dev, resolve or document their cause, then obtain a complete fast result on ed4178aef. Stopping within the common brief's 2x estimate leash. All own qualification processes have exited. Confirmed deleted: the dev02 root target, both crate-local target directories, J196 test state, compute01 target, and the root proof checkout/binary/state. No own queued or running build/test work remains; source worktrees and evidence are retained. Evidence remains under /home/kayg/.cache/codex-scratch/j196/r4 and /home/claude/j196-evidence/r4.

J196 stop report: implementation pushed; full qualification remains blocked. Branch `agent-fsync-offloop-j196`, head **`ed4178aef6ea795a15fdef2763cd7a6140a80bca`**. Merged origin/dev at `e89da66bf` (base `a8b4ee720`). No merge to dev and no deployment. Implemented owner D352 option (c): no host-wide latch; bounded per-person security admission independently of ordinary saves; durable server outbox retains denial until the infraternal agent durably acknowledges; busy replies backpressure without a finite retry budget; reconnect replays cleanup; a fresh daemon begins without positive login grants. Pending denial prevents ordinary admission, renewal and provisional access until durable receipt. Documentation is `docs/operations/login-revocation-backpressure.md`; the latch runbook was removed. Atomic owner-option commits: `9711159e0` (latch removal/bounded person admission), `b6e1c8e94` (busy backpressure), `24d92da75` (durable server acceptance/replay). Latest `ed4178aef` only changes the removal regression to observe pending-row/request ordering without a short row-creation timer. Its focused remote-feature test and strict Clippy pass. No production code changed after root-proven `adf7640e7`. Evidence: production regressions failed first ([comment 79452](https://git.kayg.org/kayg/infraternal/issues/347#issuecomment-79452), [comment 79532](https://git.kayg.org/kayg/infraternal/issues/347#issuecomment-79532)). Final production head passed quick (243 infraternal-agent, 518 server), 7 ordering/cancellation tests, 5 security tests, all 12 remote-feature replay/retirement tests, and passkey removal ([comment 79608](https://git.kayg.org/kayg/infraternal/issues/347#issuecomment-79608), [comment 79619](https://git.kayg.org/kayg/infraternal/issues/347#issuecomment-79619)). Root disk-load proof passed on compute01 as UID 0 with a real bounded fsync writer ([comment 79658](https://git.kayg.org/kayg/infraternal/issues/347#issuecomment-79658)): 7 ordering tests, 5 security tests, memory flood and publisher contention. `flood=1024 submitted_fds=18 cancelled_fds=0 retained_fences=0`; `flood=4096 peak_owned_bytes=1179648 retained_owned_bytes=0`. The writer had issued 1,114,112 bytes before the tests began. Root source, binary and state were removed. All five original durable paths remain off the async runtime: login restoration directory sync; refusal-marker file/directory sync; terminal-reservation file/directory sync; reservation cleanup directory sync; login-binary publication/pruning file/directory sync. The follow-up scan found no additional runtime fsync sites; automation names already use the background-save pattern and startup edits run in a separate helper. Qualification blockers are NOT waived. First fast failed a CLI restart child timeout; exact head rerun and two unmodified-dev CLI suites passed ([comment 79709](https://git.kayg.org/kayg/infraternal/issues/347#issuecomment-79709)), so that timeout was not reproduced on baseline. The second fast reported removal and unchanged plugin-lifecycle failures; its server suite then stopped advancing and was terminated by exact PID with SIGTERM. The shim row failed unchanged `exhausted_pool_drains_live_ptys_during_random_session_churn` (left 2 / right 1). A third pre-check with three test threads passed the infraternal-agent suite, then reported SQLite fixture `PoolTimedOut` at `review_control.rs:175` and snapshot-fixture timeout before quick's 1,500 s unit timeout. There is no baseline proof for these server/shim failures. Latest-head fast did not start because quick failed. Full task qualification is therefore incomplete. D243 static builds passed: x86 23,716,784 bytes, baseline 22,418,256, delta +1,298,528; ARM 21,564,896 bytes, baseline 20,453,280, delta +1,111,616 (report-only). Last complete `scripts/verify fast` invocation, production head `adf7640e7`, verbatim: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | FAIL (101) | | SSH inode reuse fixture | PASS | | shim process-per-user | FAIL (101) | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | static builds | PASS | +---------------------------+--------------+ FAST_RERUN_EXIT=1 ``` Latest quick, exact test bytes committed as `ed4178aef`, verbatim: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | scope guard | PASS | | migration names | PASS | | fmt | PASS | | doc coverage ratchet | PASS | | clippy (workspace) | PASS | | unit tests (changed) | FAIL (124) | +---------------------------+--------------+ ``` Remaining: reproduce the server fixture and shim failures on unmodified dev, resolve or document their cause, then obtain a complete fast result on `ed4178aef`. Stopping within the common brief's 2x estimate leash. All own qualification processes have exited. Confirmed deleted: the dev02 root target, both crate-local target directories, J196 test state, compute01 target, and the root proof checkout/binary/state. No own queued or running build/test work remains; source worktrees and evidence are retained. Evidence remains under `/home/kayg/.cache/codex-scratch/j196/r4` and `/home/claude/j196-evidence/r4`.
event-order-quick.log
141 KiB
event-order-qualification.log
154 B
event-order-focused.log
2 KiB
kayg commented 2026-11-05 11:22:30 +00:00
Author
Owner
Copy link

J237 independent security re-review started (review only: no push, merge or close). Target agent-fsync-offloop-j196 at ed4178aef6ea795a15fdef2763cd7a6140a80bca, merge-base with origin/dev a8b4ee720. Scope: whether J221's P1 (79308) is closed under option (c) / D352, crash windows reproduced on compute01, backpressure isolation and bound, pending-denial scope, runtime fsync paths, and the backpressure doc. Per the brief, no verify fast/quick rerun; only focused reproducers.

J237 independent security re-review started (review only: no push, merge or close). Target `agent-fsync-offloop-j196` at `ed4178aef6ea795a15fdef2763cd7a6140a80bca`, merge-base with origin/dev `a8b4ee720`. Scope: whether J221's P1 (79308) is closed under option (c) / D352, crash windows reproduced on compute01, backpressure isolation and bound, pending-denial scope, runtime fsync paths, and the backpressure doc. Per the brief, no `verify fast`/`quick` rerun; only focused reproducers.
kayg commented 2026-11-05 11:34:51 +00:00
Author
Owner
Copy link

J237 progress (review of ed4178aef). Code findings so far, by inspection. Reproducers are queued on compute01 build-slot (position 5); results will follow with logs.

  1. J221's P1 looks closed in design. The server commits a remote_login_cleanup row with confirmed=0. Certificate admission, renewal, provisional SSH, bootstrap ownership and admit_operation all add NOT EXISTS(... confirmed=0) (remote_authority.rs:36,147, remote_guard.rs:286,420, bootstrap_authorization.rs:24). revoked_at and confirmed=1 are written in one transaction, only on a durable reply (remote_authority.rs:99-122). A fresh daemon has no grant: logins.rs:462-503 keeps AUTHORIZED in process memory only, and its one producer is the server-only terminal.open/prepare on stream 0 (daemon.rs:424).
  2. Candidate P1: offline removals pin the global removal admission. remote_removal.rs:44 creates Semaphore::new(8). finish takes a permit and keeps it inside a detached task through publish → commit_revocation → wait_login_cleanup (remote_removal.rs:118-131, remote_authority.rs:270-296), and that wait has no deadline. The HTTP 15 s timeout (remote_removal_http.rs:57) drops only the waiter. A pending host stays listed (remote_inventory.rs:161 checks only revoked_at), so retries pin more permits. After 8, any owner's removal of any host fails with Denied before its revocation row commits. A reproducer is queued.
  3. Candidate P2: cleanup delivery has one global lane. LOGIN_CLEANUPS is a static Semaphore::const_new(8) (remote_runtime.rs:23), used with try-acquire and held for up to 300 s per link (remote_runtime.rs:43-47), with no per-host deduplication. A retired host whose leaf is still valid (renewal is refused but the leaf lives up to 7 days; admit_cleanup needs only an unrevoked, unexpired leaf) can hold all 8 slots from 4+4 TCP/QUIC links per source, so other hosts' cleanups get WouldBlock.
J237 progress (review of `ed4178aef`). Code findings so far, by inspection. Reproducers are queued on compute01 build-slot (position 5); results will follow with logs. 1. **J221's P1 looks closed in design.** The server commits a `remote_login_cleanup` row with `confirmed=0`. Certificate admission, renewal, provisional SSH, bootstrap ownership and `admit_operation` all add `NOT EXISTS(... confirmed=0)` (`remote_authority.rs:36,147`, `remote_guard.rs:286,420`, `bootstrap_authorization.rs:24`). `revoked_at` and `confirmed=1` are written in one transaction, only on a durable reply (`remote_authority.rs:99-122`). A fresh daemon has no grant: `logins.rs:462-503` keeps `AUTHORIZED` in process memory only, and its one producer is the server-only `terminal.open`/`prepare` on stream 0 (`daemon.rs:424`). 2. **Candidate P1: offline removals pin the global removal admission.** `remote_removal.rs:44` creates `Semaphore::new(8)`. `finish` takes a permit and keeps it inside a detached task through `publish` → `commit_revocation` → `wait_login_cleanup` (`remote_removal.rs:118-131`, `remote_authority.rs:270-296`), and that wait has no deadline. The HTTP 15 s timeout (`remote_removal_http.rs:57`) drops only the waiter. A pending host stays listed (`remote_inventory.rs:161` checks only `revoked_at`), so retries pin more permits. After 8, any owner's removal of any host fails with `Denied` before its revocation row commits. A reproducer is queued. 3. **Candidate P2: cleanup delivery has one global lane.** `LOGIN_CLEANUPS` is a static `Semaphore::const_new(8)` (`remote_runtime.rs:23`), used with try-acquire and held for up to 300 s per link (`remote_runtime.rs:43-47`), with no per-host deduplication. A retired host whose leaf is still valid (renewal is refused but the leaf lives up to 7 days; `admit_cleanup` needs only an unrevoked, unexpired leaf) can hold all 8 slots from 4+4 TCP/QUIC links per source, so other hosts' cleanups get `WouldBlock`.
kayg commented 2026-11-05 12:40:50 +00:00
Author
Owner
Copy link

J237 independent security re-review: REQUEST CHANGES

Target agent-fsync-offloop-j196 at ed4178aef6ea795a15fdef2763cd7a6140a80bca (merge-base a8b4ee720). Review only: nothing pushed, merged or closed. Per the brief I did not rerun verify fast or verify quick. Evidence is in claude@infraternal-dev-01-compute01:~/j237-evidence/ (review-tests.diff, agent.log, server.log, removal.log, plus the two first-round logs explained below). The VM target, test state and worktree are deleted; the logs are kept.

1. J221's P1: closed

  • Server. The server commits a remote_login_cleanup row with confirmed=0 before it delivers anything. Admission (remote_authority.rs:36), renewal (:147), provisional SSH (remote_guard.rs:286), provisioning operations (remote_guard.rs:420) and bootstrap ownership (bootstrap_authorization.rs:24) all refuse a host with a pending row. revoked_at and confirmed=1 are written in one transaction, and only on a durable reply (remote_authority.rs:99-122).
  • Delivery. A cleanup-only link can carry only Hello, Ping, Pong and terminal.login.revoke (remote_link.rs:756-772). It never enters the host map (remote_runtime.rs:374-376).
  • Agent. Positive grants live only in process memory (logins.rs:462-503). Their only source is the server's terminal.open/prepare on stream 0 (daemon.rs:424). Pending and on-disk refusals override any grant.
  • Agent crash windows, reproduced as real abrupt child-process exits with the server's replay order applied in a fresh process (agent.log):
J237 window=before_write lose_unsynced=false marker_after_stop=false fresh_denied=true replay=ok resync_refused=true still_denied=true unrelated_permitted=true
J237 window=before_fsync lose_unsynced=false marker_after_stop=true fresh_denied=true replay=ok resync_refused=true still_denied=true unrelated_permitted=true
J237 window=before_fsync lose_unsynced=true marker_after_stop=false fresh_denied=true replay=ok resync_refused=true still_denied=true unrelated_permitted=true
J237 window=before_ack lose_unsynced=false marker_after_stop=true fresh_denied=true replay=ok resync_refused=true still_denied=true unrelated_permitted=true
test result: ok. 6 passed; 0 failed   (J196's 5 + J237)
  • After the ack, before the server records it. A SQLite RAISE(ABORT) trigger stops the confirm transaction after the durable reply arrives. Then: a store restart, the trigger removed, and a fresh reconnect (server.log):
J237 after-ack-before-record: confirmed=false retired=false ordinary_refused=true cleanup_only=true waiter_pending=true
J237 replay after restart: confirmed=true retired=true
test result: ok. 13 passed; 0 failed   (J196's 12 + J237)
  • Server restart before acceptance is covered by J196's pending_revocation_survives_database_reopen_before_acceptance and unconfirmed_cleanup_replays_after_restart…, both green here.
  • I found no path that grants access after the server has committed the denial.

Findings

P1: offline removals take the server-wide removal admission, so unrelated revocations are refused (breaks D352: "only the affected people are blocked … no host-wide lockout").

  • Cause. remote_removal.rs:44 creates one Semaphore::new(8) for the whole server. finish (remote_removal.rs:118-131) moves its permit into a detached task that runs publish → commit_revocation → wait_login_cleanup (remote_authority.rs:270-296). That wait now has no deadline: it ends only when the host replies durably. On dev it returned after about 300 ms.
  • Effect. The 15 s HTTP timeout (remote_removal_http.rs:57) drops only the request. The task keeps the permit until that host reconnects, which may be never: removing a dead host is the usual case.
  • Reproduced. After eight removals of offline hosts, an unrelated host's removal fails with Denied before its revocation row commits. So that host is not revoked at all (removal.log):
J237 removal admission: offline_removals_pending=8 unrelated_revocation_accepted=false unrelated_early_result=Ok(Ok(Err(Denied)))
panicked at crates/infraternal-server/src/auth/remote_removal_tests.rs:219:5: eight pending offline removals refused an unrelated host's revocation
  • Recovery is a server restart, and no document mentions it. Removal is the product's only UI path for revoking a host; nothing in apps/ or packages/ calls /remote/revocations.
  • Possible fix. Release the admission permit once the row has committed. The committed outbox, not a held permit, is what keeps the obligation alive. The waiter can then subscribe without a permit, or the removal can answer "pending".

P2: one global cleanup-delivery lane, which a single retired host can occupy.

  • LOGIN_CLEANUPS is a static Semaphore::const_new(8), taken with try-acquire (remote_runtime.rs:23,33-35).
  • Each cleanup link holds its slot for up to 300 s (remote_runtime.rs:43-47), and nothing deduplicates per host.
  • admit_cleanup (remote_authority.rs:61-71) accepts any unexpired, unrevoked leaf of a pending host. Renewal is refused, but a leaf lives up to 7 days.
  • The TCP and QUIC listeners each allow 4 links per source address (remote_accept.rs:34-35), and cleanup links keep their handshake permits for the whole wait. So one retired host that holds its links open without replying can take all 8 slots and keep re-dialing. Every other host's cleanup then gets WouldBlock, and its retirement never confirms.
  • This fails closed: no access is granted. But a stuck or hostile agent blocks other people's revocations from completing.
  • I found this by inspection and did not reproduce it.
  • Possible fix. Limit cleanup to one in-flight link per host, and take slots from a per-host or fair queue.

P3: the doc is accurate on the mechanism but leaves out operator-visible outcomes (docs/operations/login-revocation-backpressure.md). I checked these claims against the code and they hold: the 18-person bound with busy-before-retain, no host latch, cleanup-only links, a fresh daemon starting with no grants, the row surviving a restart, and no finite retry budget. Missing:

  • Removing an offline host returns an HTTP timeout or 403 to the person while the denial is in fact committed.
  • The pending host still appears in inventory (remote_inventory.rs:161 checks only revoked_at).
  • Once the host's leaf expires, the row can never be delivered and stays pending forever. The doc gives no operator action for that case.

P3: the J196 regression depends on the working directory. remote_login_revocation_tests.rs:497-508 falls back to a relative target/. Without CARGO_TARGET_DIR set it fails with SqliteError 14 unable to open database file; my first run shows this (server-r1-no-target-dir.log). With the variable set it passes. This is the brief's known J155 gotcha, not a product defect.

(My first removal reproducer retried the same host and was invalid: retries fail fast on UNIQUE remote_host_removals.host_id and do not take more permits. See removal-r1-invalid-same-host.log. The corrected version uses eight distinct hosts.)

Other checks

  • 2. Agent backpressure. One person's flood coalesces into a single slot. A full registry returns busy before it retains anything (revocations.rs:103-111). The server retries every 100 ms with no budget (remote_guard.rs:134-151). No event-loop blocking.
  • 3. Pending denial. It blocks ordinary admission, renewal, provisional SSH and restore for that host's person. Unrelated people stay permitted, as the agent test above shows (unrelated_permitted=true).
  • 4. Event-loop latency. Every sync_all site runs inside Admission::run (spawn_blocking) or in the startup-guard helper process: logins.rs:582,692-693, rootless_sessions.rs:73-74,120, login_binary.rs:327-328,414,421, startup_guard.rs:286,297. This confirms J196's five paths, and I found no new fsync on the runtime. Rootless cleanup keeps its 5 s timeout (rootless_sessions.rs:109).
  • J196's gate. I read #79814. The last full fast (on adf7640e7) failed test and shim process-per-user. quick on ed4178aef failed with 124. Qualification is still incomplete, as J196 reported.

Verdict: REQUEST CHANGES (1×P1, 1×P2, 2×P3). J221's P1 is closed.

J237 independent security re-review: **REQUEST CHANGES** Target `agent-fsync-offloop-j196` at `ed4178aef6ea795a15fdef2763cd7a6140a80bca` (merge-base `a8b4ee720`). Review only: nothing pushed, merged or closed. Per the brief I did not rerun `verify fast` or `verify quick`. Evidence is in `claude@infraternal-dev-01-compute01:~/j237-evidence/` (`review-tests.diff`, `agent.log`, `server.log`, `removal.log`, plus the two first-round logs explained below). The VM target, test state and worktree are deleted; the logs are kept. ## 1. J221's P1: closed - **Server.** The server commits a `remote_login_cleanup` row with `confirmed=0` before it delivers anything. Admission (`remote_authority.rs:36`), renewal (`:147`), provisional SSH (`remote_guard.rs:286`), provisioning operations (`remote_guard.rs:420`) and bootstrap ownership (`bootstrap_authorization.rs:24`) all refuse a host with a pending row. `revoked_at` and `confirmed=1` are written in one transaction, and only on a durable reply (`remote_authority.rs:99-122`). - **Delivery.** A cleanup-only link can carry only Hello, Ping, Pong and `terminal.login.revoke` (`remote_link.rs:756-772`). It never enters the host map (`remote_runtime.rs:374-376`). - **Agent.** Positive grants live only in process memory (`logins.rs:462-503`). Their only source is the server's `terminal.open`/`prepare` on stream 0 (`daemon.rs:424`). Pending and on-disk refusals override any grant. - **Agent crash windows**, reproduced as real abrupt child-process exits with the server's replay order applied in a fresh process (`agent.log`): ```text J237 window=before_write lose_unsynced=false marker_after_stop=false fresh_denied=true replay=ok resync_refused=true still_denied=true unrelated_permitted=true J237 window=before_fsync lose_unsynced=false marker_after_stop=true fresh_denied=true replay=ok resync_refused=true still_denied=true unrelated_permitted=true J237 window=before_fsync lose_unsynced=true marker_after_stop=false fresh_denied=true replay=ok resync_refused=true still_denied=true unrelated_permitted=true J237 window=before_ack lose_unsynced=false marker_after_stop=true fresh_denied=true replay=ok resync_refused=true still_denied=true unrelated_permitted=true test result: ok. 6 passed; 0 failed (J196's 5 + J237) ``` - **After the ack, before the server records it.** A SQLite `RAISE(ABORT)` trigger stops the confirm transaction after the durable reply arrives. Then: a store restart, the trigger removed, and a fresh reconnect (`server.log`): ```text J237 after-ack-before-record: confirmed=false retired=false ordinary_refused=true cleanup_only=true waiter_pending=true J237 replay after restart: confirmed=true retired=true test result: ok. 13 passed; 0 failed (J196's 12 + J237) ``` - **Server restart before acceptance** is covered by J196's `pending_revocation_survives_database_reopen_before_acceptance` and `unconfirmed_cleanup_replays_after_restart…`, both green here. - I found no path that grants access after the server has committed the denial. ## Findings **P1: offline removals take the server-wide removal admission, so unrelated revocations are refused (breaks D352: "only the affected people are blocked … no host-wide lockout").** - **Cause.** `remote_removal.rs:44` creates one `Semaphore::new(8)` for the whole server. `finish` (`remote_removal.rs:118-131`) moves its permit into a detached task that runs `publish` → `commit_revocation` → `wait_login_cleanup` (`remote_authority.rs:270-296`). That wait now has no deadline: it ends only when the host replies durably. On dev it returned after about 300 ms. - **Effect.** The 15 s HTTP timeout (`remote_removal_http.rs:57`) drops only the request. The task keeps the permit until that host reconnects, which may be never: removing a dead host is the usual case. - **Reproduced.** After eight removals of offline hosts, an unrelated host's removal fails with `Denied` before its revocation row commits. So that host is not revoked at all (`removal.log`): ```text J237 removal admission: offline_removals_pending=8 unrelated_revocation_accepted=false unrelated_early_result=Ok(Ok(Err(Denied))) panicked at crates/infraternal-server/src/auth/remote_removal_tests.rs:219:5: eight pending offline removals refused an unrelated host's revocation ``` - **Recovery** is a server restart, and no document mentions it. Removal is the product's only UI path for revoking a host; nothing in `apps/` or `packages/` calls `/remote/revocations`. - **Possible fix.** Release the admission permit once the row has committed. The committed outbox, not a held permit, is what keeps the obligation alive. The waiter can then subscribe without a permit, or the removal can answer "pending". **P2: one global cleanup-delivery lane, which a single retired host can occupy.** - `LOGIN_CLEANUPS` is a static `Semaphore::const_new(8)`, taken with try-acquire (`remote_runtime.rs:23,33-35`). - Each cleanup link holds its slot for up to 300 s (`remote_runtime.rs:43-47`), and nothing deduplicates per host. - `admit_cleanup` (`remote_authority.rs:61-71`) accepts any unexpired, unrevoked leaf of a pending host. Renewal is refused, but a leaf lives up to 7 days. - The TCP and QUIC listeners each allow 4 links per source address (`remote_accept.rs:34-35`), and cleanup links keep their handshake permits for the whole wait. So one retired host that holds its links open without replying can take all 8 slots and keep re-dialing. Every other host's cleanup then gets `WouldBlock`, and its retirement never confirms. - This fails closed: no access is granted. But a stuck or hostile agent blocks other people's revocations from completing. - I found this by inspection and did not reproduce it. - **Possible fix.** Limit cleanup to one in-flight link per host, and take slots from a per-host or fair queue. **P3: the doc is accurate on the mechanism but leaves out operator-visible outcomes** (`docs/operations/login-revocation-backpressure.md`). I checked these claims against the code and they hold: the 18-person bound with busy-before-retain, no host latch, cleanup-only links, a fresh daemon starting with no grants, the row surviving a restart, and no finite retry budget. Missing: - Removing an offline host returns an HTTP timeout or 403 to the person while the denial is in fact committed. - The pending host still appears in inventory (`remote_inventory.rs:161` checks only `revoked_at`). - Once the host's leaf expires, the row can never be delivered and stays pending forever. The doc gives no operator action for that case. **P3: the J196 regression depends on the working directory.** `remote_login_revocation_tests.rs:497-508` falls back to a relative `target/`. Without `CARGO_TARGET_DIR` set it fails with `SqliteError 14 unable to open database file`; my first run shows this (`server-r1-no-target-dir.log`). With the variable set it passes. This is the brief's known J155 gotcha, not a product defect. (My first removal reproducer retried the same host and was invalid: retries fail fast on `UNIQUE remote_host_removals.host_id` and do not take more permits. See `removal-r1-invalid-same-host.log`. The corrected version uses eight distinct hosts.) ## Other checks - **2. Agent backpressure.** One person's flood coalesces into a single slot. A full registry returns busy before it retains anything (`revocations.rs:103-111`). The server retries every 100 ms with no budget (`remote_guard.rs:134-151`). No event-loop blocking. - **3. Pending denial.** It blocks ordinary admission, renewal, provisional SSH and restore for that host's person. Unrelated people stay permitted, as the agent test above shows (`unrelated_permitted=true`). - **4. Event-loop latency.** Every `sync_all` site runs inside `Admission::run` (`spawn_blocking`) or in the startup-guard helper process: `logins.rs:582,692-693`, `rootless_sessions.rs:73-74,120`, `login_binary.rs:327-328,414,421`, `startup_guard.rs:286,297`. This confirms J196's five paths, and I found no new fsync on the runtime. Rootless cleanup keeps its 5 s timeout (`rootless_sessions.rs:109`). - **J196's gate.** I read #79814. The last full `fast` (on `adf7640e7`) failed `test` and `shim process-per-user`. `quick` on `ed4178aef` failed with 124. Qualification is still incomplete, as J196 reported. **Verdict: REQUEST CHANGES** (1×P1, 1×P2, 2×P3). J221's P1 is closed.
kayg commented 2026-11-05 12:47:32 +00:00
Author
Owner
Copy link

J196 continuing J237 review 79885 on branch agent-fsync-offloop-j196, base/head ed4178aef6ea795a15fdef2763cd7a6140a80bca. Plan: merge current origin/dev; reproduce the offline admission failure; release admission after durable outbox commit and return pending; cap cleanup delivery at one per host with fair bounded admission; expose/document pending and retired-unreachable; make the regression path independent of cwd; run focused red/green tests, quick, then fast in the foreground. Estimate: 180 minutes including shared build queues. Existing restart protection remains required. Evidence: /home/kayg/.cache/codex-scratch/j196/r5/, reviewer evidence /home/claude/j237-evidence/ on compute01. No merge to dev or deployment.

J196 continuing J237 review 79885 on branch `agent-fsync-offloop-j196`, base/head `ed4178aef6ea795a15fdef2763cd7a6140a80bca`. Plan: merge current origin/dev; reproduce the offline admission failure; release admission after durable outbox commit and return pending; cap cleanup delivery at one per host with fair bounded admission; expose/document pending and retired-unreachable; make the regression path independent of cwd; run focused red/green tests, quick, then fast in the foreground. Estimate: 180 minutes including shared build queues. Existing restart protection remains required. Evidence: `/home/kayg/.cache/codex-scratch/j196/r5/`, reviewer evidence `/home/claude/j237-evidence/` on compute01. No merge to dev or deployment.
kayg commented 2026-11-05 12:53:51 +00:00
Author
Owner
Copy link

J196 contract detail for review 79885: removal will return state: pending once the denial/outbox row commits, with “revoked, cleanup pending”; removed remains reserved for durable host acceptance. Inventory adds state: pending | retired-unreachable on undelivered cleanup rows, always offline; delivered rows disappear. retired-unreachable is derived from the exact certificate ledger when no individually unrevoked leaf has remaining validity, without inventing a durable acknowledgement or reopening access. The retained unconfirmed row continues denying admission/renewal. Existing UI/CLI adapters will carry the pending result instead of presenting an error or removing the pending host. Cleanup delivery will have eight active distinct hosts and sixteen FIFO waiting distinct hosts; duplicate/full admission returns busy before retaining a link actor. No new dependencies or background poller.

J196 contract detail for review 79885: removal will return `state: pending` once the denial/outbox row commits, with “revoked, cleanup pending”; `removed` remains reserved for durable host acceptance. Inventory adds `state: pending | retired-unreachable` on undelivered cleanup rows, always offline; delivered rows disappear. `retired-unreachable` is derived from the exact certificate ledger when no individually unrevoked leaf has remaining validity, without inventing a durable acknowledgement or reopening access. The retained unconfirmed row continues denying admission/renewal. Existing UI/CLI adapters will carry the pending result instead of presenting an error or removing the pending host. Cleanup delivery will have eight active distinct hosts and sixteen FIFO waiting distinct hosts; duplicate/full admission returns busy before retaining a link actor. No new dependencies or background poller.
kayg commented 2026-11-05 13:09:01 +00:00
Author
Owner
Copy link

J196 red evidence after integrating current origin/dev authorization paths: offline_removals_do_not_block_other_revocations fails with Denied on the ninth distinct offline host; cleanup duplicate-host and FIFO-wait tests both fail; direct invocation from a directory without target and with CARGO_TARGET_DIR unset reproduces SQLite error 14. Raw log attached and retained at /home/claude/j196-evidence/r5/red.log on compute01 and control scratch j196/r5/red.log. The first status-expiry fixture run was invalid (it changed certificate lifetime away from the required seven days); corrected fixture is queued before the status implementation. Client parser and palette status tests also failed first, then passed after the targeted adapters were changed (hosts-red.log, hosts-green.log, palette-red.log, palette-green.log under the same VM evidence directory). No final gate claim yet.

J196 red evidence after integrating current origin/dev authorization paths: `offline_removals_do_not_block_other_revocations` fails with `Denied` on the ninth distinct offline host; cleanup duplicate-host and FIFO-wait tests both fail; direct invocation from a directory without target and with CARGO_TARGET_DIR unset reproduces SQLite error 14. Raw log attached and retained at `/home/claude/j196-evidence/r5/red.log` on compute01 and control scratch `j196/r5/red.log`. The first status-expiry fixture run was invalid (it changed certificate lifetime away from the required seven days); corrected fixture is queued before the status implementation. Client parser and palette status tests also failed first, then passed after the targeted adapters were changed (`hosts-red.log`, `hosts-green.log`, `palette-red.log`, `palette-green.log` under the same VM evidence directory). No final gate claim yet.
red.log
20 KiB
kayg commented 2026-11-05 13:10:03 +00:00
Author
Owner
Copy link

J196 checkpoint pushed at 147f5ae38 on agent-fsync-offloop-j196: origin/dev merge plus the removal-admission fix and pending response adapters. This is explicitly WIP, not qualified. The focused green run acquired compute01 slot 3; its log is /home/claude/j196-evidence/r5/focused.log. One target globally: /home/claude/worktrees/j196/target. Next: confirm focused results, complete inventory expiry status, run browser evidence and quick/fast in the foreground. No merge/deploy.

J196 checkpoint pushed at `147f5ae38` on `agent-fsync-offloop-j196`: origin/dev merge plus the removal-admission fix and pending response adapters. This is explicitly WIP, not qualified. The focused green run acquired compute01 slot 3; its log is `/home/claude/j196-evidence/r5/focused.log`. One target globally: `/home/claude/worktrees/j196/target`. Next: confirm focused results, complete inventory expiry status, run browser evidence and quick/fast in the foreground. No merge/deploy.
kayg commented 2026-11-05 13:32:40 +00:00
Author
Owner
Copy link

J196 checkpoint 6796309cc pushed to preserve all in-progress work during the queue wait. This is WIP and will be split into atomic verified commits. Removal regression: 6/6 pass; new cleanup cap/FIFO regressions: 2/2 pass (focused.log, attached). Corrected status fixture fails on missing pending as intended. Replay recheck is queued with unique fixture host IDs and a disconnect-observation fence. I also added an intentionally red resource-count test before transferring cleanup links out of normal handshake admission, so the new waiting queue cannot consume the reconnect reserve. Current queued run: /home/claude/j196-evidence/r5/focused2.log; exactly one target remains /home/claude/worktrees/j196/target. No gate claim, merge or deploy.

J196 checkpoint `6796309cc` pushed to preserve all in-progress work during the queue wait. This is WIP and will be split into atomic verified commits. Removal regression: 6/6 pass; new cleanup cap/FIFO regressions: 2/2 pass (`focused.log`, attached). Corrected status fixture fails on missing `pending` as intended. Replay recheck is queued with unique fixture host IDs and a disconnect-observation fence. I also added an intentionally red resource-count test before transferring cleanup links out of normal handshake admission, so the new waiting queue cannot consume the reconnect reserve. Current queued run: `/home/claude/j196-evidence/r5/focused2.log`; exactly one target remains `/home/claude/worktrees/j196/target`. No gate claim, merge or deploy.
focused.log
11 KiB
kayg commented 2026-11-05 13:54:52 +00:00
Author
Owner
Copy link

J196 checkpoint: the P2 handoff regression failed first (focused2.log: cleanup retained both normal handshake permits). TCP and QUIC now release those permits after the validated cleanup Hello, before joining the bounded distinct-host delivery lane. Ordinary links retain their permits. The focused3 combined server/replay/cwd/production-browser run is queued on compute01 (PID 523298, log ~/j196-evidence/r5/focused3.log); both VMs currently have all slots busy. I am polling in the foreground. Final split commits, current-dev merge, quick and fast remain. Exactly one target remains: ~/worktrees/j196/target on compute01.

J196 checkpoint: the P2 handoff regression failed first (`focused2.log`: cleanup retained both normal handshake permits). TCP and QUIC now release those permits after the validated cleanup Hello, before joining the bounded distinct-host delivery lane. Ordinary links retain their permits. The `focused3` combined server/replay/cwd/production-browser run is queued on compute01 (PID 523298, log `~/j196-evidence/r5/focused3.log`); both VMs currently have all slots busy. I am polling in the foreground. Final split commits, current-dev merge, quick and fast remain. Exactly one target remains: `~/worktrees/j196/target` on compute01.
kayg commented 2026-11-05 14:15:51 +00:00
Author
Owner
Copy link

J196 WIP checkpoint aac1102a2 pushed. focused3.log passes six removal tests, three cleanup-admission/handoff tests, and the pending/expiry inventory regression. The replay suite passed 12 tests but stalled in busy-reply-then-disconnect replay; I stopped that exact test PID, so this run is exit 101, not a green gate or a waived flake.

Cause: after a busy reply the private cleanup actor can lose its socket during retry backoff, then wait for an impossible replacement while retaining its host claim. The delivery owner now watches the actor’s unavailable state and ends that attempt, retaining the durable row for reconnect. A queued in-slot red/green run first requires the regression to fail with its generous deadlock guard, then applies this fix and repeats all focused checks/browser specs. PID 921776, ~/j196-evidence/r5/lifecycle.log on compute01. Both VMs were busy when chosen; compute01 had the shorter queue. Final atomic split, current-dev merge, quick/fast, and target cleanup remain.

J196 WIP checkpoint `aac1102a2` pushed. `focused3.log` passes six removal tests, three cleanup-admission/handoff tests, and the pending/expiry inventory regression. The replay suite passed 12 tests but stalled in busy-reply-then-disconnect replay; I stopped that exact test PID, so this run is exit 101, not a green gate or a waived flake. Cause: after a busy reply the private cleanup actor can lose its socket during retry backoff, then wait for an impossible replacement while retaining its host claim. The delivery owner now watches the actor’s unavailable state and ends that attempt, retaining the durable row for reconnect. A queued in-slot red/green run first requires the regression to fail with its generous deadlock guard, then applies this fix and repeats all focused checks/browser specs. PID 921776, `~/j196-evidence/r5/lifecycle.log` on compute01. Both VMs were busy when chosen; compute01 had the shorter queue. Final atomic split, current-dev merge, quick/fast, and target cleanup remain.
focused3.log
9.9 KiB
kayg commented 2026-11-05 14:41:54 +00:00
Author
Owner
Copy link

J196 atomic fixes pushed on agent-fsync-offloop-j196, head e6cc8573e3e5f6e3cf96c93ec9c584aa12fa85fb, merged with current origin/dev 32a2f7046. No merge to dev or deployment.

  • aa2fc6d50: release removal admission after durable denial commit, and report pending. The ninth distinct offline removal is recorded.
  • 7a41ae060: one host claim across eight active and sixteen FIFO waiting cleanup deliveries; ordinary handshake capacity is returned after the validated Hello. A busy-reply disconnect ends its private attempt without losing the durable row.
  • a30bb0196: unique temporary restart database independent of cwd/Cargo output.
  • 36fbfe2d5: document five module declarations required by current D345 checks.
  • 5ab4395d1: pending and retired-unreachable inventory/status adapters, tests, contract and operations documentation.
  • e6cc8573e: merge current dev and resolve lint-metadata conflicts while preserving the durable ordering documentation.

Red/green evidence: attached disconnect-red.log shows the retained-host-claim deadlock guard fail first. Attached lifecycle.log records six removal tests, three delivery/handoff tests, the pending/expiry inventory test, all thirteen cleanup/replay tests, and direct compiled invocation of the restart regression from /home/claude/j196-cwd with CARGO_TARGET_DIR unset, all passing. Parser and palette regressions each pass; typecheck succeeds; all seven production-build browser specs pass across 1440/393/375 and ink/paper, with icon/text centre and replacement left-edge assertions within 0.5px. All twelve pending/unreachable screenshots were inspected at their native DPR2 desktop/DPR3 phone resolution and are attached for UI review; inherited mobile hints elide to fit the row.

Quick ran verbatim on this exact head and stopped at shape lints off (D351): crates/plugins/mesh/tests/gotatun_persistence.rs retains a local shape allow. This branch has no diff in that file. A clean checkout of unmodified origin/dev 32a2f7046 reproduces the identical failure (origin-shape.log, ORIGIN_SHAPE_EXIT=1). No gate code was changed or failure hidden. Other cheap quick rows passed: scope, migration names, fmt, doc coverage, changed-item docs, fixed waits. Full quick output attached.

scripts/verify fast is now queued through build-slot on compute01 (PID1209593, ~/j196-evidence/r5/fast-final.log), polled in the foreground. Both queues were checked first. Exactly one target remains, ~/worktrees/j196/target there; it will be deleted after final evidence. No final gate claim yet.

J196 atomic fixes pushed on `agent-fsync-offloop-j196`, head `e6cc8573e3e5f6e3cf96c93ec9c584aa12fa85fb`, merged with current `origin/dev` `32a2f7046`. No merge to dev or deployment. - `aa2fc6d50`: release removal admission after durable denial commit, and report pending. The ninth distinct offline removal is recorded. - `7a41ae060`: one host claim across eight active and sixteen FIFO waiting cleanup deliveries; ordinary handshake capacity is returned after the validated Hello. A busy-reply disconnect ends its private attempt without losing the durable row. - `a30bb0196`: unique temporary restart database independent of cwd/Cargo output. - `36fbfe2d5`: document five module declarations required by current D345 checks. - `5ab4395d1`: pending and retired-unreachable inventory/status adapters, tests, contract and operations documentation. - `e6cc8573e`: merge current dev and resolve lint-metadata conflicts while preserving the durable ordering documentation. Red/green evidence: attached `disconnect-red.log` shows the retained-host-claim deadlock guard fail first. Attached `lifecycle.log` records six removal tests, three delivery/handoff tests, the pending/expiry inventory test, all thirteen cleanup/replay tests, and direct compiled invocation of the restart regression from `/home/claude/j196-cwd` with `CARGO_TARGET_DIR` unset, all passing. Parser and palette regressions each pass; typecheck succeeds; all seven production-build browser specs pass across 1440/393/375 and ink/paper, with icon/text centre and replacement left-edge assertions within 0.5px. All twelve pending/unreachable screenshots were inspected at their native DPR2 desktop/DPR3 phone resolution and are attached for UI review; inherited mobile hints elide to fit the row. Quick ran verbatim on this exact head and stopped at `shape lints off (D351)`: `crates/plugins/mesh/tests/gotatun_persistence.rs` retains a local shape allow. This branch has no diff in that file. A clean checkout of unmodified `origin/dev` `32a2f7046` reproduces the identical failure (`origin-shape.log`, `ORIGIN_SHAPE_EXIT=1`). No gate code was changed or failure hidden. Other cheap quick rows passed: scope, migration names, fmt, doc coverage, changed-item docs, fixed waits. Full quick output attached. `scripts/verify fast` is now queued through build-slot on compute01 (PID1209593, `~/j196-evidence/r5/fast-final.log`), polled in the foreground. Both queues were checked first. Exactly one target remains, `~/worktrees/j196/target` there; it will be deleted after final evidence. No final gate claim yet.
lifecycle.log
17 KiB
disconnect-red.log
2.5 KiB
quick-final.log
8.4 KiB
origin-shape.log
1.1 KiB
1440-ink-pending.png
225 KiB
1440-ink-unreachable.png
233 KiB
1440-paper-pending.png
225 KiB
1440-paper-unreachable.png
233 KiB
375-ink-pending.png
155 KiB
375-ink-unreachable.png
156 KiB
375-paper-pending.png
160 KiB
375-paper-unreachable.png
160 KiB
393-ink-pending.png
161 KiB
393-ink-unreachable.png
163 KiB
393-paper-pending.png
166 KiB
393-paper-unreachable.png
168 KiB
1440-ink-pending.png 1440-ink-unreachable.png 1440-paper-pending.png 1440-paper-unreachable.png 375-ink-pending.png 375-ink-unreachable.png 375-paper-pending.png 375-paper-unreachable.png 393-ink-pending.png 393-ink-unreachable.png 393-paper-pending.png 393-paper-unreachable.png
kayg commented 2026-11-05 14:55:52 +00:00
Author
Owner
Copy link

J196 gate correction: the first scripts/verify fast run on e6cc8573e reached clippy and reported two unreadable_literal errors in the new expiry fixture (604800). These were my changes, not a baseline failure. I stopped that failed run and its exact descendant PIDs, preserved ~/j196-evidence/r5/fast-e6-first.log, and pushed the separator-only fix d0cade92e.

Dev advanced, so I merged ce1f10932 too. Current pushed head is 810a90aab36e2c3d75e0c1cb950a0355706353bc. Quick followed by fast is queued on compute01 (PID1344578; ~/j196-evidence/r5/gates-retry.log), monitored in the foreground. The one upstream shape-lint failure reproduces on clean current dev as well (~/j196-evidence/r5/origin-shape-current.log, exit1). No gate/lint infrastructure was changed. The one target remains ~/worktrees/j196/target until evidence is captured and cleanup finishes.

J196 gate correction: the first `scripts/verify fast` run on `e6cc8573e` reached clippy and reported two `unreadable_literal` errors in the new expiry fixture (`604800`). These were my changes, not a baseline failure. I stopped that failed run and its exact descendant PIDs, preserved `~/j196-evidence/r5/fast-e6-first.log`, and pushed the separator-only fix `d0cade92e`. Dev advanced, so I merged `ce1f10932` too. Current pushed head is `810a90aab36e2c3d75e0c1cb950a0355706353bc`. Quick followed by fast is queued on compute01 (PID1344578; `~/j196-evidence/r5/gates-retry.log`), monitored in the foreground. The one upstream shape-lint failure reproduces on clean current dev as well (`~/j196-evidence/r5/origin-shape-current.log`, exit1). No gate/lint infrastructure was changed. The one target remains `~/worktrees/j196/target` until evidence is captured and cleanup finishes.
fast-e6-first.log
54 KiB
kayg commented 2026-11-05 15:11:06 +00:00
Author
Owner
Copy link

J196 foreground gate update at head 810a90aab36e2c3d75e0c1cb950a0355706353bc (merged origin/dev ce1f10932). Clippy and privileged integration passed; the full gate is still running its release checks on compute01. Log: ~/j196-evidence/r5/fast-final.log.

The workspace test row failed in real_binary_rootless_revocation_after_restart_stops_every_owned_terminal: during initial terminal selection, before restart/revocation, agent_fixture.py:420 asserted (seq, self.out) == (385, 0). I have not classified this as a flake. Once the gate exits, I will run the same daemon suite on clean origin/dev in this worktree/target, and run the final-head ordering and removal/cleanup regressions separately because the workspace run stopped before reaching those libraries.

Quick's unchanged shape-lint failure was also reproduced on clean current origin/dev ce1f10932 (origin-shape-current.log). No gate or lint infrastructure changed. The one target is still /home/claude/worktrees/j196/target on compute01 and will be deleted after verification.

J196 foreground gate update at head `810a90aab36e2c3d75e0c1cb950a0355706353bc` (merged origin/dev `ce1f10932`). Clippy and privileged integration passed; the full gate is still running its release checks on compute01. Log: `~/j196-evidence/r5/fast-final.log`. The workspace test row failed in `real_binary_rootless_revocation_after_restart_stops_every_owned_terminal`: during initial terminal selection, before restart/revocation, `agent_fixture.py:420` asserted `(seq, self.out) == (385, 0)`. I have not classified this as a flake. Once the gate exits, I will run the same daemon suite on clean origin/dev in this worktree/target, and run the final-head ordering and removal/cleanup regressions separately because the workspace run stopped before reaching those libraries. Quick's unchanged shape-lint failure was also reproduced on clean current origin/dev `ce1f10932` (`origin-shape-current.log`). No gate or lint infrastructure changed. The one target is still `/home/claude/worktrees/j196/target` on compute01 and will be deleted after verification.
kayg commented 2026-11-05 15:40:02 +00:00
Author
Owner
Copy link

J196 checkpoint, still foreground at unchanged pushed head 810a90aab36e2c3d75e0c1cb950a0355706353bc. The full fast gate has completed x86 static build/checks and is in arm64 final compilation/linking. X86 reports PT_INTERP: absent; size: 24911824 bytes; baseline 22418256; delta +2493568 bytes (report only, D243 amended 2026-11-01). This is the configured gate baseline, not a matched current-origin build comparison. Dependency duplicates remain 55 vs 55 (+0).

The same two rows remain failed: terminal fixture sequence assertion in the workspace test row (not yet classified), and the unchanged shape-lint exception independently reproduced on origin/dev. Shim process-per-user passed (215 passed, 20 ignored); privileged integration and release-probe exclusion passed. Full log remains ~/j196-evidence/r5/fast-final.log on compute01. No code edits during verification, no merge to dev or deployment. After the gate I am running focused final-head checks and the baseline daemon reproducer, then deleting the target.

J196 checkpoint, still foreground at unchanged pushed head `810a90aab36e2c3d75e0c1cb950a0355706353bc`. The full fast gate has completed x86 static build/checks and is in arm64 final compilation/linking. X86 reports `PT_INTERP: absent; size: 24911824 bytes; baseline 22418256; delta +2493568 bytes (report only, D243 amended 2026-11-01)`. This is the configured gate baseline, not a matched current-origin build comparison. Dependency duplicates remain 55 vs 55 (+0). The same two rows remain failed: terminal fixture sequence assertion in the workspace test row (not yet classified), and the unchanged shape-lint exception independently reproduced on origin/dev. Shim process-per-user passed (215 passed, 20 ignored); privileged integration and release-probe exclusion passed. Full log remains `~/j196-evidence/r5/fast-final.log` on compute01. No code edits during verification, no merge to dev or deployment. After the gate I am running focused final-head checks and the baseline daemon reproducer, then deleting the target.
kayg commented 2026-11-05 15:42:37 +00:00
Author
Owner
Copy link

J196 full gate completed in the foreground on compute01 at pushed head 810a90aab36e2c3d75e0c1cb950a0355706353bc, including merged origin/dev ce1f10932. scripts/verify quick exited 1 on the upstream shape-lint exception; the attached clean-origin log reproduces it at the same dev SHA. scripts/verify fast ran verbatim and exited 1. Verbatim final output:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | FAIL (101)   |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | FAIL (1)     |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| privileged integration    | PASS         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
FAST_EXIT=1
FAST_DONE

Static output (deltas against the gate's configured baseline, not a matched origin/dev build):

PT_INTERP: absent; size: 24911824 bytes; baseline 22418256; delta +2493568 bytes (report only, D243 amended 2026-11-01)
PT_INTERP: absent; size: 22589864 bytes; baseline 20453280; delta +2136584 bytes (report only, D243 amended 2026-11-01)

The workspace test row stopped at the terminal setup assertion described in #80021; its origin/dev comparison is still pending, so I am not waiving it as a flake. The six ordering tests and server cleanup/removal regressions are also being run separately at this final head because the workspace run stopped before those library tests.

Both queues were checked after the gate. Dev-02 had no queued jobs versus two on compute01, so these remaining checks moved to dev-02 (~/j196-evidence/r5/final-checks.log, runner PID 710203). The compute01 target and test state are deleted; the only new target will be /home/claude/worktrees/j196/target on dev-02 and will be deleted afterward. No source change during the gate, no merge to dev, no deployment. Focused/browser red-green evidence and screenshots remain attached at #79999.

J196 full gate completed in the foreground on compute01 at pushed head **`810a90aab36e2c3d75e0c1cb950a0355706353bc`**, including merged origin/dev `ce1f10932`. `scripts/verify quick` exited 1 on the upstream shape-lint exception; the attached clean-origin log reproduces it at the same dev SHA. `scripts/verify fast` ran verbatim and exited 1. Verbatim final output: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | FAIL (101) | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | FAIL (1) | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | PASS | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | static builds | PASS | +---------------------------+--------------+ FAST_EXIT=1 FAST_DONE ``` Static output (deltas against the gate's configured baseline, not a matched origin/dev build): ```text PT_INTERP: absent; size: 24911824 bytes; baseline 22418256; delta +2493568 bytes (report only, D243 amended 2026-11-01) PT_INTERP: absent; size: 22589864 bytes; baseline 20453280; delta +2136584 bytes (report only, D243 amended 2026-11-01) ``` The workspace test row stopped at the terminal setup assertion described in #80021; its origin/dev comparison is still pending, so I am not waiving it as a flake. The six ordering tests and server cleanup/removal regressions are also being run separately at this final head because the workspace run stopped before those library tests. Both queues were checked after the gate. Dev-02 had no queued jobs versus two on compute01, so these remaining checks moved to dev-02 (`~/j196-evidence/r5/final-checks.log`, runner PID 710203). The compute01 target and test state are deleted; the only new target will be `/home/claude/worktrees/j196/target` on dev-02 and will be deleted afterward. No source change during the gate, no merge to dev, no deployment. Focused/browser red-green evidence and screenshots remain attached at #79999.
fast-final.log
466 KiB
quick-final.log
8.4 KiB
origin-shape-current.log
1 KiB
kayg commented 2026-11-05 15:56:23 +00:00
Author
Owner
Copy link

J196 final-head focused checks are green on dev-02 at 810a90aab36e2c3d75e0c1cb950a0355706353bc: six ordering/event-loop responsiveness tests; six removal tests (including nine distinct offline hosts recorded); three cleanup admission/FIFO/handoff tests; pending-to-unreachable inventory test; thirteen login revocation/replay tests. FOCUSED_EXIT=0 (attached log).

The package-only daemon suite on clean origin/dev ce1f10932 passed 11/11. Twenty isolated repetitions of the gate's exact failing test, with workspace feature resolution on that same clean origin/dev, also passed on arm64. Logs attached. This has not reproduced the x86 failure, so I am not calling it a known flake or waiving it.

I am now comparing on compute01, the architecture/host where the gate failed, with a bounded loop that stops at the first failure (maximum 100 attempts). This specific x86 comparison must use compute01 although dev-02 is free. It is through build-slot; log ~/j196-evidence/r5/x86-baseline-repeat.log. Dev-02's target and test state have been deleted, and its checkout is restored to the branch head. Only the new compute01 proof target will exist, and it will be deleted when the comparison finishes. Full fast output is attached at #80070; there has been no source edit since that gate.

J196 final-head focused checks are green on dev-02 at `810a90aab36e2c3d75e0c1cb950a0355706353bc`: six ordering/event-loop responsiveness tests; six removal tests (including nine distinct offline hosts recorded); three cleanup admission/FIFO/handoff tests; pending-to-unreachable inventory test; thirteen login revocation/replay tests. `FOCUSED_EXIT=0` (attached log). The package-only daemon suite on clean origin/dev `ce1f10932` passed 11/11. Twenty isolated repetitions of the gate's exact failing test, with workspace feature resolution on that same clean origin/dev, also passed on arm64. Logs attached. This has **not** reproduced the x86 failure, so I am **not** calling it a known flake or waiving it. I am now comparing on compute01, the architecture/host where the gate failed, with a bounded loop that stops at the first failure (maximum 100 attempts). This specific x86 comparison must use compute01 although dev-02 is free. It is through build-slot; log `~/j196-evidence/r5/x86-baseline-repeat.log`. Dev-02's target and test state have been deleted, and its checkout is restored to the branch head. Only the new compute01 proof target will exist, and it will be deleted when the comparison finishes. Full fast output is attached at #80070; there has been no source edit since that gate.
final-focused.log
24 KiB
origin-daemon.log
12 KiB
arm-baseline-repeat.log
38 KiB
kayg commented 2026-11-05 16:10:03 +00:00
Author
Owner
Copy link

The x86 clean-origin comparison at ce1f10932 passed 100/100 isolated repetitions of real_binary_rootless_revocation_after_restart_stops_every_owned_terminal, with workspace feature resolution, on compute01. The attached raw log records every attempt and restores the branch to 810a90aab36e2c3d75e0c1cb950a0355706353bc afterward. Together with the arm64 comparisons, this still does not reproduce the original gate assertion. I am not waiving the test row as an upstream flake.

I have queued the last bounded comparison: up to five clean-origin runs of the complete agent_daemon suite with RUST_TEST_THREADS=3, matching the parallel test context of the gate. It stops at the first failure. Log: compute01 ~/j196-evidence/r5/x86-baseline-suite.log, PID 2165986. No code change or additional full gate run. The final-head six ordering tests and 23 focused server regressions remain green; fast's exact output is in #80070. The single compute01 proof target is retained only for this comparison and will be deleted afterward. Dev-02's target is deleted.

The x86 clean-origin comparison at `ce1f10932` passed **100/100 isolated repetitions** of `real_binary_rootless_revocation_after_restart_stops_every_owned_terminal`, with workspace feature resolution, on compute01. The attached raw log records every attempt and restores the branch to `810a90aab36e2c3d75e0c1cb950a0355706353bc` afterward. Together with the arm64 comparisons, this still does **not** reproduce the original gate assertion. I am not waiving the test row as an upstream flake. I have queued the last bounded comparison: up to five clean-origin runs of the complete `agent_daemon` suite with `RUST_TEST_THREADS=3`, matching the parallel test context of the gate. It stops at the first failure. Log: compute01 `~/j196-evidence/r5/x86-baseline-suite.log`, PID 2165986. No code change or additional full gate run. The final-head six ordering tests and 23 focused server regressions remain green; fast's exact output is in #80070. The single compute01 proof target is retained only for this comparison and will be deleted afterward. Dev-02's target is deleted.
x86-baseline-repeat.log
192 KiB
dependency-duplicates.log
54 B
kayg commented 2026-11-05 16:26:35 +00:00
Author
Owner
Copy link

J196 verification finding; branch remains at 810a90aab36e2c3d75e0c1cb950a0355706353bc with no source changes since fast.

The five complete x86 baseline daemon-suite runs all passed (11 tests each, RUST_TEST_THREADS=3, workspace features, clean origin/dev ce1f10932). Attached raw log. Alongside the one arm64 full suite and 120 isolated baseline repetitions already attached, the original integration assertion has not recurred. I am not claiming those runs reproduced it.

Source inspection found a specific unsupported fixture assumption: agent_fixture.py:467-478 returns from fence() at a stream-0 Pong, and select_terminal immediately resets the stream-1 output cursor. However, daemon_writer.rs:112 deliberately prioritizes the control queue over already-queued bulk output. The existing control_precedes_bulk_without_reordering_snapshot_notice test explicitly protects that priority. Thus an old stream-1 output can legally follow the Pong and trip the reset cursor assertion. Changing control-first scheduling would harm the latency priority.

I have queued a deterministic test-only baseline probe to confirm this exact path: put Output(seq=385) in the real writer's bulk queue and Pong in its control queue before polling it; feed its emitted bytes to the unchanged Python fixture, then perform the same fence/reset/process sequence. The expected result is the same AssertionError: (385, 0). This has not run yet and is not being claimed as evidence until it does.

Current runner: compute01 PID 2281048, ~/j196-evidence/r5/fence-probe.log, through build-slot. Temporary harness: fence-probe-test.rs appended only to the baseline test module; its exact diff will be saved as fence-probe.diff, then the checkout restored to the pushed head. The only target remains /home/claude/worktrees/j196/target on compute01. Dev-02 target is deleted. This diagnostic and cleanup are the remaining work; the requested final-head six ordering and 23 server regressions passed, and full fast output is attached at #80070.

J196 verification finding; branch remains at `810a90aab36e2c3d75e0c1cb950a0355706353bc` with no source changes since fast. The five complete x86 baseline daemon-suite runs all passed (11 tests each, `RUST_TEST_THREADS=3`, workspace features, clean origin/dev `ce1f10932`). Attached raw log. Alongside the one arm64 full suite and 120 isolated baseline repetitions already attached, the original integration assertion has not recurred. I am not claiming those runs reproduced it. Source inspection found a specific unsupported fixture assumption: `agent_fixture.py:467-478` returns from `fence()` at a stream-0 Pong, and `select_terminal` immediately resets the stream-1 output cursor. However, `daemon_writer.rs:112` deliberately prioritizes the control queue over already-queued bulk output. The existing `control_precedes_bulk_without_reordering_snapshot_notice` test explicitly protects that priority. Thus an old stream-1 output can legally follow the Pong and trip the reset cursor assertion. Changing control-first scheduling would harm the latency priority. I have queued a deterministic test-only baseline probe to confirm this exact path: put Output(seq=385) in the real writer's bulk queue and Pong in its control queue before polling it; feed its emitted bytes to the unchanged Python fixture, then perform the same fence/reset/process sequence. The expected result is the same `AssertionError: (385, 0)`. This has not run yet and is not being claimed as evidence until it does. Current runner: compute01 PID **2281048**, `~/j196-evidence/r5/fence-probe.log`, through build-slot. Temporary harness: `fence-probe-test.rs` appended only to the baseline test module; its exact diff will be saved as `fence-probe.diff`, then the checkout restored to the pushed head. The only target remains `/home/claude/worktrees/j196/target` on compute01. Dev-02 target is deleted. This diagnostic and cleanup are the remaining work; the requested final-head six ordering and 23 server regressions passed, and full fast output is attached at #80070.
x86-baseline-suite.log
30 KiB
kayg commented 2026-11-05 16:31:21 +00:00
Author
Owner
Copy link

J249 security re-review started (review only: no push, merge or close). Target agent-fsync-offloop-j196 at 810a90aab36e2c3d75e0c1cb950a0355706353bc, delta since ed4178aef (J196 commits aa2fc6d50, 7a41ae060, a30bb0196, 36fbfe2d5, 5ab4395d1, d0cade92e plus dev merges). Scope: J237's P1 (removal admission), P2 (cleanup lane fairness/per-host cap), both P3s, J221's P1 crash/replay recheck, and whether the real-binary daemon test failure (80070/80100/80116) is caused by this branch. Per the brief, no verify fast/quick; only focused reproducers on compute01. Evidence will go to claude@infraternal-dev-01-compute01:~/j249-evidence/.

J249 security re-review started (review only: no push, merge or close). Target `agent-fsync-offloop-j196` at `810a90aab36e2c3d75e0c1cb950a0355706353bc`, delta since `ed4178aef` (J196 commits `aa2fc6d50`, `7a41ae060`, `a30bb0196`, `36fbfe2d5`, `5ab4395d1`, `d0cade92e` plus dev merges). Scope: J237's P1 (removal admission), P2 (cleanup lane fairness/per-host cap), both P3s, J221's P1 crash/replay recheck, and whether the real-binary daemon `test` failure (80070/80100/80116) is caused by this branch. Per the brief, no `verify fast`/`quick`; only focused reproducers on compute01. Evidence will go to `claude@infraternal-dev-01-compute01:~/j249-evidence/`.
kayg commented 2026-11-05 16:45:41 +00:00
Author
Owner
Copy link

J196 final report: agent-fsync-offloop-j196, pushed head 810a90aab36e2c3d75e0c1cb950a0355706353bc, including origin/dev ce1f10932589da84b7d87e88ae302ba2bfb66aaf. No merge to dev, deployment, or issue closure.

J237's requested changes are implemented in separate commits:

  • aa2fc6d50: offline removal releases admission after its durable denial commits and reports revoked, cleanup pending. Nine distinct offline removals are recorded; no offline receipt waiter retains the eight-slot admission pool.
  • 7a41ae060: cleanup delivery has one claim per host, eight active distinct hosts, sixteen FIFO waiters, prompt cancellation/disconnect release, and separate ordinary handshake capacity.
  • a30bb0196: the restart regression uses a unique system temporary path, independent of cwd and CARGO_TARGET_DIR.
  • 5ab4395d1: inventory, CLI/UI adapters, contracts, and operations documentation expose pending cleanup and retired-unreachable once no usable leaf remains. Pending hosts remain listed without shell/removal/privilege actions; expired delivery does not forge a durable receipt.

Fail-first evidence, the seven production-browser checks, and twelve screenshots at desktop/phone widths are attached at #79999. Final-head checks passed: six ordering/responsiveness tests, six removal tests, three cleanup capacity/fairness/handoff tests, one pending/expiry test, and thirteen revocation/replay tests (29 total; final-focused.log, attached at #80085).

Qualification remains red; no gate failure is hidden. Quick exits 1 on the unchanged shape-lint exception in crates/plugins/mesh/tests/gotatun_persistence.rs, independently reproduced on clean current origin/dev (origin-shape-current.log, #80070). Fast was run verbatim in the foreground at this head; full raw output is attached at #80070. Its final output is:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | FAIL (101)   |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | FAIL (1)     |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| privileged integration    | PASS         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+
FAST_EXIT=1
FAST_DONE

The test row hit agent_fixture.py:420 with AssertionError: (385, 0), while switching terminals during setup, before restart/revocation. I did not reproduce that integration failure in 120 isolated baseline repetitions or six complete baseline daemon-suite runs (arm64 and x86 logs in #80085, #80100, #80116). Those passing comparisons are not being described as failures.

The attached deterministic baseline probe does reproduce the exact assertion against origin/dev product code, with only a temporary test harness added. It runs the real control-first writer with an already-queued old output, then feeds its actual bytes through the unchanged Python fixture's fence/reset/process path:

BASELINE_TRACE=Pong(stream=0, nonce=1), Output(stream=1, seq=385); fixture cursor reset 385->0
AssertionError: (385, 0)
FENCE_PROBE_EXIT=101

This identifies the unsupported fence assumption: Binary.fence() returns at a control Pong, but the control-first writer may still have older terminal output queued. Resetting the reused terminal stream's cursor then trips the assertion. The existing control_precedes_bulk_without_reordering_snapshot_notice test explicitly protects control-first scheduling. The writer and all three relevant fixture/integration files are identical between origin/dev and this head (fence-code-comparison.log: exit 0). The diagnostic used event ordering, no timing oracle, and changed no product code. Its exact test-only diff is attached as fence-probe-diff.log because Forgejo rejected raw Rust/shell attachments. The temporary harness was removed and the checkout restored. A fixture fix needs a genuine terminal-close fence or fresh stream IDs; this branch preserves the control-first latency behavior.

Static builds passed for both shipping targets:

PT_INTERP: absent; size: 24911824 bytes; baseline 22418256; delta +2493568 bytes (report only, D243 amended 2026-11-01)
PT_INTERP: absent; size: 22589864 bytes; baseline 20453280; delta +2136584 bytes (report only, D243 amended 2026-11-01)

These are deltas against the configured gate baselines, not matched current-origin static builds. Dependency duplicate report: 55 → 55, delta +0 (#80100); no dependency added.

Cleanup is complete: both VM checkouts are restored to the pushed head and clean, all J196 test/build runners have ended, and the compute01 and dev-02 targets/test state are deleted. The control worktree has no target. Evidence remains in ~/j196-evidence/r5/ on the respective VMs and /home/kayg/.cache/codex-scratch/j196/r5/ on the control host. The remaining qualification blockers are the upstream shape-lint exception and the existing fixture fence assumption; the gate is not claimed green.

J196 final report: **`agent-fsync-offloop-j196`**, pushed head **`810a90aab36e2c3d75e0c1cb950a0355706353bc`**, including origin/dev **`ce1f10932589da84b7d87e88ae302ba2bfb66aaf`**. No merge to dev, deployment, or issue closure. J237's requested changes are implemented in separate commits: - `aa2fc6d50`: offline removal releases admission after its durable denial commits and reports **revoked, cleanup pending**. Nine distinct offline removals are recorded; no offline receipt waiter retains the eight-slot admission pool. - `7a41ae060`: cleanup delivery has one claim per host, eight active distinct hosts, sixteen FIFO waiters, prompt cancellation/disconnect release, and separate ordinary handshake capacity. - `a30bb0196`: the restart regression uses a unique system temporary path, independent of cwd and `CARGO_TARGET_DIR`. - `5ab4395d1`: inventory, CLI/UI adapters, contracts, and operations documentation expose pending cleanup and **retired-unreachable** once no usable leaf remains. Pending hosts remain listed without shell/removal/privilege actions; expired delivery does not forge a durable receipt. Fail-first evidence, the seven production-browser checks, and twelve screenshots at desktop/phone widths are attached at #79999. Final-head checks passed: **six ordering/responsiveness tests**, **six removal tests**, **three cleanup capacity/fairness/handoff tests**, **one pending/expiry test**, and **thirteen revocation/replay tests** (29 total; `final-focused.log`, attached at #80085). **Qualification remains red; no gate failure is hidden.** Quick exits 1 on the unchanged shape-lint exception in `crates/plugins/mesh/tests/gotatun_persistence.rs`, independently reproduced on clean current origin/dev (`origin-shape-current.log`, #80070). Fast was run verbatim in the foreground at this head; full raw output is attached at #80070. Its final output is: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | FAIL (101) | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | FAIL (1) | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | PASS | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | static builds | PASS | +---------------------------+--------------+ FAST_EXIT=1 FAST_DONE ``` The test row hit `agent_fixture.py:420` with **`AssertionError: (385, 0)`**, while switching terminals during setup, before restart/revocation. I did **not** reproduce that integration failure in 120 isolated baseline repetitions or six complete baseline daemon-suite runs (arm64 and x86 logs in #80085, #80100, #80116). Those passing comparisons are not being described as failures. The attached **deterministic baseline probe does reproduce the exact assertion** against origin/dev product code, with only a temporary test harness added. It runs the real control-first writer with an already-queued old output, then feeds its actual bytes through the unchanged Python fixture's fence/reset/process path: ```text BASELINE_TRACE=Pong(stream=0, nonce=1), Output(stream=1, seq=385); fixture cursor reset 385->0 AssertionError: (385, 0) FENCE_PROBE_EXIT=101 ``` This identifies the unsupported fence assumption: `Binary.fence()` returns at a control Pong, but the control-first writer may still have older terminal output queued. Resetting the reused terminal stream's cursor then trips the assertion. The existing `control_precedes_bulk_without_reordering_snapshot_notice` test explicitly protects control-first scheduling. The writer and all three relevant fixture/integration files are identical between origin/dev and this head (`fence-code-comparison.log`: exit 0). The diagnostic used event ordering, no timing oracle, and changed no product code. Its exact test-only diff is attached as `fence-probe-diff.log` because Forgejo rejected raw Rust/shell attachments. The temporary harness was removed and the checkout restored. A fixture fix needs a genuine terminal-close fence or fresh stream IDs; this branch preserves the control-first latency behavior. Static builds passed for both shipping targets: ```text PT_INTERP: absent; size: 24911824 bytes; baseline 22418256; delta +2493568 bytes (report only, D243 amended 2026-11-01) PT_INTERP: absent; size: 22589864 bytes; baseline 20453280; delta +2136584 bytes (report only, D243 amended 2026-11-01) ``` These are deltas against the configured gate baselines, not matched current-origin static builds. Dependency duplicate report: **55 → 55, delta +0** (#80100); no dependency added. Cleanup is complete: both VM checkouts are restored to the pushed head and clean, all J196 test/build runners have ended, and the compute01 and dev-02 targets/test state are deleted. The control worktree has no target. Evidence remains in `~/j196-evidence/r5/` on the respective VMs and `/home/kayg/.cache/codex-scratch/j196/r5/` on the control host. The remaining qualification blockers are the upstream shape-lint exception and the existing fixture fence assumption; the gate is not claimed green.
fence-probe.log
3.8 KiB
fence-probe-diff.log
2.7 KiB
fence-code-comparison.log
122 B
cleanup-compute01.log
66 B
cleanup-dev02.log
62 B
kayg commented 2026-11-05 17:02:03 +00:00
Author
Owner
Copy link

J249 security re-review: REQUEST CHANGES (1×P2, 3×P3). Every security item passes: J237's P1 and P2 are fixed, both of J237's P3s are fixed, and J221's P1 still holds.

Target agent-fsync-offloop-j196 at 810a90aab36e2c3d75e0c1cb950a0355706353bc, delta since ed4178aef. Review only: nothing pushed, merged or closed, and no verify fast/quick per the brief. Evidence is in claude@infraternal-dev-01-compute01:~/j249-evidence/: review-tests.diff (J237's reproducers adapted to the admin-window API), run.sh, removal.log, server.log, admission.log, agent.log, cwd.log. The worktree and target are deleted.

J237 P1, removal admission: fixed

  • commit_revocation_intent (remote_authority.rs:234-289) commits and publishes the denial, then returns. publish (remote_removal.rs:329-350) answers state: pending and doesn't wait, so the permit taken at remote_removal.rs:134 / :184 is released when the request returns.
  • J237's 8-offline-hosts reproducer, rerun with nine distinct hosts and uninstall=true as the UI sends it, then a concurrent burst of nine more (removal.log):
J249 removal 0..8: state="pending" pending_row=true tombstone_absent=true host_cleanup="unreachable" message="host.example: revoked, cleanup pending. Its infraternal agent must reconnect to confirm durable revocation."
J249 concurrent burst: accepted=9/9
test result: ok. 7 passed; 0 failed
  • The CLI maps pending to CommandState::Pending (host_remove.rs:52-56). The UI shows the message as a toast (host-removal.on-demand.ts:35-38).

J237 P2, cleanup lane: fixed for TCP/QUIC

  • CleanupSlots (remote_runtime.rs:23-85) gives each host one claim across 8 active and 16 waiting slots. A duplicate gets WouldBlock before any actor exists. Waiters are served in tokio's FIFO order, and a dropped waiter releases its claim. A host that re-dials after its 300 s slot ends goes to the back of the queue, so one retired host can't hold more than one slot. cleanup_delivery_deduplicates_hosts and cleanup_delivery_is_fair_and_bounded are green (admission.log).
  • TCP and QUIC hand the handshake permits back after a validated cleanup Hello (remote_accept.rs:122-124, :247-248). The WebSocket path doesn't: see P3-1.

J237 P3s: fixed

  • Doc: docs/operations/login-revocation-backpressure.md now covers the pending response, the "removal pending" row, delivery bounds, retired-unreachable after certificate expiry, and the operator path. One gap is in P2-1.
  • Test cwd: a30bb0196 uses temp_dir() and removes the file. I ran the compiled test from /tmp/j249-cwd with CARGO_TARGET_DIR and INFRATERNAL_TEST_STATE_ROOT unset, and it passed (cwd.log).

J221 P1 recheck: still holds

The agent crate changes only doc comments in this delta. On 810a90aab, I reran J237's crash windows: real abrupt child exits, then a fresh process replaying the server's order (agent.log):

J237 window=before_write lose_unsynced=false marker_after_stop=false fresh_denied=true replay=ok resync_refused=true still_denied=true unrelated_permitted=true
J237 window=before_fsync lose_unsynced=false marker_after_stop=true  fresh_denied=true replay=ok resync_refused=true still_denied=true unrelated_permitted=true
J237 window=before_fsync lose_unsynced=true  marker_after_stop=false fresh_denied=true replay=ok resync_refused=true still_denied=true unrelated_permitted=true
J237 window=before_ack   lose_unsynced=false marker_after_stop=true  fresh_denied=true replay=ok resync_refused=true still_denied=true unrelated_permitted=true
test result: ok. 6 passed; 0 failed

Server side (server.log, 14 passed): receipt lost before record gives confirmed=false retired=false ordinary_refused=true cleanup_only=true, then after a restart confirmed=true retired=true. pending_revocation_survives_database_reopen_before_acceptance and unconfirmed_cleanup_replays_after_restart_without_granting_remote_access are green. That covers the delta's new window, a server crash after the pending answer and before delivery. The INSERT and the denial share one transaction. I found no path that grants access again.

Findings

P2-1: removing an offline host now loses the leftover-install command (#235 regression).

  • When pending, publish overwrites the outcome message (remote_removal.rs:342-347), and the bootstrap row it needs is deleted in the same transaction (remote_removal.rs:305).
  • The UI's pending branch returns before the manual "Copy command" toast (host-removal.on-demand.ts:35-38 vs :40-48).
  • The human CLI prints only message; data is {} unless --machine (host_remove.rs:57-71).
  • For a UUID owner, every removal is pending, so the command is lost on every removal. On origin/dev an unreachable host returns "…infraternal is still installed on it. Remove it there as root with: …" and the copy action.
  • Reproduced: all nine responses carry manual="/usr/local/bin/infraternal --bootstrap-uninstall <id>" that nobody shows (removal.log).
  • A dead host never reconnects, so the person is told to wait for a reconnect that won't happen. The doc's retired-unreachable step says to use "its ownership-checked uninstall command" but not where to find it.
  • Fix: keep outcome()'s sentence and manual together with "revoked, cleanup pending", and keep the Copy-command toast on the pending branch. The message must also be right for an online host, where request_uninstall returned requested. Today it says "must reconnect" there too.

P3-1: the WebSocket transport still holds handshake permits through cleanup queueing and delivery (remote_agent_http.rs:137-147).

  • The per-source (4) and global (64) HTTPS agent-link permits stay held through the FIFO wait and the up-to-300 s delivery. TCP and QUIC release them (remote_accept.rs:122-124, :247-248).
  • J196's report (79999) and the doc say ordinary handshake capacity is returned.
  • Effect: four pending hosts behind one NAT address block that address's ordinary HTTPS reconnects for that time, and 24 cleanup links take 24 of the 64 global permits.
  • The per-host claim now does the job the comment gives for keeping them (bounding detached actors), so they can be released as on the other transports.
  • Found by inspection, not reproduced.

P3-2: the P1 fix is inside a merge commit. aa2fc6d50 is the merge of dev eb0684cbf. Its remerge diff holds the whole behaviour change: remote_authority.rs, remote_removal.rs, host_remove.rs, host-removal.on-demand.ts and the tests. git log --no-merges, bisect and review-by-commit don't show it. CLAUDE.md asks for atomic commits that don't mix changes. Split the fix out of the merge when the branch is next rebuilt.

P3-3: retired-unreachable rows have no way out. They have no actions (contracts/remote-host-inventory.md) and stay listed for good. That matches the doc, but nobody can clear the row after doing the manual uninstall. This is a product call for kayg, not a blocker.

Gate failures J196 reported: none caused by this branch

  • test row, real_binary_rootless_revocation_after_restart_stops_every_owned_terminal AssertionError: (385, 0) (80070): not this branch. The branch doesn't touch daemon_writer.rs, scripts/agent_fixture.py or scripts/test-rootless-revoke-live.py (git diff --stat ce1f10932 810a90aab).
    • J196's deterministic probe on clean origin/dev ce1f10932 (~/j196-evidence/r5/fence-probe.log) fills the real writer's bulk and control queues and gets Pong(stream=0) then Output(stream=1, seq=385). The unchanged fixture then fails with the identical AssertionError: (385, 0).
    • Cause: fence() (agent_fixture.py:467-478) treats Pong as a drain barrier for terminal output, but the writer sends control frames first by design. detach() → select_terminal() (:694-700) then resets the cursor while the old terminal's output is still queued.
    • It's a latent dev fixture race. Fix it in the fixture (drain the detached terminal to its close before resetting), in its own issue.
    • One caveat: the branch's off-loop fsync changes timing on the open path, which could make the race more frequent. none of the 120 isolated runs and 6 full-suite runs on baseline reproduced it, and nobody has looped the branch head.
  • 80100/80116 are J196's baseline comparisons (100/100 isolated and 5/5 suite runs green on x86 dev). They're consistent with a rare race, and the probe above shows how it happens.
  • shape lints off (D351): upstream. The #[allow(clippy::too_many_lines)] at crates/plugins/mesh/tests/gotatun_persistence.rs:147 is on dev ce1f10932 and not on current origin/dev 19737d86d. Merging current dev clears it. The branch's own leftover allow in remote_removal.rs from the aa2fc6d50 conflict is gone at 810a90aab.

Verdict: REQUEST CHANGES: fix P2-1 (keep the manual uninstall command and an accurate message on pending). The P3s can ride along. The security changes are approved as they stand.

J249 security re-review: **REQUEST CHANGES** (1×P2, 3×P3). Every security item passes: J237's P1 and P2 are fixed, both of J237's P3s are fixed, and J221's P1 still holds. Target `agent-fsync-offloop-j196` at `810a90aab36e2c3d75e0c1cb950a0355706353bc`, delta since `ed4178aef`. Review only: nothing pushed, merged or closed, and no `verify fast`/`quick` per the brief. Evidence is in `claude@infraternal-dev-01-compute01:~/j249-evidence/`: `review-tests.diff` (J237's reproducers adapted to the admin-window API), `run.sh`, `removal.log`, `server.log`, `admission.log`, `agent.log`, `cwd.log`. The worktree and target are deleted. ## J237 P1, removal admission: fixed - `commit_revocation_intent` (`remote_authority.rs:234-289`) commits and publishes the denial, then returns. `publish` (`remote_removal.rs:329-350`) answers `state: pending` and doesn't wait, so the permit taken at `remote_removal.rs:134` / `:184` is released when the request returns. - J237's 8-offline-hosts reproducer, rerun with nine distinct hosts and `uninstall=true` as the UI sends it, then a concurrent burst of nine more (`removal.log`): ```text J249 removal 0..8: state="pending" pending_row=true tombstone_absent=true host_cleanup="unreachable" message="host.example: revoked, cleanup pending. Its infraternal agent must reconnect to confirm durable revocation." J249 concurrent burst: accepted=9/9 test result: ok. 7 passed; 0 failed ``` - The CLI maps `pending` to `CommandState::Pending` (`host_remove.rs:52-56`). The UI shows the message as a toast (`host-removal.on-demand.ts:35-38`). ## J237 P2, cleanup lane: fixed for TCP/QUIC - `CleanupSlots` (`remote_runtime.rs:23-85`) gives each host one claim across 8 active and 16 waiting slots. A duplicate gets `WouldBlock` before any actor exists. Waiters are served in tokio's FIFO order, and a dropped waiter releases its claim. A host that re-dials after its 300 s slot ends goes to the back of the queue, so one retired host can't hold more than one slot. `cleanup_delivery_deduplicates_hosts` and `cleanup_delivery_is_fair_and_bounded` are green (`admission.log`). - TCP and QUIC hand the handshake permits back after a validated cleanup Hello (`remote_accept.rs:122-124`, `:247-248`). The WebSocket path doesn't: see P3-1. ## J237 P3s: fixed - **Doc:** `docs/operations/login-revocation-backpressure.md` now covers the `pending` response, the "removal pending" row, delivery bounds, `retired-unreachable` after certificate expiry, and the operator path. One gap is in P2-1. - **Test cwd:** `a30bb0196` uses `temp_dir()` and removes the file. I ran the compiled test from `/tmp/j249-cwd` with `CARGO_TARGET_DIR` and `INFRATERNAL_TEST_STATE_ROOT` unset, and it passed (`cwd.log`). ## J221 P1 recheck: still holds The agent crate changes only doc comments in this delta. On `810a90aab`, I reran J237's crash windows: real abrupt child exits, then a fresh process replaying the server's order (`agent.log`): ```text J237 window=before_write lose_unsynced=false marker_after_stop=false fresh_denied=true replay=ok resync_refused=true still_denied=true unrelated_permitted=true J237 window=before_fsync lose_unsynced=false marker_after_stop=true fresh_denied=true replay=ok resync_refused=true still_denied=true unrelated_permitted=true J237 window=before_fsync lose_unsynced=true marker_after_stop=false fresh_denied=true replay=ok resync_refused=true still_denied=true unrelated_permitted=true J237 window=before_ack lose_unsynced=false marker_after_stop=true fresh_denied=true replay=ok resync_refused=true still_denied=true unrelated_permitted=true test result: ok. 6 passed; 0 failed ``` Server side (`server.log`, 14 passed): receipt lost before record gives `confirmed=false retired=false ordinary_refused=true cleanup_only=true`, then after a restart `confirmed=true retired=true`. `pending_revocation_survives_database_reopen_before_acceptance` and `unconfirmed_cleanup_replays_after_restart_without_granting_remote_access` are green. That covers the delta's new window, a server crash after the `pending` answer and before delivery. The INSERT and the denial share one transaction. I found no path that grants access again. ## Findings **P2-1: removing an offline host now loses the leftover-install command (#235 regression).** - When `pending`, `publish` overwrites the outcome message (`remote_removal.rs:342-347`), and the bootstrap row it needs is deleted in the same transaction (`remote_removal.rs:305`). - The UI's pending branch returns before the `manual` "Copy command" toast (`host-removal.on-demand.ts:35-38` vs `:40-48`). - The human CLI prints only `message`; `data` is `{}` unless `--machine` (`host_remove.rs:57-71`). - For a UUID owner, every removal is `pending`, so the command is lost on every removal. On origin/dev an unreachable host returns "…infraternal is still installed on it. Remove it there as root with: …" and the copy action. - Reproduced: all nine responses carry `manual="/usr/local/bin/infraternal --bootstrap-uninstall <id>"` that nobody shows (`removal.log`). - A dead host never reconnects, so the person is told to wait for a reconnect that won't happen. The doc's `retired-unreachable` step says to use "its ownership-checked uninstall command" but not where to find it. - Fix: keep `outcome()`'s sentence and `manual` together with "revoked, cleanup pending", and keep the Copy-command toast on the pending branch. The message must also be right for an online host, where `request_uninstall` returned `requested`. Today it says "must reconnect" there too. **P3-1: the WebSocket transport still holds handshake permits through cleanup queueing and delivery** (`remote_agent_http.rs:137-147`). - The per-source (4) and global (64) HTTPS agent-link permits stay held through the FIFO wait and the up-to-300 s delivery. TCP and QUIC release them (`remote_accept.rs:122-124`, `:247-248`). - J196's report (79999) and the doc say ordinary handshake capacity is returned. - Effect: four pending hosts behind one NAT address block that address's ordinary HTTPS reconnects for that time, and 24 cleanup links take 24 of the 64 global permits. - The per-host claim now does the job the comment gives for keeping them (bounding detached actors), so they can be released as on the other transports. - Found by inspection, not reproduced. **P3-2: the P1 fix is inside a merge commit.** `aa2fc6d50` is the merge of dev `eb0684cbf`. Its remerge diff holds the whole behaviour change: `remote_authority.rs`, `remote_removal.rs`, `host_remove.rs`, `host-removal.on-demand.ts` and the tests. `git log --no-merges`, bisect and review-by-commit don't show it. CLAUDE.md asks for atomic commits that don't mix changes. Split the fix out of the merge when the branch is next rebuilt. **P3-3: `retired-unreachable` rows have no way out.** They have no actions (`contracts/remote-host-inventory.md`) and stay listed for good. That matches the doc, but nobody can clear the row after doing the manual uninstall. This is a product call for kayg, not a blocker. ## Gate failures J196 reported: none caused by this branch - **`test` row, `real_binary_rootless_revocation_after_restart_stops_every_owned_terminal` `AssertionError: (385, 0)` (80070): not this branch.** The branch doesn't touch `daemon_writer.rs`, `scripts/agent_fixture.py` or `scripts/test-rootless-revoke-live.py` (`git diff --stat ce1f10932 810a90aab`). - J196's deterministic probe on clean origin/dev `ce1f10932` (`~/j196-evidence/r5/fence-probe.log`) fills the real writer's bulk and control queues and gets `Pong(stream=0)` then `Output(stream=1, seq=385)`. The unchanged fixture then fails with the identical `AssertionError: (385, 0)`. - Cause: `fence()` (`agent_fixture.py:467-478`) treats Pong as a drain barrier for terminal output, but the writer sends control frames first by design. `detach()` → `select_terminal()` (`:694-700`) then resets the cursor while the old terminal's output is still queued. - It's a latent dev fixture race. Fix it in the fixture (drain the detached terminal to its close before resetting), in its own issue. - One caveat: the branch's off-loop fsync changes timing on the open path, which could make the race more frequent. none of the 120 isolated runs and 6 full-suite runs on baseline reproduced it, and nobody has looped the branch head. - **80100/80116** are J196's baseline comparisons (100/100 isolated and 5/5 suite runs green on x86 dev). They're consistent with a rare race, and the probe above shows how it happens. - **`shape lints off (D351)`: upstream.** The `#[allow(clippy::too_many_lines)]` at `crates/plugins/mesh/tests/gotatun_persistence.rs:147` is on dev `ce1f10932` and not on current origin/dev `19737d86d`. Merging current dev clears it. The branch's own leftover allow in `remote_removal.rs` from the `aa2fc6d50` conflict is gone at `810a90aab`. **Verdict: REQUEST CHANGES**: fix P2-1 (keep the manual uninstall command and an accurate message on `pending`). The P3s can ride along. The security changes are approved as they stand.
kayg commented 2026-11-05 17:05:28 +00:00
Author
Owner
Copy link

J196 starts J249's follow-up (#80211) on agent-fsync-offloop-j196-r6, using the existing isolated J196 worktree. Starting tree/head: 810a90aab36e2c3d75e0c1cb950a0355706353bc; current origin/dev: 19737d86d9eac6d556d769ee6e4bb7457a4353a9.

Plan and estimate: 180 minutes, including queue time. First rebuild history so J237's removal-admission fix is a normal atomic commit after the dev integration, with a tree-equivalence check against the reviewed head. Merge current dev. Then add failing tests for the pending manual-command response/bootstrap retention, the UI Copy command action, human CLI output, and WebSocket cleanup handshake release. Implement those fixes in separate logical commits with doc comments and push each. Run focused tests, production-browser evidence, quick, and fast through build-slot, polling only in the foreground. Record exact gate output and clean all targets/runners.

P3-3 is explicitly deferred to kayg; no behavior for clearing retired-unreachable hosts will be built. The existing daemon fixture race is assigned to the flake job; I will report any recurrence and reference its existing baseline proof without changing its scheduler or fixture here. No merge to dev or deployment.

No targets or runners are active for J196 at start. Evidence for this round will be under ~/j196-evidence/r6/ on the chosen build VM and /home/kayg/.cache/codex-scratch/j196/r6/ on the control host.

J196 starts J249's follow-up (#80211) on **`agent-fsync-offloop-j196-r6`**, using the existing isolated J196 worktree. Starting tree/head: `810a90aab36e2c3d75e0c1cb950a0355706353bc`; current origin/dev: `19737d86d9eac6d556d769ee6e4bb7457a4353a9`. Plan and estimate: **180 minutes**, including queue time. First rebuild history so J237's removal-admission fix is a normal atomic commit after the dev integration, with a tree-equivalence check against the reviewed head. Merge current dev. Then add failing tests for the pending manual-command response/bootstrap retention, the UI Copy command action, human CLI output, and WebSocket cleanup handshake release. Implement those fixes in separate logical commits with doc comments and push each. Run focused tests, production-browser evidence, quick, and fast through build-slot, polling only in the foreground. Record exact gate output and clean all targets/runners. P3-3 is explicitly deferred to kayg; no behavior for clearing retired-unreachable hosts will be built. The existing daemon fixture race is assigned to the flake job; I will report any recurrence and reference its existing baseline proof without changing its scheduler or fixture here. No merge to dev or deployment. No targets or runners are active for J196 at start. Evidence for this round will be under `~/j196-evidence/r6/` on the chosen build VM and `/home/kayg/.cache/codex-scratch/j196/r6/` on the control host.
kayg commented 2026-11-05 17:15:22 +00:00
Author
Owner
Copy link

J196 r6 progress: fresh branch agent-fsync-offloop-j196-r6, current head 3a7446c8f. Rebuilt history: pure integration 1c18779ff; P1 removal-admission fix is now normal one-parent commit 0dcbbbd99. Recreated descendants preserve the exact tree of reviewed 810a90aab (tree equivalence recorded in r6/history-map.log), then merged origin/dev 19737d86d at 69c06f4e3.

Failing-first tests are written for server message/bootstrap retention, human CLI rendering, the production UI Copy command action, and actual WebSocket proof/Hello/cleanup delivery permit ownership. WIP pushed for recovery; production behavior not changed yet (CLI adapter extracted unchanged for its rendering test). Both VM queues checked: compute01 has four busy slots and five queued; dev-02 two busy and four queued, so compute01 is the shorter queue per slot. Red run will use only compute01:~/worktrees/j196/target; logs ~/j196-evidence/r6/. P3-3 remains kayg's product decision, no implementation.

J196 r6 progress: fresh branch `agent-fsync-offloop-j196-r6`, current head `3a7446c8f`. Rebuilt history: pure integration `1c18779ff`; P1 removal-admission fix is now normal one-parent commit `0dcbbbd99`. Recreated descendants preserve the exact tree of reviewed `810a90aab` (tree equivalence recorded in `r6/history-map.log`), then merged origin/dev `19737d86d` at `69c06f4e3`. Failing-first tests are written for server message/bootstrap retention, human CLI rendering, the production UI Copy command action, and actual WebSocket proof/Hello/cleanup delivery permit ownership. WIP pushed for recovery; production behavior not changed yet (CLI adapter extracted unchanged for its rendering test). Both VM queues checked: compute01 has four busy slots and five queued; dev-02 two busy and four queued, so compute01 is the shorter queue per slot. Red run will use only `compute01:~/worktrees/j196/target`; logs `~/j196-evidence/r6/`. P3-3 remains kayg's product decision, no implementation.
history-map.log
837 B
kayg commented 2026-11-05 17:40:19 +00:00
Author
Owner
Copy link

J196 r6 checkpoint: pushed WIP 29845f47e on agent-fsync-offloop-j196-r6. The rebuilt history is now finalized: pure integration f68ea9eaf, standalone one-parent P1 fix f43038d23, and descendants preserve the reviewed 810 tree exactly before the new round. This supersedes the intermediate history hashes in #80220; the pure-integration fixtures now share the receipt watch they await, and are formatted.

Expected red failures are recorded on compute01:

  • ~/j196-evidence/r6/red.log: server pending message hides manual command (101); human CLI prints pending status without structured manual command (101).
  • ~/j196-evidence/r6/retention-red.log: pending bootstrap count is 0, expected 1 (101).
  • ~/j196-evidence/r6/wording-red.log: message incorrectly requires reconnect (101).

The WebSocket test first hit its fixture's exact seven-day certificate validity constraint. Correcting that fixture and rerunning before changing handshake permit ownership; this does not count as the requested red proof. The production browser red stage is running in the foreground-polled build-slot run. Server fix drafted locally; no UI or handshake production fix yet. Exactly one target: compute01 ~/worktrees/j196/target. Gate still outstanding; no success claim.

J196 r6 checkpoint: pushed WIP `29845f47e` on `agent-fsync-offloop-j196-r6`. The rebuilt history is now finalized: pure integration `f68ea9eaf`, standalone one-parent P1 fix `f43038d23`, and descendants preserve the reviewed 810 tree exactly before the new round. This supersedes the intermediate history hashes in #80220; the pure-integration fixtures now share the receipt watch they await, and are formatted. Expected red failures are recorded on compute01: - `~/j196-evidence/r6/red.log`: server pending message hides manual command (101); human CLI prints pending status without structured manual command (101). - `~/j196-evidence/r6/retention-red.log`: pending bootstrap count is 0, expected 1 (101). - `~/j196-evidence/r6/wording-red.log`: message incorrectly requires reconnect (101). The WebSocket test first hit its fixture's exact seven-day certificate validity constraint. Correcting that fixture and rerunning before changing handshake permit ownership; this does not count as the requested red proof. The production browser red stage is running in the foreground-polled build-slot run. Server fix drafted locally; no UI or handshake production fix yet. Exactly one target: compute01 `~/worktrees/j196/target`. Gate still outstanding; no success claim.
history-map.log
837 B
retention-red.log
738 B
wording-red.log
709 B
kayg commented 2026-11-05 18:01:09 +00:00
Author
Owner
Copy link

J196 r6: all requested red proofs now reached their intended assertions before the corresponding production changes. compute01:~/j196-evidence/r6/red-retry.log records:

cleanup retained global handshake slot: left 0, right 1
WS_RED_RETRY_EXIT=101

Both 1440px desktop and 393px touch/mobile production-build tests displayed the pending toast, then failed because Copy command was absent (UI_RED_RETRY_EXIT=1). Screenshots are under ~/j196-evidence/r6/browser-red-retry/pending-command/; the tested source diff is ~/j196-evidence/r6/red-source-diff.log. Earlier server, retained-bootstrap, wording and human-CLI red proofs remain in red.log, retention-red.log, wording-red.log.

Implementation checkpoint pushed: 3ecb516c2ceb4ad508f118045a50b38ea626418f, branch agent-fsync-offloop-j196-r6. Keep the pending outcome/manual command together, retain removal bootstrap coordinates until the exact durable receipt, retain the UI copy action while leaving the pending host listed, and render a structured manual command in human CLI output when absent from the status sentence. Other revocation bootstrap restoration contracts stay unchanged. Validated WebSocket proof + Hello now releases source/global handshake slots before entering the bounded per-host cleanup lane.

Green batch queued on compute01 after checking both VMs. It checks the rebuilt pure integration (f68ea9eaf) and standalone P1 (f43038d23) removal suites, then current removal/cleanup/replay/CLI/typecheck/production-browser tests. One target only: ~/worktrees/j196/target. Logs ~/j196-evidence/r6/green.log, history-pure.log, history-p1.log. WIP commits will be split into two final logical commits after green, then latest dev (5ea05cc2e or newer), quick and fast. No gate success claimed yet; P3-3 remains untouched.

J196 r6: all requested red proofs now reached their intended assertions before the corresponding production changes. `compute01:~/j196-evidence/r6/red-retry.log` records: ```text cleanup retained global handshake slot: left 0, right 1 WS_RED_RETRY_EXIT=101 ``` Both 1440px desktop and 393px touch/mobile production-build tests displayed the pending toast, then failed because `Copy command` was absent (`UI_RED_RETRY_EXIT=1`). Screenshots are under `~/j196-evidence/r6/browser-red-retry/pending-command/`; the tested source diff is `~/j196-evidence/r6/red-source-diff.log`. Earlier server, retained-bootstrap, wording and human-CLI red proofs remain in `red.log`, `retention-red.log`, `wording-red.log`. Implementation checkpoint pushed: `3ecb516c2ceb4ad508f118045a50b38ea626418f`, branch `agent-fsync-offloop-j196-r6`. Keep the pending outcome/manual command together, retain removal bootstrap coordinates until the exact durable receipt, retain the UI copy action while leaving the pending host listed, and render a structured manual command in human CLI output when absent from the status sentence. Other revocation bootstrap restoration contracts stay unchanged. Validated WebSocket proof + Hello now releases source/global handshake slots before entering the bounded per-host cleanup lane. Green batch queued on compute01 after checking both VMs. It checks the rebuilt pure integration (`f68ea9eaf`) and standalone P1 (`f43038d23`) removal suites, then current removal/cleanup/replay/CLI/typecheck/production-browser tests. One target only: `~/worktrees/j196/target`. Logs `~/j196-evidence/r6/green.log`, `history-pure.log`, `history-p1.log`. WIP commits will be split into two final logical commits after green, then latest dev (`5ea05cc2e` or newer), quick and fast. No gate success claimed yet; P3-3 remains untouched.
red-retry.log
8.7 KiB
red-source-diff.log
19 KiB
kayg commented 2026-11-05 18:34:50 +00:00
Author
Owner
Copy link

J196 checkpoint: fresh branch agent-fsync-offloop-j196-r6, pushed head 86bd14362 (merged dev f9f8ced338). No dev merge or deployment.

The round fixes are now atomic: 79380513f preserves pending manual uninstall guidance on server/UI/human CLI and bootstrap coordinates until the matching receipt; 2c7b0be27 returns validated WebSocket handshake permits. P1 admission is the standalone one-parent f43038d234c284784422977a6c15eceac85188e1, preceded by pure integration f68ea9eaf6c9bca6100a8d3fda5eab34bf4975ac. Historical tests pass: pure integration 5/5, P1 6/6 (compute01 ~/j196-evidence/r6/history-{pure,p1}.log).

Focused current-head results before dev integration, raw compute01:~/j196-evidence/r6/green.log attached: removal, real WebSocket permit lifetime, cleanup lane, revocation replay, human CLI and typecheck all exit 0. New production browser Copy-command tests pass at 1440 and 393px, including exact clipboard text and alignment within 0.5px. Desktop/mobile screenshots attached for independent review. An existing pending-inventory test failed once at its non-retrying boundingBox assertion (375 paper); an isolated rerun is queued, no product fix inferred from it.

Dev integration preserves both upstream resolved-account grant checking and positive-authority resync, and adapts the reservation path handoff to upstream failed-launch cleanup. Next: isolated browser result, quick, then fast on the final clean head. P3-3 is untouched as requested.

J196 checkpoint: fresh branch `agent-fsync-offloop-j196-r6`, pushed head `86bd14362` (merged dev `f9f8ced338`). No dev merge or deployment. The round fixes are now atomic: `79380513f` preserves pending manual uninstall guidance on server/UI/human CLI and bootstrap coordinates until the matching receipt; `2c7b0be27` returns validated WebSocket handshake permits. P1 admission is the standalone one-parent `f43038d234c284784422977a6c15eceac85188e1`, preceded by pure integration `f68ea9eaf6c9bca6100a8d3fda5eab34bf4975ac`. Historical tests pass: pure integration 5/5, P1 6/6 (compute01 `~/j196-evidence/r6/history-{pure,p1}.log`). Focused current-head results before dev integration, raw `compute01:~/j196-evidence/r6/green.log` attached: removal, real WebSocket permit lifetime, cleanup lane, revocation replay, human CLI and typecheck all exit 0. New production browser Copy-command tests pass at 1440 and 393px, including exact clipboard text and alignment within 0.5px. Desktop/mobile screenshots attached for independent review. An existing pending-inventory test failed once at its non-retrying boundingBox assertion (375 paper); an isolated rerun is queued, no product fix inferred from it. Dev integration preserves both upstream resolved-account grant checking and positive-authority resync, and adapts the reservation path handoff to upstream failed-launch cleanup. Next: isolated browser result, quick, then fast on the final clean head. P3-3 is untouched as requested.
green.log
25 KiB
1440.png
162 KiB
393.png
142 KiB
1440.png 393.png
kayg commented 2026-11-05 18:50:59 +00:00
Author
Owner
Copy link

J196 validation checkpoint, head 86bd143625 on agent-fsync-offloop-j196-r6: the existing pending-inventory 375 paper browser test passes in isolation (attached ui-isolated.log, source on compute01 at ~/j196-evidence/r6/ui-isolated.log). Both new Copy-command browser cases already passed in green.log.

Final quick/fast wrapper is waiting in the foreground-polling loop for dev-02 build admission, currently first queued after ~15 minutes. Both VM queues were checked; neither has free capacity. Compute01 Rust target was deleted before moving the gate, so this job retains only one Rust target at dev-02:~/worktrees/j196/target. Gate source/log: dev-02:~/j196-evidence/r6/final-gate.{sh,log}. No gate pass is claimed yet. No own browser or server run remains active.

J196 validation checkpoint, head `86bd143625` on `agent-fsync-offloop-j196-r6`: the existing pending-inventory `375 paper` browser test passes in isolation (attached `ui-isolated.log`, source on compute01 at `~/j196-evidence/r6/ui-isolated.log`). Both new Copy-command browser cases already passed in `green.log`. Final quick/fast wrapper is waiting in the foreground-polling loop for dev-02 build admission, currently first queued after ~15 minutes. Both VM queues were checked; neither has free capacity. Compute01 Rust target was deleted before moving the gate, so this job retains only one Rust target at `dev-02:~/worktrees/j196/target`. Gate source/log: `dev-02:~/j196-evidence/r6/final-gate.{sh,log}`. No gate pass is claimed yet. No own browser or server run remains active.
ui-isolated.log
3.3 KiB
kayg commented 2026-11-05 19:22:44 +00:00
Author
Owner
Copy link

J196 foreground-wait checkpoint: final gate is still first in dev-02's queue after about 47 minutes; both slots remain occupied. No quick/fast result is claimed. Wrapper PID 1156566, log dev-02:~/j196-evidence/r6/final-gate.log. Compute01 also remains full, so there is no free alternate slot. Head is clean and pushed at 86bd143625c08077a775f27df8836f334328482d on agent-fsync-offloop-j196-r6. Focused results, red proofs, standalone P1 history and screenshots are in comments 80291/80346/80359. Continuing foreground polling; no additional scope or source changes.

J196 foreground-wait checkpoint: final gate is still first in dev-02's queue after about 47 minutes; both slots remain occupied. No quick/fast result is claimed. Wrapper PID `1156566`, log `dev-02:~/j196-evidence/r6/final-gate.log`. Compute01 also remains full, so there is no free alternate slot. Head is clean and pushed at `86bd143625c08077a775f27df8836f334328482d` on `agent-fsync-offloop-j196-r6`. Focused results, red proofs, standalone P1 history and screenshots are in comments 80291/80346/80359. Continuing foreground polling; no additional scope or source changes.
kayg commented 2026-11-05 19:35:10 +00:00
Author
Owner
Copy link

Final preflight found missing D345 module docs on two existing test modules edited this round (remote_agent_http.rs::tests, host_remove.rs::tests). Quick stopped before compilation; its other preflight rows passed. Added the two module docs and pushed ec5e72dd32 on agent-fsync-offloop-j196-r6. The exact diff-doc checker now passes (attached doc-green.log); the failed quick output is attached as quick-doc-red.log.

Quick then fast is requeued on dev-02, current wrapper PID 1450967, logs under ~/j196-evidence/r6/. Both host queues were checked, and compute01 is longer. No test, source behaviour or gate infrastructure changed in this correction. Continuing foreground polling.

Final preflight found missing D345 module docs on two existing test modules edited this round (`remote_agent_http.rs::tests`, `host_remove.rs::tests`). Quick stopped before compilation; its other preflight rows passed. Added the two module docs and pushed `ec5e72dd32` on `agent-fsync-offloop-j196-r6`. The exact diff-doc checker now passes (attached `doc-green.log`); the failed quick output is attached as `quick-doc-red.log`. Quick then fast is requeued on dev-02, current wrapper PID `1450967`, logs under `~/j196-evidence/r6/`. Both host queues were checked, and compute01 is longer. No test, source behaviour or gate infrastructure changed in this correction. Continuing foreground polling.
quick-doc-red.log
7.7 KiB
doc-green.log
40 B
kayg commented 2026-11-05 20:09:14 +00:00
Author
Owner
Copy link

J196 gate checkpoint: dev-02 acquired a slot at 01:29 IST for pushed head ec5e72dd32b01424cbfc0ddea6a196e990229743 on agent-fsync-offloop-j196-r6. Final formatting passed; quick has passed its preflight rows, including the corrected diff-doc check, and is now compiling/checking workspace dependencies for clippy. Logs: dev-02:~/j196-evidence/r6/final-gate.log, quick.log; process/resource snapshot gate-resources.log. One target only, dev-02:~/worktrees/j196/target. Quick then fast remain in the same build-slot wrapper, PID 1450967, foreground-polled. No final gate result claimed yet.

J196 gate checkpoint: dev-02 acquired a slot at 01:29 IST for pushed head `ec5e72dd32b01424cbfc0ddea6a196e990229743` on `agent-fsync-offloop-j196-r6`. Final formatting passed; quick has passed its preflight rows, including the corrected diff-doc check, and is now compiling/checking workspace dependencies for clippy. Logs: `dev-02:~/j196-evidence/r6/final-gate.log`, `quick.log`; process/resource snapshot `gate-resources.log`. One target only, `dev-02:~/worktrees/j196/target`. Quick then fast remain in the same build-slot wrapper, PID `1450967`, foreground-polled. No final gate result claimed yet.
kayg commented 2026-11-05 20:20:20 +00:00
Author
Owner
Copy link

Quick passed all preflight rows and reached workspace clippy, which found a merge-integration type mismatch in the newly upstream launch_detached test helper: it passed Option<Uuid> to the reservation path requiring Option<(Uuid, PersonGuard)>. Attached quick-merge-red.log is the compile-failure evidence (the actual runtime regression tests could not run on that head).

Pushed 66f8144c4 on agent-fsync-offloop-j196-r6: model the fixture's trusted server grant and acquire the real person launch fence, then pass it to the production reservation API. Production code is unchanged by this fixture correction. The helper's documentation explains the grant and transfer. Quick then fast is requeued on dev-02 with the same target (~/worktrees/j196/target), reusing the compiled/check artifacts. Foreground polling continues; no final gate pass claimed.

Quick passed all preflight rows and reached workspace clippy, which found a merge-integration type mismatch in the newly upstream `launch_detached` test helper: it passed `Option<Uuid>` to the reservation path requiring `Option<(Uuid, PersonGuard)>`. Attached `quick-merge-red.log` is the compile-failure evidence (the actual runtime regression tests could not run on that head). Pushed `66f8144c4` on `agent-fsync-offloop-j196-r6`: model the fixture's trusted server grant and acquire the real person launch fence, then pass it to the production reservation API. Production code is unchanged by this fixture correction. The helper's documentation explains the grant and transfer. Quick then fast is requeued on dev-02 with the same target (`~/worktrees/j196/target`), reusing the compiled/check artifacts. Foreground polling continues; no final gate pass claimed.
quick-merge-red.log
24 KiB
kayg commented 2026-11-05 20:49:57 +00:00
Author
Owner
Copy link

J196 foreground checkpoint: 66f8144c4d35d6b2996fd4a1b8ef9192f03c0d65 remains clean/pushed on agent-fsync-offloop-j196-r6. Current quick/fast rerun is waiting for dev-02 admission, last observed second in queue; actual wrapper PID 1566870 (parent shell 1566813), log ~/j196-evidence/r6/final-gate.log. Both VM queues remain busy. The sole Rust target remains dev-02:~/worktrees/j196/target; compute01's target is deleted.

All requested functional changes and historical atomic P1 split are pushed. Focused green/red evidence and production screenshots remain attached in earlier comments. Last gate attempt passed every preflight row and exposed the merged launch-test helper mismatch, now corrected as described in80438. No final fast result is claimed; continuing foreground polling. No merge/deploy, no P3-3 implementation.

J196 foreground checkpoint: `66f8144c4d35d6b2996fd4a1b8ef9192f03c0d65` remains clean/pushed on `agent-fsync-offloop-j196-r6`. Current quick/fast rerun is waiting for dev-02 admission, last observed second in queue; actual wrapper PID `1566870` (parent shell `1566813`), log `~/j196-evidence/r6/final-gate.log`. Both VM queues remain busy. The sole Rust target remains `dev-02:~/worktrees/j196/target`; compute01's target is deleted. All requested functional changes and historical atomic P1 split are pushed. Focused green/red evidence and production screenshots remain attached in earlier comments. Last gate attempt passed every preflight row and exposed the merged launch-test helper mismatch, now corrected as described in80438. No final fast result is claimed; continuing foreground polling. No merge/deploy, no P3-3 implementation.
kayg commented 2026-11-05 21:20:49 +00:00
Author
Owner
Copy link

J196 foreground checkpoint: the corrected head 66f8144c4d35d6b2996fd4a1b8ef9192f03c0d65 remains pushed/clean. This quick/fast attempt has now waited about one hour for dev-02 admission and is first queued; both occupied slots remain active. Compute01 has also remained full at the repeated capacity checks. No final gate result is claimed.

Actual wrapper PID 1566870, parent 1566813, log dev-02:~/j196-evidence/r6/final-gate.log; one target only at dev-02:~/worktrees/j196/target. All functional fixes, the historical P1 split, valid red/focused green evidence and production screenshots remain saved/pushed/attached in the preceding comments. Continuing foreground polling, with no new scope or source changes.

J196 foreground checkpoint: the corrected head `66f8144c4d35d6b2996fd4a1b8ef9192f03c0d65` remains pushed/clean. This quick/fast attempt has now waited about one hour for dev-02 admission and is first queued; both occupied slots remain active. Compute01 has also remained full at the repeated capacity checks. No final gate result is claimed. Actual wrapper PID `1566870`, parent `1566813`, log `dev-02:~/j196-evidence/r6/final-gate.log`; one target only at `dev-02:~/worktrees/j196/target`. All functional fixes, the historical P1 split, valid red/focused green evidence and production screenshots remain saved/pushed/attached in the preceding comments. Continuing foreground polling, with no new scope or source changes.
kayg commented 2026-11-05 21:43:49 +00:00
Author
Owner
Copy link

J216b: fixture fix for the Pong/terminal-output race (J196 80070, J249 80211). Branch fixture-fence-j216g, head 574003cc6, from origin/dev f9f8ced33. One commit; nothing merged.

Cause (confirmed with J196's probe method): detach() sends Close on the terminal stream and relies on fence()'s Pong, but the writer sends control replies ahead of queued terminal output by design (biased control channel, bulk FIFO). On a client Close the daemon removes the stream without any reply, so no frame is ordered behind that stream's last output. select_terminal() then resets the cursor and reads the leftover against the new terminal: (385, 0).

Fix (fixture only, no product change): select_terminal keeps the old cursor and credit window; process() drops terminal Output frames that continue that cursor within that credit, until the first stream-1 frame that does not, which starts the new attachment. This is exact because the bulk queue is FIFO: all leftover output precedes the new attachment's first frame. One documented edge: an old attachment that had read nothing (cursor 0) can't be told apart from a fresh replay at 0, so no window is opened for it.

Failing-first probe daemon::readiness_tests::fixture_drops_output_left_over_from_the_previous_terminal: the real writer is fed Pong, the leftover Output(385) and the new attachment's Output(0); the real fixture's fence() → select_terminal() → process() must end at cursor 12 with only the new bytes. origin/dev fixture 0/5 (AssertionError: (385, 0)) → branch 5/5. Full agent_daemon suite ×5 beside two looping agent suites (load 22–29): origin/dev fixture 4/5, branch 5/5. The one origin/dev failure had a different signature ("fixture terminal did not retire"), so I don't claim it.

scripts/verify quick: exit 0, all rows PASS.

scripts/verify fast on 574003cc6, dev-02: VERIFY_FAST_EXIT=1

=== /home/claude/.cache/codex-scratch/j216b/evidence/verify-fast-g.log
+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | FAIL (1)     |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | FAIL (101)   |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| privileged integration    | PASS         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| static builds             | PASS         |

None of these tests touch this branch (it changes agent_fixture.py and one agent test). oversized_debug_image… is the dev regression J198 owns. The three restart tests hit their 60 s child bound in the compile-out run while a second full gate (mine, j216e) ran on the same VM; that's my scheduling, and I won't run two gates in parallel on one VM again. up::install::j218_review::j226_failed_stop_keeps_recovery_authority failed with "Another infraternal up installation is still running", which looks like a test-isolation issue in the new up install tests; new, not chased.

J216b: fixture fix for the Pong/terminal-output race (J196 80070, J249 80211). **Branch `fixture-fence-j216g`, head `574003cc6`**, from origin/dev `f9f8ced33`. One commit; nothing merged. **Cause** (confirmed with J196's probe method): `detach()` sends Close on the terminal stream and relies on `fence()`'s Pong, but the writer sends control replies ahead of queued terminal output by design (`biased` control channel, bulk FIFO). On a client Close the daemon removes the stream without any reply, so no frame is ordered behind that stream's last output. `select_terminal()` then resets the cursor and reads the leftover against the new terminal: `(385, 0)`. **Fix (fixture only, no product change):** `select_terminal` keeps the old cursor and credit window; `process()` drops terminal Output frames that continue that cursor within that credit, until the first stream-1 frame that does not, which starts the new attachment. This is exact because the bulk queue is FIFO: all leftover output precedes the new attachment's first frame. One documented edge: an old attachment that had read nothing (cursor 0) can't be told apart from a fresh replay at 0, so no window is opened for it. **Failing-first probe** `daemon::readiness_tests::fixture_drops_output_left_over_from_the_previous_terminal`: the real `writer` is fed Pong, the leftover `Output(385)` and the new attachment's `Output(0)`; the real fixture's `fence()` → `select_terminal()` → `process()` must end at cursor 12 with only the new bytes. **origin/dev fixture 0/5 (`AssertionError: (385, 0)`) → branch 5/5.** Full `agent_daemon` suite ×5 beside two looping agent suites (load 22–29): origin/dev fixture 4/5, branch 5/5. The one origin/dev failure had a different signature ("fixture terminal did not retire"), so I don't claim it. `scripts/verify quick`: exit 0, all rows PASS. **`scripts/verify fast` on `574003cc6`, dev-02: `VERIFY_FAST_EXIT=1`** ``` === /home/claude/.cache/codex-scratch/j216b/evidence/verify-fast-g.log +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | FAIL (1) | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | FAIL (101) | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | PASS | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | static builds | PASS | ``` None of these tests touch this branch (it changes `agent_fixture.py` and one agent test). `oversized_debug_image…` is the dev regression J198 owns. The three restart tests hit their 60 s child bound in the compile-out run while a second full gate (mine, j216e) ran on the same VM; that's my scheduling, and I won't run two gates in parallel on one VM again. `up::install::j218_review::j226_failed_stop_keeps_recovery_authority` failed with "Another infraternal up installation is still running", which looks like a test-isolation issue in the new `up` install tests; new, not chased.
kayg commented 2026-11-05 21:48:45 +00:00
Author
Owner
Copy link

J196 foreground gate checkpoint: branch agent-fsync-offloop-j196-r6, pushed head 66f8144c4d35d6b2996fd4a1b8ef9192f03c0d65. The dev-02 slot was acquired at 03:03 IST. Formatting passed; quick passed clippy and completed infraternal-agent tests, and is progressing through server tests. No final gate result yet. Raw logs: dev-02:~/j196-evidence/r6/final-gate.log and quick.log. The script runs scripts/verify fast after quick succeeds. Foreground polling continues.

The gate began while the optional tip-commit folding was being considered, so head66 is kept fixed for an exact-SHA result. P3-2 is already addressed: the original admission P1 is standalone one-parent commit f43038d234c284784422977a6c15eceac85188e1, with historical tests recorded in comment80346. No latch or P3-3 product change. One Rust target only: /home/claude/worktrees/j196/target on dev-02, to be deleted after the run.

J196 foreground gate checkpoint: branch `agent-fsync-offloop-j196-r6`, pushed head `66f8144c4d35d6b2996fd4a1b8ef9192f03c0d65`. The dev-02 slot was acquired at 03:03 IST. Formatting passed; quick passed clippy and completed infraternal-agent tests, and is progressing through server tests. No final gate result yet. Raw logs: `dev-02:~/j196-evidence/r6/final-gate.log` and `quick.log`. The script runs `scripts/verify fast` after quick succeeds. Foreground polling continues. The gate began while the optional tip-commit folding was being considered, so head66 is kept fixed for an exact-SHA result. P3-2 is already addressed: the original admission P1 is standalone one-parent commit `f43038d234c284784422977a6c15eceac85188e1`, with historical tests recorded in comment80346. No latch or P3-3 product change. One Rust target only: `/home/claude/worktrees/j196/target` on dev-02, to be deleted after the run.
kayg commented 2026-11-05 21:53:08 +00:00
Author
Owner
Copy link

J196 final-head quick finished with exit1. Every preflight, workspace clippy and changed-library test row passed; binary-only tests failed at restart::recovery_tests::oversized_debug_image_does_not_limit_restart_fixtures (208 passed, 1 failed, 14 ignored).

Ran that test alone from the compiled final-head binary: same assertion, exit101. Raw isolated log attached; full run is dev-02:~/j196-evidence/r6/quick.log. This is an upstream fixture mismatch: dev commit d2acd9847 raises RUNNING_IMAGE_LIMIT to 768 MiB, while restart_recovery_tests.rs:40 still pads to 512 MiB + 1 and expects refusal at line31. Both files are identical on head66 and merged dev f9f8ced338: blobs e14dbe918e5e817d58a4b13984b0211278cfd4da (restart_binary.rs) and f82880e16e1f1158e9ba318bcb1a2142b9dc1246 (restart_recovery_tests.rs). No fix to unrelated restart behavior is included.

scripts/verify fast is queued on dev-02 through build-slot, wrapper PID1848577. Foreground polling continues. Logs: ~/j196-evidence/r6/final-fast.log, fast.log. Target remains the same sole target.

J196 final-head quick finished with exit1. Every preflight, workspace clippy and changed-library test row passed; binary-only tests failed at `restart::recovery_tests::oversized_debug_image_does_not_limit_restart_fixtures` (208 passed, 1 failed, 14 ignored). Ran that test alone from the compiled final-head binary: same assertion, exit101. Raw isolated log attached; full run is `dev-02:~/j196-evidence/r6/quick.log`. This is an upstream fixture mismatch: dev commit `d2acd9847` raises `RUNNING_IMAGE_LIMIT` to 768 MiB, while `restart_recovery_tests.rs:40` still pads to 512 MiB + 1 and expects refusal at line31. Both files are identical on head66 and merged dev `f9f8ced338`: blobs `e14dbe918e5e817d58a4b13984b0211278cfd4da` (restart_binary.rs) and `f82880e16e1f1158e9ba318bcb1a2142b9dc1246` (restart_recovery_tests.rs). No fix to unrelated restart behavior is included. `scripts/verify fast` is queued on dev-02 through build-slot, wrapper PID1848577. Foreground polling continues. Logs: `~/j196-evidence/r6/final-fast.log`, `fast.log`. Target remains the same sole target.
restart-isolated.log
1.2 KiB
kayg commented 2026-11-05 21:59:04 +00:00
Author
Owner
Copy link

J196 moved the final run to compute01 as required by the brief when slot4 became free. Cancelled only own dev-02 queued PID1848577 and deleted its 11G target. Compute01 branch is exact pushed head66; sole target will be /home/claude/worktrees/j196/target there. Wrapper PID2006125, ~/j196-evidence/r6/compute-final.log, running quick then fast; quick failure will not suppress the requested fast gate.

Queue discrepancy: build-slot --status reports compute01 slot4 free, slots1–3 occupied and four configured slots; head waiter is root-owned PID2004850 (J197 root proof), with J196 second. This remained unchanged across repeated checks, despite >300G free disk. No queue bypass or changes to shared build infrastructure. Foreground polling continues.

J196 moved the final run to compute01 as required by the brief when slot4 became free. Cancelled only own dev-02 queued PID1848577 and deleted its 11G target. Compute01 branch is exact pushed head66; sole target will be `/home/claude/worktrees/j196/target` there. Wrapper PID2006125, `~/j196-evidence/r6/compute-final.log`, running quick then fast; quick failure will not suppress the requested fast gate. Queue discrepancy: `build-slot --status` reports compute01 slot4 free, slots1–3 occupied and four configured slots; head waiter is root-owned PID2004850 (J197 root proof), with J196 second. This remained unchanged across repeated checks, despite >300G free disk. No queue bypass or changes to shared build infrastructure. Foreground polling continues.
kayg commented 2026-11-05 22:29:10 +00:00
Author
Owner
Copy link

J196 checkpoint: compute01 acquired slot4 and reran quick at exact pushed head 66f8144c4d35d6b2996fd4a1b8ef9192f03c0d65. Quick exit1: all preflights, clippy and changed-library tests passed; binary-only suite 208 passed, 1 failed, 14 ignored, with the identical upstream oversized restart fixture documented in comment80534. No other failure.

scripts/verify fast is now running in the same slot/target, foreground-polled. Logs: compute01:~/j196-evidence/r6/compute-final.log, final-quick.log, final-fast.log. Wrapper PID2006125. One Rust target only, compute01 /home/claude/worktrees/j196/target; dev-02 target deleted. The reviewed remote-feature regressions and browser greens remain in green.log and comment80346; default-feature quick does not select those server modules. No further source changes.

J196 checkpoint: compute01 acquired slot4 and reran quick at exact pushed head `66f8144c4d35d6b2996fd4a1b8ef9192f03c0d65`. Quick exit1: all preflights, clippy and changed-library tests passed; binary-only suite 208 passed, 1 failed, 14 ignored, with the identical upstream oversized restart fixture documented in comment80534. No other failure. `scripts/verify fast` is now running in the same slot/target, foreground-polled. Logs: `compute01:~/j196-evidence/r6/compute-final.log`, `final-quick.log`, `final-fast.log`. Wrapper PID2006125. One Rust target only, compute01 `/home/claude/worktrees/j196/target`; dev-02 target deleted. The reviewed remote-feature regressions and browser greens remain in `green.log` and comment80346; default-feature quick does not select those server modules. No further source changes.
kayg commented 2026-11-05 22:47:12 +00:00
Author
Owner
Copy link

J196 estimate checkpoint, 04:17 IST: reporting before continuing beyond the common brief's 2× original 180-minute estimate (round began around22:20–22:30 IST). Implementation and push are complete at head66. The long queues, doc/fixture quick corrections, upstream restart fixture failure, and required free-slot VM move consumed the estimate; no added product or infrastructure scope.

The explicitly requested scripts/verify fast is now in static builds on compute01, PID2006125. Workspace test row has the exact oversized restart fixture failure reproduced alone and traced to byte-identical dev files in comment80534. Probe release exclusion, nightly fuzz reporter, and privileged integration completed; zero privileged test cases were selected. Full gate summary and static sizes not yet available. Evidence remains ~/j196-evidence/r6/final-fast.log and compute-final.log.

Continuing only the already-authorized foreground gate, final evidence/report, and target/process cleanup. No merge, deployment, latch change, or P3-3 work. One target: compute01 /home/claude/worktrees/j196/target.

J196 estimate checkpoint, 04:17 IST: reporting before continuing beyond the common brief's 2× original 180-minute estimate (round began around22:20–22:30 IST). Implementation and push are complete at head66. The long queues, doc/fixture quick corrections, upstream restart fixture failure, and required free-slot VM move consumed the estimate; no added product or infrastructure scope. The explicitly requested `scripts/verify fast` is now in static builds on compute01, PID2006125. Workspace test row has the exact oversized restart fixture failure reproduced alone and traced to byte-identical dev files in comment80534. Probe release exclusion, nightly fuzz reporter, and privileged integration completed; zero privileged test cases were selected. Full gate summary and static sizes not yet available. Evidence remains `~/j196-evidence/r6/final-fast.log` and `compute-final.log`. Continuing only the already-authorized foreground gate, final evidence/report, and target/process cleanup. No merge, deployment, latch change, or P3-3 work. One target: compute01 `/home/claude/worktrees/j196/target`.
kayg commented 2026-11-05 23:17:40 +00:00
Author
Owner
Copy link

J196 foreground checkpoint: unchanged pushed head66. Final fast is still running on compute01 in the arm64 CLI final optimized compilation, CPU-active; x86_64 is complete, PT_INTERP absent, size27328944 bytes vs recorded gate baseline22418256, delta+4910688 (report-only). No additional failure beyond the isolated upstream oversized restart fixture. Raw artifact: compute01:~/j196-evidence/r6/final-fast.log; wrapper PID2006125, completion marker in compute-final.log. Dev-02 target/state/processes are confirmed absent. Sole compute01 target will be deleted after the gate result is preserved. No extra source scope, merge, deployment or P3-3 work.

J196 foreground checkpoint: unchanged pushed head66. Final fast is still running on compute01 in the arm64 CLI final optimized compilation, CPU-active; x86_64 is complete, PT_INTERP absent, size27328944 bytes vs recorded gate baseline22418256, delta+4910688 (report-only). No additional failure beyond the isolated upstream oversized restart fixture. Raw artifact: `compute01:~/j196-evidence/r6/final-fast.log`; wrapper PID2006125, completion marker in `compute-final.log`. Dev-02 target/state/processes are confirmed absent. Sole compute01 target will be deleted after the gate result is preserved. No extra source scope, merge, deployment or P3-3 work.
kayg commented 2026-11-05 23:22:16 +00:00
Author
Owner
Copy link

J196 / J249 review fixes complete and pushed for review. Branch agent-fsync-offloop-j196-r6, head 66f8144c4d35d6b2996fd4a1b8ef9192f03c0d65. Integrated origin/dev f9f8ced338278d924803a691f9636a9519babcb2 (newer than required19737d86d). No merge to dev, deployment, issue closure, or P3-3 implementation.

  • Pending removals preserve the physical-cleanup message and structured manual command. Browser pending responses offer the existing Copy command action; human CLI output includes the command. Bootstrap coordinates remain until the matching durable cleanup receipt. Foreign receipts cannot erase them. Wording no longer requires a reconnect for an online host. Commit 79380513f.
  • Validated WebSocket cleanup links return source/global handshake permits after proof and Hello, before cleanup queuing/delivery. Real WebSocket proof regression asserts permits are available before sending the durable receipt. Commit 2c7b0be27.
  • Rebuilt history: the admission P1 from old merge aa2fc6d50 is now standalone one-parent commit f43038d234c284784422977a6c15eceac85188e1, after pure integration f68ea9eaf6c9bca6100a8d3fda5eab34bf4975ac. Reviewed source tree was preserved. Historical pure integration5/5 and P1 admission tests6/6: compute01:~/j196-evidence/r6/history-pure.log, history-p1.log (comment80346). Dev integration and its documented fixture/doc corrections are explicit later commits.

Failing-first and focused proof: comment80291 (red.log, retention-red.log, wording-red.log, corrected real transport/UI reproducer red-retry.log). Focused greens before the final dev merge: removal8, WebSocket1, cleanup lane3, revocation replay13, human CLI1; typecheck exit0; new production browser Copy command tests2/2 at1440 and393 pixels, including exact clipboard content and alignment. Artifact compute01:~/j196-evidence/r6/green.log, comment80346. Existing 375px paper inventory test failed once at its non-retrying boundingBox check and passed alone: ui-isolated.log, comment80359. Production screenshots are attached to80346 (browser-green/pending-command/1440.png, 393.png) for independent visual review. Final-head quick also passed the changed library suites; fast passed host_remove::tests::pending_removal_prints_manual_command_for_people.

Final gate, compute01, foreground-polled: scripts/verify quick exit1; then scripts/verify fast exit1. Both fast failures below are the same upstream fixture, restart::recovery_tests::oversized_debug_image_does_not_limit_restart_fixtures; no other failing test was found. It fails alone, too (restart-isolated.log, attached in80534). Dev commit d2acd9847 raised the running-image limit to768MiB, but this fixture still pads to512MiB+1 and expects refusal. Both relevant files are byte-identical to merged dev; exact blob hashes and isolated proof are in80534. The acknowledged daemon (385,0) fixture race was not this run's failure. No unrelated restart or gate change was made.

Verbatim fast gate summary:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| migration names           | PASS         |
| migration tooling tests   | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | FAIL (1)     |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| Traefik component manifest | PASS         |
| release component publication | PASS         |
| test                      | FAIL (101)   |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| privileged integration    | PASS         |
| probe release exclusion   | PASS         |
| nightly fuzz reporter     | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+

Verbatim static size/linkage measurements (deltas use the gate's recorded baselines, not an attribution to this patch alone):

PT_INTERP: absent; size: 27328944 bytes; baseline 22418256; delta +4910688 bytes (report only, D243 amended 2026-11-01)
PT_INTERP: absent; size: 24633232 bytes; baseline 20453280; delta +4179952 bytes (report only, D243 amended 2026-11-01)
SKIP key memory: aarch64-unknown-linux-musl (host x86_64; native execution required)

Native x86 key-memory check passed; arm64 native execution was skipped on x86. Privileged integration's planted escapes were contained and0 test runs were selected. Because fast exits on the failed Rust gate, later frontend gate stages did not run; focused production browser/typecheck evidence is recorded above.

Raw final artifacts attached: final-fast.log, final-quick.log, compute-final.log. Originals remain at compute01:~/j196-evidence/r6/. The gate wrapper exited. Deleted both VMs' J196 targets and test state; compute01 worktree-process sweep returned [] (cleanup.log attached), dev-02 gate PIDs are absent. Evidence and worktrees remain; source worktree is clean and local/remote head match.

J196 / J249 review fixes complete and pushed for review. Branch **`agent-fsync-offloop-j196-r6`**, head **`66f8144c4d35d6b2996fd4a1b8ef9192f03c0d65`**. Integrated origin/dev `f9f8ced338278d924803a691f9636a9519babcb2` (newer than required19737d86d). No merge to dev, deployment, issue closure, or P3-3 implementation. - Pending removals preserve the physical-cleanup message and structured `manual` command. Browser pending responses offer the existing Copy command action; human CLI output includes the command. Bootstrap coordinates remain until the matching durable cleanup receipt. Foreign receipts cannot erase them. Wording no longer requires a reconnect for an online host. Commit `79380513f`. - Validated WebSocket cleanup links return source/global handshake permits after proof and Hello, before cleanup queuing/delivery. Real WebSocket proof regression asserts permits are available before sending the durable receipt. Commit `2c7b0be27`. - Rebuilt history: the admission P1 from old merge aa2fc6d50 is now standalone one-parent commit **`f43038d234c284784422977a6c15eceac85188e1`**, after pure integration `f68ea9eaf6c9bca6100a8d3fda5eab34bf4975ac`. Reviewed source tree was preserved. Historical pure integration5/5 and P1 admission tests6/6: `compute01:~/j196-evidence/r6/history-pure.log`, `history-p1.log` (comment80346). Dev integration and its documented fixture/doc corrections are explicit later commits. Failing-first and focused proof: comment80291 (`red.log`, `retention-red.log`, `wording-red.log`, corrected real transport/UI reproducer `red-retry.log`). Focused greens before the final dev merge: removal8, WebSocket1, cleanup lane3, revocation replay13, human CLI1; typecheck exit0; new production browser Copy command tests2/2 at1440 and393 pixels, including exact clipboard content and alignment. Artifact `compute01:~/j196-evidence/r6/green.log`, comment80346. Existing 375px paper inventory test failed once at its non-retrying boundingBox check and passed alone: `ui-isolated.log`, comment80359. Production screenshots are attached to80346 (`browser-green/pending-command/1440.png`, `393.png`) for independent visual review. Final-head quick also passed the changed library suites; fast passed `host_remove::tests::pending_removal_prints_manual_command_for_people`. Final gate, compute01, foreground-polled: `scripts/verify quick` **exit1**; then **`scripts/verify fast` exit1**. Both fast failures below are the same upstream fixture, `restart::recovery_tests::oversized_debug_image_does_not_limit_restart_fixtures`; no other failing test was found. It fails alone, too (`restart-isolated.log`, attached in80534). Dev commit d2acd9847 raised the running-image limit to768MiB, but this fixture still pads to512MiB+1 and expects refusal. Both relevant files are byte-identical to merged dev; exact blob hashes and isolated proof are in80534. The acknowledged daemon (385,0) fixture race was not this run's failure. No unrelated restart or gate change was made. Verbatim fast gate summary: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | migration names | PASS | | migration tooling tests | PASS | | VT native fmt | PASS | | CLI plugin compile-out | FAIL (1) | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | Traefik component manifest | PASS | | release component publication | PASS | | test | FAIL (101) | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | privileged integration | PASS | | probe release exclusion | PASS | | nightly fuzz reporter | PASS | | static builds | PASS | +---------------------------+--------------+ ``` Verbatim static size/linkage measurements (deltas use the gate's recorded baselines, not an attribution to this patch alone): ```text PT_INTERP: absent; size: 27328944 bytes; baseline 22418256; delta +4910688 bytes (report only, D243 amended 2026-11-01) PT_INTERP: absent; size: 24633232 bytes; baseline 20453280; delta +4179952 bytes (report only, D243 amended 2026-11-01) SKIP key memory: aarch64-unknown-linux-musl (host x86_64; native execution required) ``` Native x86 key-memory check passed; arm64 native execution was skipped on x86. Privileged integration's planted escapes were contained and0 test runs were selected. Because fast exits on the failed Rust gate, later frontend gate stages did not run; focused production browser/typecheck evidence is recorded above. Raw final artifacts attached: `final-fast.log`, `final-quick.log`, `compute-final.log`. Originals remain at `compute01:~/j196-evidence/r6/`. The gate wrapper exited. Deleted both VMs' J196 targets and test state; compute01 worktree-process sweep returned `[]` (`cleanup.log` attached), dev-02 gate PIDs are absent. Evidence and worktrees remain; source worktree is clean and local/remote head match.
final-fast.log
478 KiB
final-quick.log
151 KiB
compute-final.log
188 B
cleanup.log
74 B
kayg commented 2026-11-05 23:48:52 +00:00
Author
Owner
Copy link

J269 review of round 6: REQUEST CHANGES (1×P2, 2×P3). Every earlier finding is fixed and has a test that fails without the fix. Security items 3–5 pass. The new P2 comes from keeping the bootstrap row: a pending removal still counts as a live host for the duplicate-address and stuck-install checks.

Target agent-fsync-offloop-j196-r6 at 66f8144c4d35d6b2996fd4a1b8ef9192f03c0d65. Review only: nothing pushed, merged or closed, and no verify fast/quick per the brief. Scratch merge with origin/dev 9b0c731e8 was 92ed29f00 (clean, local to compute01, now deleted). Evidence is in claude@infraternal-dev-01-compute01:~/j269-evidence/: run.sh, run.log, green-*.log, red-retention.log, red-ws.log, repro.rs.

Earlier findings: all fixed, red/green confirmed

My own red/green, not just the author's logs:

Finding Test Branch + dev Fix reverted
J249 P2-1, keep the bootstrap row pending_removal_retains_bootstrap_until_durable_receipt ok (green-removal.log) FAILED: "pending cleanup discarded its bootstrap coordinates" (red-retention.log; the delete made unconditional again, as at r5)
J249 P3-1, WebSocket permits websocket_cleanup_returns_handshake_slots_before_receipt ok (green-ws.log) FAILED: "cleanup retained global handshake slot" (red-ws.log; drop changed to a binding held through cleanup)
P2-1 wording/manual reviewed_removal_revokes_offline_host_and_cannot_replay, pending_removal_does_not_claim_a_reconnect_is_required, CLI pending_removal_prints_manual_command_for_people ok author's red.log:399, wording-red.log, red.log:601 show them failing first
J249 P3-2, P1 inside a merge f43038d23 has one parent, f68ea9eaf. Its diff is the admission fix alone (5 files) n/a n/a

Also green on branch + dev: remote_removal_tests 8/8 for the branch's own tests, cleanup_delivery_ 3/3 (green-lane.log), login_revocation 13/13 (green-replay.log), and host_remove:: (green-cli.log).

Gate failure: dev fixed it

restart::recovery_tests::oversized_debug_image_does_not_limit_restart_fixtures ... ok on branch + dev (green-cli.log). It fails on the branch alone because of a dev fixture bug that 9b0c731e8 fixes. Both rows that failed in the author's fast (CLI plugin compile-out and test) are this one fixture (~/j196-evidence/r6/final-fast.log:314,449,992,1129), so the merged gate should be green on those rows.

Security checks

  • 3, the uninstall command can't be pointed at another install: passes. manual_uninstall (remote_removal.rs:471) builds the command from entry.host.simple() (a server-made UUID) and a fixed path picked by mode. Nothing from the person or the host (address, hostname, OS) gets into it. --bootstrap-uninstall <host> refuses unless that account's .bootstrap-owner equals the host id (plugins/remote/src/uninstall.rs:146), and the branch doesn't touch that code. The UI copies result.manual exactly as the server sent it, and only after checking result.host_id === id.
  • 4, keeping the bootstrap row doesn't leak secrets: passes. remote_host_bootstrap holds only address, trust_lookup (a pin lookup key) and os (migrations 0019/0025), with no tokens or keys. Only the owner's own inventory and removal responses read it. Deleting it on the receipt needs remote_host_removals plus a confirmed receipt from the same person, and the test shows a receipt from another person can't delete it.
  • 5, releasing the handshake slots can't be abused to use them all up: passes. The permits are dropped only after proof authentication and the protocol Hello, so an unauthenticated client still holds them for the whole handshake. After that, cleanup links go through CleanupSlots: one claim per host, 8 active and 16 waiting, and a second link for the same host gets WouldBlock (cleanup_delivery_deduplicates_hosts). Each slot needs a valid, unrevoked certificate for a host whose cleanup is pending. Ordinary WebSocket links now drop the permits a few lines early as well, unlike TCP/QUIC's release_cleanup_handshake, which keeps them for ordinary links. That's harmless: RemoteLink::websocket passes no Admission, so it can't be primed, and ordinary admit_observed only takes a lock and does a try_send, with the map capped at 10 000.

Findings

P2-1: a pending removal still blocks re-adding its address (and its SSH key), forever for a dead host.

  • For a UUID owner, commit_revocation_intent leaves remote_hosts.revoked_at NULL until the receipt (remote_authority.rs:258-271). Round 6 now keeps the remote_host_bootstrap row as well (remote_removal.rs:306-313).
  • Every duplicate check only looks at h.revoked_at IS NULL and the bootstrap row, so all of them treat the removed host as still added:
    • existing_host / ensure_not_already_added (bootstrap_remote.rs:175) → "This host is already added. Open its terminal from Console." with the removed host's id.
    • issue_invitation's occupied (remote_http.rs:452) → 409.
    • check_stuck_install (bootstrap_remote.rs:676) and reissue_stuck_install's registered (:738) → ours=true. A journalled reinstall then takes Replace, and issue_authorized_in (which checks only revoked_at IS NULL) mints a join token for the removed host. Access isn't regained: admit and renewal refuse while remote_login_cleanup.confirmed=0, and a redeemed cert only gets cleanup-only admission. But setup reinstalls a host the person just removed.
  • Reproducer (repro.rs, which runs the three production queries word for word after a real finish_host):
r6 (branch+dev):     J269 pending removal: already_added_host=true issue_occupied=true stuck_install_ours=true ...   -> FAILED (green-removal.log)
r5 (delete restored): J269 pending removal: already_added_host=false issue_occupied=false stuck_install_ours=false ... -> ok (red-retention.log)
  • Scenario: a machine dies and is reinstalled at the same address. The owner removes it, it stays retired-unreachable because no receipt ever comes, and adding the address again is refused for good. The row it points at has no actions (J249 P3-3). Before round 6 the add worked.
  • Fix: have these checks leave out hosts with a remote_host_removals row, as remote_removal.rs:319 already does for the pin-sharing query. Add a test that re-adding the address after a pending removal is accepted.
  • Defence in depth: issue_authorized_in (enrollment.rs:278) and redeem (:404) should also refuse while remote_login_cleanup.confirmed=0, matching admit, renewal and bootstrap_host.

P3-1: retained rows still reserve machine names. wanted_name (plugins/mesh/src/store.rs:283) reads every bootstrap row. A pending-removed host's typed name therefore stays in records, and a new host reporting that hostname gets a suffix. Leave out removal rows there too.

P3-2: the pending sentence contradicts itself. "host.example: revoked, cleanup pending. host.example was removed here, but infraternal is still installed on it…" says "removed" while the row stays listed as pending, and repeats the name. Drop the second subject ("revoked, cleanup pending. infraternal is still installed on it. Remove it there as root with: …"). The UI toast already reads that way.

Verdict: REQUEST CHANGES. Fix P2-1 with a regression test. The P3s can ride along. Every earlier finding is fixed, and the security items pass.

J269 review of round 6: **REQUEST CHANGES** (1×P2, 2×P3). Every earlier finding is fixed and has a test that fails without the fix. Security items 3–5 pass. The new P2 comes from keeping the bootstrap row: a pending removal still counts as a live host for the duplicate-address and stuck-install checks. Target `agent-fsync-offloop-j196-r6` at `66f8144c4d35d6b2996fd4a1b8ef9192f03c0d65`. Review only: nothing pushed, merged or closed, and no `verify fast`/`quick` per the brief. Scratch merge with origin/dev `9b0c731e8` was `92ed29f00` (clean, local to compute01, now deleted). Evidence is in `claude@infraternal-dev-01-compute01:~/j269-evidence/`: `run.sh`, `run.log`, `green-*.log`, `red-retention.log`, `red-ws.log`, `repro.rs`. ## Earlier findings: all fixed, red/green confirmed My own red/green, not just the author's logs: | Finding | Test | Branch + dev | Fix reverted | |---|---|---|---| | J249 P2-1, keep the bootstrap row | `pending_removal_retains_bootstrap_until_durable_receipt` | ok (`green-removal.log`) | FAILED: "pending cleanup discarded its bootstrap coordinates" (`red-retention.log`; the delete made unconditional again, as at r5) | | J249 P3-1, WebSocket permits | `websocket_cleanup_returns_handshake_slots_before_receipt` | ok (`green-ws.log`) | FAILED: "cleanup retained global handshake slot" (`red-ws.log`; `drop` changed to a binding held through cleanup) | | P2-1 wording/manual | `reviewed_removal_revokes_offline_host_and_cannot_replay`, `pending_removal_does_not_claim_a_reconnect_is_required`, CLI `pending_removal_prints_manual_command_for_people` | ok | author's `red.log:399`, `wording-red.log`, `red.log:601` show them failing first | | J249 P3-2, P1 inside a merge | `f43038d23` has one parent, `f68ea9eaf`. Its diff is the admission fix alone (5 files) | n/a | n/a | Also green on branch + dev: `remote_removal_tests` 8/8 for the branch's own tests, `cleanup_delivery_` 3/3 (`green-lane.log`), `login_revocation` 13/13 (`green-replay.log`), and `host_remove::` (`green-cli.log`). ## Gate failure: dev fixed it `restart::recovery_tests::oversized_debug_image_does_not_limit_restart_fixtures ... ok` on branch + dev (`green-cli.log`). It fails on the branch alone because of a dev fixture bug that `9b0c731e8` fixes. Both rows that failed in the author's fast (`CLI plugin compile-out` and `test`) are this one fixture (`~/j196-evidence/r6/final-fast.log:314,449,992,1129`), so the merged gate should be green on those rows. ## Security checks - **3, the uninstall command can't be pointed at another install: passes.** `manual_uninstall` (`remote_removal.rs:471`) builds the command from `entry.host.simple()` (a server-made UUID) and a fixed path picked by `mode`. Nothing from the person or the host (address, hostname, OS) gets into it. `--bootstrap-uninstall <host>` refuses unless that account's `.bootstrap-owner` equals the host id (`plugins/remote/src/uninstall.rs:146`), and the branch doesn't touch that code. The UI copies `result.manual` exactly as the server sent it, and only after checking `result.host_id === id`. - **4, keeping the bootstrap row doesn't leak secrets: passes.** `remote_host_bootstrap` holds only `address`, `trust_lookup` (a pin lookup key) and `os` (migrations 0019/0025), with no tokens or keys. Only the owner's own inventory and removal responses read it. Deleting it on the receipt needs `remote_host_removals` plus a confirmed receipt from the same person, and the test shows a receipt from another person can't delete it. - **5, releasing the handshake slots can't be abused to use them all up: passes.** The permits are dropped only after proof authentication and the protocol Hello, so an unauthenticated client still holds them for the whole handshake. After that, cleanup links go through `CleanupSlots`: one claim per host, 8 active and 16 waiting, and a second link for the same host gets `WouldBlock` (`cleanup_delivery_deduplicates_hosts`). Each slot needs a valid, unrevoked certificate for a host whose cleanup is pending. Ordinary WebSocket links now drop the permits a few lines early as well, unlike TCP/QUIC's `release_cleanup_handshake`, which keeps them for ordinary links. That's harmless: `RemoteLink::websocket` passes no `Admission`, so it can't be primed, and ordinary `admit_observed` only takes a lock and does a `try_send`, with the map capped at 10 000. ## Findings **P2-1: a pending removal still blocks re-adding its address (and its SSH key), forever for a dead host.** - For a UUID owner, `commit_revocation_intent` leaves `remote_hosts.revoked_at` NULL until the receipt (`remote_authority.rs:258-271`). Round 6 now keeps the `remote_host_bootstrap` row as well (`remote_removal.rs:306-313`). - Every duplicate check only looks at `h.revoked_at IS NULL` and the bootstrap row, so all of them treat the removed host as still added: - `existing_host` / `ensure_not_already_added` (`bootstrap_remote.rs:175`) → "This host is already added. Open its terminal from Console." with the removed host's id. - `issue_invitation`'s `occupied` (`remote_http.rs:452`) → 409. - `check_stuck_install` (`bootstrap_remote.rs:676`) and `reissue_stuck_install`'s `registered` (`:738`) → `ours=true`. A journalled reinstall then takes `Replace`, and `issue_authorized_in` (which checks only `revoked_at IS NULL`) mints a join token for the removed host. Access isn't regained: `admit` and renewal refuse while `remote_login_cleanup.confirmed=0`, and a redeemed cert only gets cleanup-only admission. But setup reinstalls a host the person just removed. - Reproducer (`repro.rs`, which runs the three production queries word for word after a real `finish_host`): ```text r6 (branch+dev): J269 pending removal: already_added_host=true issue_occupied=true stuck_install_ours=true ... -> FAILED (green-removal.log) r5 (delete restored): J269 pending removal: already_added_host=false issue_occupied=false stuck_install_ours=false ... -> ok (red-retention.log) ``` - Scenario: a machine dies and is reinstalled at the same address. The owner removes it, it stays `retired-unreachable` because no receipt ever comes, and adding the address again is refused for good. The row it points at has no actions (J249 P3-3). Before round 6 the add worked. - Fix: have these checks leave out hosts with a `remote_host_removals` row, as `remote_removal.rs:319` already does for the pin-sharing query. Add a test that re-adding the address after a pending removal is accepted. - Defence in depth: `issue_authorized_in` (`enrollment.rs:278`) and `redeem` (`:404`) should also refuse while `remote_login_cleanup.confirmed=0`, matching `admit`, renewal and `bootstrap_host`. **P3-1: retained rows still reserve machine names.** `wanted_name` (`plugins/mesh/src/store.rs:283`) reads every bootstrap row. A pending-removed host's typed name therefore stays in `records`, and a new host reporting that hostname gets a suffix. Leave out removal rows there too. **P3-2: the pending sentence contradicts itself.** "host.example: revoked, cleanup pending. host.example was removed here, but infraternal is still installed on it…" says "removed" while the row stays listed as pending, and repeats the name. Drop the second subject ("revoked, cleanup pending. infraternal is still installed on it. Remove it there as root with: …"). The UI toast already reads that way. **Verdict: REQUEST CHANGES.** Fix P2-1 with a regression test. The P3s can ride along. Every earlier finding is fixed, and the security items pass.
kayg commented 2026-11-05 23:51:55 +00:00
Author
Owner
Copy link

J196 round7 starts from reviewed r6 66f8144c4d35d6b2996fd4a1b8ef9192f03c0d65, branch agent-fsync-offloop-j196-r7. Read J269 comment81294 and compute01:~/j269-evidence/repro.rs. Merged and pushed origin/dev 9b0c731e89b70408ac719ef3a10cb1bf36bebf99 first, retaining the reviewed implementation and taking its oversized restart fixture fix.

Plan/estimate: 120 minutes plus build-slot queue. Copy J269 reproducer first and observe all three production lookup results; add actual adapter/re-enrollment regressions. Exclude removal records from address/key/stuck-install checks and prevent pending-cleanup issuance/redemption. Separately release typed machine-name reservations, then simplify pending copy to one sentence naming the host once. Each fix gets a failing-first test, its own commit and immediate push. Run targeted regressions and scripts/verify quick in the foreground, as requested for this round; no new fast run. Keep one Rust target, delete it at the end. Post head, red/green receipts and exact quick output. No merge/deployment or unrelated retired-row product change.

J196 round7 starts from reviewed r6 `66f8144c4d35d6b2996fd4a1b8ef9192f03c0d65`, branch `agent-fsync-offloop-j196-r7`. Read J269 comment81294 and `compute01:~/j269-evidence/repro.rs`. Merged and pushed origin/dev `9b0c731e89b70408ac719ef3a10cb1bf36bebf99` first, retaining the reviewed implementation and taking its oversized restart fixture fix. Plan/estimate: 120 minutes plus build-slot queue. Copy J269 reproducer first and observe all three production lookup results; add actual adapter/re-enrollment regressions. Exclude removal records from address/key/stuck-install checks and prevent pending-cleanup issuance/redemption. Separately release typed machine-name reservations, then simplify pending copy to one sentence naming the host once. Each fix gets a failing-first test, its own commit and immediate push. Run targeted regressions and `scripts/verify quick` in the foreground, as requested for this round; no new fast run. Keep one Rust target, delete it at the end. Post head, red/green receipts and exact quick output. No merge/deployment or unrelated retired-row product change.
kayg commented 2026-11-06 00:13:38 +00:00
Author
Owner
Copy link

J196 r7 checkpoint: dev integration 674e76ed581577409b17cd4d36359151aa61b305 is pushed, taking required dev9b0c731e8. Test-only recovery checkpoint 6e3563472 is now pushed on agent-fsync-offloop-j196-r7; it will be split into the atomic verified fixes before final report. No production fix yet.

Red run acquired compute01 slot4 after about15 minutes queued; it is compiling the remote-feature test binary. PID3324262, compute01:~/j196-evidence/r7/red-run.log, individual red-*.log files. Copied J269 reproducer requires (already_added_host, issue_occupied, stuck_install_ours) == (false,false,false) after real pending removal. Added actual adapter tests for address/key reuse, fresh invitation and refusal to resume the old identity; pending-cleanup issuance/redemption tests; retained-name and one-host message tests. Existing bootstrap coordinates are kept throughout. One target only: compute01 /home/claude/worktrees/j196/target; foreground polling continues.

J196 r7 checkpoint: dev integration `674e76ed581577409b17cd4d36359151aa61b305` is pushed, taking required dev9b0c731e8. Test-only recovery checkpoint `6e3563472` is now pushed on `agent-fsync-offloop-j196-r7`; it will be split into the atomic verified fixes before final report. No production fix yet. Red run acquired compute01 slot4 after about15 minutes queued; it is compiling the remote-feature test binary. PID3324262, `compute01:~/j196-evidence/r7/red-run.log`, individual `red-*.log` files. Copied J269 reproducer requires `(already_added_host, issue_occupied, stuck_install_ours) == (false,false,false)` after real pending removal. Added actual adapter tests for address/key reuse, fresh invitation and refusal to resume the old identity; pending-cleanup issuance/redemption tests; retained-name and one-host message tests. Existing bootstrap coordinates are kept throughout. One target only: compute01 `/home/claude/worktrees/j196/target`; foreground polling continues.
kayg commented 2026-11-06 00:22:14 +00:00
Author
Owner
Copy link

J196 r7 red evidence: the copied J269 reproducer fails with (already_added_host, issue_occupied, stuck_install_ours) = (true,true,true) after a real pending removal; all must be false. Actual adapter tests also reject both address/key reuse and a fresh invitation. The naming reproducer gets minipc01-2 instead of minipc01, and the message test sees the host twice. Defence-in-depth tests demonstrate issuance and redemption still accept pending-cleanup authority.

Artifacts: compute01 ~/j196-evidence/r7/red-*.log, red-run.log. Production fixes are prepared; the corrected reinstall regression now requires fresh enrollment rather than simply refusing setup. Its additional red run and the green suite are queued together on compute01 (PID3676720; green-run.log), reusing the one target /home/claude/worktrees/j196/target. Both VM queues were checked before queuing; neither had free capacity. Foreground polling continues. The test-only WIP will be replaced by the three atomic verified commits.

J196 r7 red evidence: the copied J269 reproducer fails with `(already_added_host, issue_occupied, stuck_install_ours) = (true,true,true)` after a real pending removal; all must be false. Actual adapter tests also reject both address/key reuse and a fresh invitation. The naming reproducer gets `minipc01-2` instead of `minipc01`, and the message test sees the host twice. Defence-in-depth tests demonstrate issuance and redemption still accept pending-cleanup authority. Artifacts: compute01 `~/j196-evidence/r7/red-*.log`, `red-run.log`. Production fixes are prepared; the corrected reinstall regression now requires fresh enrollment rather than simply refusing setup. Its additional red run and the green suite are queued together on compute01 (PID3676720; `green-run.log`), reusing the one target `/home/claude/worktrees/j196/target`. Both VM queues were checked before queuing; neither had free capacity. Foreground polling continues. The test-only WIP will be replaced by the three atomic verified commits.
kayg commented 2026-11-06 00:28:25 +00:00
Author
Owner
Copy link

J196 r7 fixes are verified and pushed individually on agent-fsync-offloop-j196-r7:

  • fc4ac1a7d: active bootstrap checks exclude removal records; removed journals take fresh enrollment, never Replace. Pending-cleanup invitation issuance/redemption also refuse, matching admission. Bootstrap history remains for manual cleanup.
  • c42408e4a: pending removal releases typed machine-name reservations without deleting its bootstrap history.
  • ad92f1ccb4b92cb3d21154c6c299d9f069aade9c: pending response is one sentence naming the host once and keeps the manual command.

The temporary test-only WIP was removed from final history. Required dev 9b0c731e8 is merged in 674e76ed5.

Red/green artifacts are under compute01 ~/j196-evidence/r7/. Copied J269 reproducer: (true,true,true) before, (false,false,false) after. Green results: real removal tests10/10; pending-removal adapter tests3/3; pending token tests2/2; existing already-added tests4/4; existing foreign-install reenrollment1/1; naming10/10. The corrected reinstall test also failed first (red-corrected-reinstall.log). Logs named green-remote_removal_tests.log, green-pending_removal_tests.log, green-pending_cleanup_refuses_join_token.log, green-already_added.log, green-a_machine_another_server_enrolled.log, green-name.log.

nix develop -c scripts/verify quick is running through compute01 build-slot on the full head above, foreground-polled (PID3699503, quick.log). Both VM queues were checked; compute01 had free slots. Only target: /home/claude/worktrees/j196/target. No merge or deployment.

J196 r7 fixes are verified and pushed individually on `agent-fsync-offloop-j196-r7`: - `fc4ac1a7d`: active bootstrap checks exclude removal records; removed journals take fresh enrollment, never Replace. Pending-cleanup invitation issuance/redemption also refuse, matching admission. Bootstrap history remains for manual cleanup. - `c42408e4a`: pending removal releases typed machine-name reservations without deleting its bootstrap history. - `ad92f1ccb4b92cb3d21154c6c299d9f069aade9c`: pending response is one sentence naming the host once and keeps the manual command. The temporary test-only WIP was removed from final history. Required dev `9b0c731e8` is merged in `674e76ed5`. Red/green artifacts are under compute01 `~/j196-evidence/r7/`. Copied J269 reproducer: `(true,true,true)` before, `(false,false,false)` after. Green results: real removal tests10/10; pending-removal adapter tests3/3; pending token tests2/2; existing already-added tests4/4; existing foreign-install reenrollment1/1; naming10/10. The corrected reinstall test also failed first (`red-corrected-reinstall.log`). Logs named `green-remote_removal_tests.log`, `green-pending_removal_tests.log`, `green-pending_cleanup_refuses_join_token.log`, `green-already_added.log`, `green-a_machine_another_server_enrolled.log`, `green-name.log`. `nix develop -c scripts/verify quick` is running through compute01 build-slot on the full head above, foreground-polled (PID3699503, `quick.log`). Both VM queues were checked; compute01 had free slots. Only target: `/home/claude/worktrees/j196/target`. No merge or deployment.
kayg commented 2026-11-06 00:32:44 +00:00
Author
Owner
Copy link

J196 r7 quick pre-check stopped on D345: the new include! test fragment could not carry an inner module doc. Fixed by loading a normal documented Rust test module, separately committed/pushed as a73d7e1d01e782059eefb55e016283b8cdfe2dba; no production behavior change.

Rerun has passed the cheap checks and is compiling workspace clippy. Foreground polling continues on compute01, PID3727156, ~/j196-evidence/r7/quick.log; first D345 failure retained as quick-doc-red.log. Both VM slot statuses were checked; compute01 had a free slot, reusing the existing target. The three production fixes remain their own commits. No merge or deployment.

J196 r7 quick pre-check stopped on D345: the new `include!` test fragment could not carry an inner module doc. Fixed by loading a normal documented Rust test module, separately committed/pushed as `a73d7e1d01e782059eefb55e016283b8cdfe2dba`; no production behavior change. Rerun has passed the cheap checks and is compiling workspace clippy. Foreground polling continues on compute01, PID3727156, `~/j196-evidence/r7/quick.log`; first D345 failure retained as `quick-doc-red.log`. Both VM slot statuses were checked; compute01 had a free slot, reusing the existing target. The three production fixes remain their own commits. No merge or deployment.
kayg commented 2026-11-06 00:38:58 +00:00
Author
Owner
Copy link

J196 r7 workspace clippy found one avoidable temporary-string allocation in pending message rendering and two missing code quotes in the copied reproducer's doc comment. Fixed without suppressions in f7cc793d27 (pushed). The failed output is retained in compute01 ~/j196-evidence/r7/quick-clippy-red.log.

The complete quick pre-check is queued again on this head using the same target (quick.log). Both VMs were busy when checked; compute01 had the shorter queue and warm artifacts. Foreground polling continues. No change to the three production fixes or their regression behavior.

J196 r7 workspace clippy found one avoidable temporary-string allocation in pending message rendering and two missing code quotes in the copied reproducer's doc comment. Fixed without suppressions in `f7cc793d27` (pushed). The failed output is retained in compute01 `~/j196-evidence/r7/quick-clippy-red.log`. The complete quick pre-check is queued again on this head using the same target (`quick.log`). Both VMs were busy when checked; compute01 had the shorter queue and warm artifacts. Foreground polling continues. No change to the three production fixes or their regression behavior.
kayg commented 2026-11-06 00:58:00 +00:00
Author
Owner
Copy link

J196 r7 verification checkpoint: final source remains pushed at f7cc793d27, no edits since the lint correction. Quick has passed workspace clippy and all changed library suites: infraternal agent266 passed/23 ignored, mesh34 passed, server542 passed/26 ignored, all0 failed. The existing scale/socket tests completed successfully after serial-lock waiting. Quick is compiling the separate CLI bin test feature set, then runs tracer typecheck.

Foreground polling continues; current log compute01 ~/j196-evidence/r7/quick.log, wrapper PID3830433. One target only: /home/claude/worktrees/j196/target. Earlier red/green artifacts are retained in the same evidence directory. No merge or deployment.

J196 r7 verification checkpoint: final source remains pushed at `f7cc793d27`, no edits since the lint correction. Quick has passed workspace clippy and all changed library suites: infraternal agent266 passed/23 ignored, mesh34 passed, server542 passed/26 ignored, all0 failed. The existing scale/socket tests completed successfully after serial-lock waiting. Quick is compiling the separate CLI bin test feature set, then runs tracer typecheck. Foreground polling continues; current log compute01 `~/j196-evidence/r7/quick.log`, wrapper PID3830433. One target only: `/home/claude/worktrees/j196/target`. Earlier red/green artifacts are retained in the same evidence directory. No merge or deployment.
kayg commented 2026-11-06 01:00:57 +00:00
Author
Owner
Copy link

J196 round 7 complete. Branch agent-fsync-offloop-j196-r7, head f7cc793d27800d15ddde590fb51ed70ba7fe7e16. Required dev 9b0c731e89b70408ac719ef3a10cb1bf36bebf99 is merged in 674e76ed5. Pushed each logical commit to this branch. No merge to dev, deployment or issue closure.

All three J269 findings are fixed:

  • P2, fc4ac1a7d: all active address/key and interrupted-install checks exclude removal records. Reinstall takes fresh enrollment with a new host identity; it never reissues the removed identity. Pending-cleanup token issuance/redemption also refuse as defence in depth. Cleanup retains its bootstrap metadata and manual command.
  • P3 names, c42408e4a: removal records no longer reserve typed machine names; the bootstrap row remains available for cleanup.
  • P3 message, ad92f1ccb: one sentence naming the host once, without “was removed” or a reconnect requirement; the manual command remains both structured and visible in the message.
  • Verification corrections are separate: a73d7e1d0 loads the new regression file as a documented Rust module; f7cc793d2 resolves workspace lint findings without suppressions. The temporary test-only WIP is absent from final history.

Failing-first evidence and green results, compute01 ~/j196-evidence/r7/ (selected raw logs attached):

Check Before After
J269 copied reproducer after real pending removal (true,true,true) (false,false,false)
Actual address/key reuse, fresh invitation and removed journal 3 failed 3 passed
Pending-cleanup issuance/redemption 2 failed 2 passed
Name release minipc01-2 minipc01; naming suite 10 passed
Pending wording host named twice one host, one sentence; removal suite 10 passed

The corrected journal test failed first too (red-corrected-reinstall.log); green checks assert set-aside/new enrollment, no Replace, a fresh invitation and no reissuance of the old invitation. Existing duplicate protections 4/4 and foreign-install reenrollment 1/1 also pass. All retained-coordinate checks preserve the old bootstrap row.

Requested nix develop -c scripts/verify quick ran on the exact full head above, through build-slot with foreground polling. Both VM queues were checked; compute01 reused the job's single target. Unit results: infraternal agent 266 passed/23 ignored; mesh 34 passed; server 542 passed/26 ignored; CLI 211 passed/15 ignored; zero failures. The oversized restart fixture from dev passes. This round ran the requested quick pre-check, not another fast gate.

Verbatim final output:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| scope guard               | PASS         |
| migration names           | PASS         |
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| clippy (workspace)        | PASS         |
| unit tests (changed)      | PASS         |
| unit tests (changed, bin-only) | PASS         |
| ui literals               | PASS         |
| tracer typecheck          | PASS         |
+---------------------------+--------------+

QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done.
QUICK_EXIT=0
QUICK_DONE

Artifacts: quick.log is the complete final output; quick-doc-red.log and quick-clippy-red.log retain the two corrected pre-check failures. The targeted red/green logs remain in the same evidence directory. Cleanup removed the 22 GiB target /home/claude/worktrees/j196/target and /home/claude/j196-st; cleanup.log and process-sweep.log confirm deletion and no own build/test/gate processes remain. Local control worktree is clean. No blockers or owner decision needed for these findings.

J196 round 7 complete. Branch `agent-fsync-offloop-j196-r7`, head `f7cc793d27800d15ddde590fb51ed70ba7fe7e16`. Required dev `9b0c731e89b70408ac719ef3a10cb1bf36bebf99` is merged in `674e76ed5`. Pushed each logical commit to this branch. No merge to dev, deployment or issue closure. All three J269 findings are fixed: - P2, `fc4ac1a7d`: all active address/key and interrupted-install checks exclude removal records. Reinstall takes fresh enrollment with a new host identity; it never reissues the removed identity. Pending-cleanup token issuance/redemption also refuse as defence in depth. Cleanup retains its bootstrap metadata and manual command. - P3 names, `c42408e4a`: removal records no longer reserve typed machine names; the bootstrap row remains available for cleanup. - P3 message, `ad92f1ccb`: one sentence naming the host once, without “was removed” or a reconnect requirement; the manual command remains both structured and visible in the message. - Verification corrections are separate: `a73d7e1d0` loads the new regression file as a documented Rust module; `f7cc793d2` resolves workspace lint findings without suppressions. The temporary test-only WIP is absent from final history. Failing-first evidence and green results, compute01 `~/j196-evidence/r7/` (selected raw logs attached): | Check | Before | After | | --- | --- | --- | | J269 copied reproducer after real pending removal | `(true,true,true)` | `(false,false,false)` | | Actual address/key reuse, fresh invitation and removed journal | 3 failed | 3 passed | | Pending-cleanup issuance/redemption | 2 failed | 2 passed | | Name release | `minipc01-2` | `minipc01`; naming suite 10 passed | | Pending wording | host named twice | one host, one sentence; removal suite 10 passed | The corrected journal test failed first too (`red-corrected-reinstall.log`); green checks assert set-aside/new enrollment, no Replace, a fresh invitation and no reissuance of the old invitation. Existing duplicate protections 4/4 and foreign-install reenrollment 1/1 also pass. All retained-coordinate checks preserve the old bootstrap row. Requested `nix develop -c scripts/verify quick` ran on the exact full head above, through build-slot with foreground polling. Both VM queues were checked; compute01 reused the job's single target. Unit results: infraternal agent 266 passed/23 ignored; mesh 34 passed; server 542 passed/26 ignored; CLI 211 passed/15 ignored; zero failures. The oversized restart fixture from dev passes. This round ran the requested quick pre-check, not another fast gate. Verbatim final output: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | scope guard | PASS | | migration names | PASS | | Nix Cargo vendoring | PASS | | fmt | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | clippy (workspace) | PASS | | unit tests (changed) | PASS | | unit tests (changed, bin-only) | PASS | | ui literals | PASS | | tracer typecheck | PASS | +---------------------------+--------------+ QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done. QUICK_EXIT=0 QUICK_DONE ``` Artifacts: `quick.log` is the complete final output; `quick-doc-red.log` and `quick-clippy-red.log` retain the two corrected pre-check failures. The targeted red/green logs remain in the same evidence directory. Cleanup removed the 22 GiB target `/home/claude/worktrees/j196/target` and `/home/claude/j196-st`; `cleanup.log` and `process-sweep.log` confirm deletion and no own build/test/gate processes remain. Local control worktree is clean. No blockers or owner decision needed for these findings.
cleanup.log
100 B
quick.log
130 KiB
red-pending_removal_releases_its_bootstrap_coordinates.log
14 KiB
green-remote_removal_tests.log
7.2 KiB
red-corrected-reinstall.log
2.5 KiB
green-pending_removal_tests.log
6.1 KiB
red-name.log
2.8 KiB
green-name.log
1 KiB
process-sweep.log
36 B
kayg commented 2026-11-06 01:03:14 +00:00
Author
Owner
Copy link

J280 review of round 7 started. Target agent-fsync-offloop-j196-r7 at f7cc793d27800d15ddde590fb51ed70ba7fe7e16, scratch merge with origin/dev 363d95468 on compute01 (~/worktrees/j280). Plan (~60 min): rerun J269's reproducer on the head, red/green the three fixes, run the removal/cleanup/replay suites on the merge, and check the security points (identity inheritance on re-add, cleanup racing a new install, duplicate names). Review only: no verify fast/quick, nothing pushed or merged.

J280 review of round 7 started. Target `agent-fsync-offloop-j196-r7` at `f7cc793d27800d15ddde590fb51ed70ba7fe7e16`, scratch merge with origin/dev `363d95468` on compute01 (`~/worktrees/j280`). Plan (~60 min): rerun J269's reproducer on the head, red/green the three fixes, run the removal/cleanup/replay suites on the merge, and check the security points (identity inheritance on re-add, cleanup racing a new install, duplicate names). Review only: no verify fast/quick, nothing pushed or merged.
kayg commented 2026-11-06 01:41:25 +00:00
Author
Owner
Copy link

J280 review of round 7: APPROVE the three fixes. All three are correct, and each has a test that fails without it. One merge blocker sits outside the fixes: the branch no longer merges cleanly with current dev. Whoever merges must resolve one conflict as described below, then run the full gate.

Target agent-fsync-offloop-j196-r7 at f7cc793d27800d15ddde590fb51ed70ba7fe7e16. Scratch merge with origin/dev 431441d53 (newer than 363d95468) is b9fe10ce1, local to compute01, never pushed. Review only: no verify fast/quick per the brief, nothing pushed, merged or closed. Evidence: claude@infraternal-dev-01-compute01:~/j280-evidence/ (run.sh, run.log, green-*.log, red-*.log, red-*.diff, j269-repro.log, scratch-*.rs, remote_accept-vs-dev.diff).

Merge blocker: conflict in remote_accept.rs (not a defect in the fixes)

dev's #285 recovery refactor moved the TCP accept body into handle_tcp_connection and added the recovery ALPN. The branch's version of that block has two semantic changes, and both must survive the merge:

  1. AuthorityGuard::admit → AuthorityGuard::admit_transport (cleanup-only admission for pending hosts);
  2. admission.release_cleanup_handshake(); right after RemoteLink::tcp_admitted(...).

My scratch resolution took dev's structure and applied exactly those two edits (remote_accept-vs-dev.diff, 50 lines). On that merge, remote_accept 1/1, remote_recovery 8/8, websocket_cleanup 1/1 and cleanup_delivery_ 3/3 pass. The merge commit still needs the full gate.

Merge interaction to know about (P3, follow-up test): dev's certificate recovery (remote_recovery.rs) checks only c.revoked_at/h.revoked_at. For a UUID owner, a pending removal leaves both NULL until the receipt, so an expired pending-removed host can now recover a new 7-day certificate. It gains no access: admit refuses while remote_login_cleanup.confirmed=0, and admit_transport falls through to admit_cleanup, which is cleanup-only with a 300 s cap and never publishes the host. Arguably this is useful, because an expired host can still receive its cleanup. But it's a new mint path outside the round-7 defence in depth. Add a test in the merge commit that a recovered certificate for a pending host gets only cleanup admission.

1. P2, a pending removal no longer counts as added: fixed

Check Branch + dev Fix reverted
Production existing_host (address and pinned key), issue_bootstrap writer check (occupied), stuck-install journal (pending_removal_tests 3 tests) ok (green-pending_removal.log, 8/8 incl. remote_removal ones) 3 FAILED (red-p2-queries.log; NOT EXISTS remote_host_removals dropped from bootstrap_remote.rs + remote_http.rs, 4 queries)
Issuance refused during cleanup ok FAILED pending_cleanup_refuses_join_token_issuance (red-defence.log)
Redemption refused, invitation not consumed ok FAILED pending_cleanup_refuses_join_token_redemption (red-redeem.log, red-redeem.diff)

J269's repro.rs rerun on the head still prints already_added_host=true issue_occupied=true stuck_install_ours=true (j269-repro.log:396). That's expected and not a regression: it copies the round-6 SQL word for word and never calls the production functions. The author's updated reproducer pending_removal_releases_its_bootstrap_coordinates uses the new SQL and passes. The stronger evidence is the three pending_removal_tests above, which call existing_host, issue_bootstrap and run_stuck directly.

Security.

  • No identity inheritance through a fresh id: the invitation request carries a client-chosen host_id. Before round 7, that id couldn't be reused at the old destination, because occupied blocked it. Now occupied is false, so I added scratch test j280_pending_removal_refuses_invitation_for_the_removed_id. It passes the removed host's own id with the old address and with a new one. Both are refused, and 0 unconsumed tokens exist for the removed id (green-j280_.log). With the issuance guard reverted, it's still refused (red-defence-j280.log), because the writer transaction rolls back on the retained bootstrap row's key. So on the bootstrap path the new guard is a second layer. On the no-destination invite path it's the only layer, and pending_cleanup_refuses_join_token_issuance covers that.
  • A reinstall isn't treated as the removed host: pending_removal_cannot_resume_the_old_stuck_install shows the journal for a removed id takes Re-enroll (supersede_stuck_install, then prepare-join-key), never replace_stuck_install. It mints exactly one token under a new host id and doesn't reissue the old invitation.
  • Cleanup can't remove the new install: supersede stops the old unit and moves the old base aside (previous_attempt.sh:842), so the old daemon can't reconnect and self-uninstall over the new one. The manual --bootstrap-uninstall <old id> refuses unless .bootstrap-owner names the old id (plugins/remote/src/uninstall.rs:146), and the new install writes its own id there. On the server, the receipt deletes only remote_host_bootstrap WHERE host_id=<old>, gated on that host's removal row and a confirmed receipt from the same person (remote_authority.rs:121). Nothing is keyed by address.

2. P3, machine names: fixed, no duplicates possible

  • pending_removal_releases_its_typed_machine_name is ok (green-naming.log, 11/11). It FAILED with c42408e4a's store.rs change reverted (red-naming.log, naming.rs:463).
  • Two hosts can't share a name. mesh_nodes.name is UNIQUE COLLATE NOCASE (migration 0012), and naming() checks every live node, not only bootstrap records. Records only make a reported name yield. Scratch test j280_attached_pending_node_and_replacement_never_share_a_name keeps the old node attached as minipc01 during cleanup, then attaches the replacement at the same address with the same hostname. The names are ["monitor01", "minipc01", "minipc01-2"] (green-naming.log). After cleanup removes the old node, the name is free for the newcomer's next attach.

3. P3, message: fixed

Head output: host.example: revoked, cleanup pending; infraternal is still installed on it, so remove it there as root with: /usr/local/bin/infraternal --bootstrap-uninstall 247a3fd33f9140a89645739528129f55. That's one sentence, the host named once, and no "was removed". pending_removal_describes_one_host_without_claiming_completion (both uninstall modes) is ok. It FAILED with ad92f1ccb's outcome change reverted (red-message.log, remote_removal_tests.rs:538). Nit, not blocking: the sentence ends with . right after the host id, so a copy of the in-message command picks up a stray dot, and --bootstrap-uninstall then refuses (safely). The UI and CLI use the structured manual field (host_remove::pending_removal_prints_manual_command_for_people ok, green-cli.log).

Suites on the merge (all green)

remote_removal 16/16, pending_removal 8/8, pending_cleanup_refuses_join_token 2/2, already_added 4/4, a_machine_another_server_enrolled 1/1, login_revocation 13/13 (replay), cleanup_delivery_ 3/3, websocket_cleanup 1/1, remote_recovery 8/8, remote_accept 1/1, mesh naming 11/11 (incl. scratch), CLI host_remove:: 2/2. The interrupted_install and cleanup_lane filters matched no lib tests here; the stuck-install case is covered by pending_removal_cannot_resume_the_old_stuck_install.

D345 doc comments are present on every new item. Vocabulary is fine ("infraternal agent").

Verdict: APPROVE. Before merge: resolve the remote_accept.rs conflict keeping admit_transport + release_cleanup_handshake(), then run the full gate on the merge commit. Follow-up P3: a test that recovery for a pending host yields only cleanup admission.

J280 review of round 7: **APPROVE** the three fixes. All three are correct, and each has a test that fails without it. One merge blocker sits outside the fixes: the branch **no longer merges cleanly with current dev**. Whoever merges must resolve one conflict as described below, then run the full gate. Target `agent-fsync-offloop-j196-r7` at `f7cc793d27800d15ddde590fb51ed70ba7fe7e16`. Scratch merge with origin/dev `431441d53` (newer than `363d95468`) is `b9fe10ce1`, local to compute01, never pushed. Review only: no `verify fast`/`quick` per the brief, nothing pushed, merged or closed. Evidence: `claude@infraternal-dev-01-compute01:~/j280-evidence/` (`run.sh`, `run.log`, `green-*.log`, `red-*.log`, `red-*.diff`, `j269-repro.log`, `scratch-*.rs`, `remote_accept-vs-dev.diff`). ## Merge blocker: conflict in `remote_accept.rs` (not a defect in the fixes) dev's #285 recovery refactor moved the TCP accept body into `handle_tcp_connection` and added the recovery ALPN. The branch's version of that block has two semantic changes, and both must survive the merge: 1. `AuthorityGuard::admit` → `AuthorityGuard::admit_transport` (cleanup-only admission for pending hosts); 2. `admission.release_cleanup_handshake();` right after `RemoteLink::tcp_admitted(...)`. My scratch resolution took dev's structure and applied exactly those two edits (`remote_accept-vs-dev.diff`, 50 lines). On that merge, `remote_accept` 1/1, `remote_recovery` 8/8, `websocket_cleanup` 1/1 and `cleanup_delivery_` 3/3 pass. The merge commit still needs the full gate. **Merge interaction to know about (P3, follow-up test):** dev's certificate recovery (`remote_recovery.rs`) checks only `c.revoked_at`/`h.revoked_at`. For a UUID owner, a pending removal leaves both NULL until the receipt, so an expired pending-removed host can now recover a new 7-day certificate. It gains no access: `admit` refuses while `remote_login_cleanup.confirmed=0`, and `admit_transport` falls through to `admit_cleanup`, which is cleanup-only with a 300 s cap and never publishes the host. Arguably this is useful, because an expired host can still receive its cleanup. But it's a new mint path outside the round-7 defence in depth. Add a test in the merge commit that a recovered certificate for a pending host gets only cleanup admission. ## 1. P2, a pending removal no longer counts as added: fixed | Check | Branch + dev | Fix reverted | |---|---|---| | Production `existing_host` (address and pinned key), `issue_bootstrap` writer check (`occupied`), stuck-install journal (`pending_removal_tests` 3 tests) | ok (`green-pending_removal.log`, 8/8 incl. remote_removal ones) | 3 FAILED (`red-p2-queries.log`; `NOT EXISTS remote_host_removals` dropped from `bootstrap_remote.rs` + `remote_http.rs`, 4 queries) | | Issuance refused during cleanup | ok | FAILED `pending_cleanup_refuses_join_token_issuance` (`red-defence.log`) | | Redemption refused, invitation not consumed | ok | FAILED `pending_cleanup_refuses_join_token_redemption` (`red-redeem.log`, `red-redeem.diff`) | J269's `repro.rs` rerun on the head still prints `already_added_host=true issue_occupied=true stuck_install_ours=true` (`j269-repro.log:396`). That's expected and not a regression: it copies the round-6 SQL word for word and never calls the production functions. The author's updated reproducer `pending_removal_releases_its_bootstrap_coordinates` uses the new SQL and passes. The stronger evidence is the three `pending_removal_tests` above, which call `existing_host`, `issue_bootstrap` and `run_stuck` directly. **Security.** - **No identity inheritance through a fresh id:** the invitation request carries a client-chosen `host_id`. Before round 7, that id couldn't be reused at the old destination, because `occupied` blocked it. Now `occupied` is false, so I added scratch test `j280_pending_removal_refuses_invitation_for_the_removed_id`. It passes the removed host's own id with the old address and with a new one. Both are refused, and 0 unconsumed tokens exist for the removed id (`green-j280_.log`). With the issuance guard reverted, it's still refused (`red-defence-j280.log`), because the writer transaction rolls back on the retained bootstrap row's key. So on the bootstrap path the new guard is a second layer. On the no-destination `invite` path it's the only layer, and `pending_cleanup_refuses_join_token_issuance` covers that. - **A reinstall isn't treated as the removed host:** `pending_removal_cannot_resume_the_old_stuck_install` shows the journal for a removed id takes Re-enroll (`supersede_stuck_install`, then `prepare-join-key`), never `replace_stuck_install`. It mints exactly one token under a new host id and doesn't reissue the old invitation. - **Cleanup can't remove the new install:** supersede stops the old unit and moves the old base aside (`previous_attempt.sh:842`), so the old daemon can't reconnect and self-uninstall over the new one. The manual `--bootstrap-uninstall <old id>` refuses unless `.bootstrap-owner` names the old id (`plugins/remote/src/uninstall.rs:146`), and the new install writes its own id there. On the server, the receipt deletes only `remote_host_bootstrap WHERE host_id=<old>`, gated on that host's removal row and a confirmed receipt from the same person (`remote_authority.rs:121`). Nothing is keyed by address. ## 2. P3, machine names: fixed, no duplicates possible - `pending_removal_releases_its_typed_machine_name` is ok (`green-naming.log`, 11/11). It FAILED with `c42408e4a`'s `store.rs` change reverted (`red-naming.log`, `naming.rs:463`). - **Two hosts can't share a name.** `mesh_nodes.name` is `UNIQUE COLLATE NOCASE` (migration 0012), and `naming()` checks every live node, not only bootstrap records. Records only make a *reported* name yield. Scratch test `j280_attached_pending_node_and_replacement_never_share_a_name` keeps the old node attached as `minipc01` during cleanup, then attaches the replacement at the same address with the same hostname. The names are `["monitor01", "minipc01", "minipc01-2"]` (`green-naming.log`). After cleanup removes the old node, the name is free for the newcomer's next attach. ## 3. P3, message: fixed Head output: `host.example: revoked, cleanup pending; infraternal is still installed on it, so remove it there as root with: /usr/local/bin/infraternal --bootstrap-uninstall 247a3fd33f9140a89645739528129f55.` That's one sentence, the host named once, and no "was removed". `pending_removal_describes_one_host_without_claiming_completion` (both uninstall modes) is ok. It FAILED with `ad92f1ccb`'s `outcome` change reverted (`red-message.log`, `remote_removal_tests.rs:538`). Nit, not blocking: the sentence ends with `.` right after the host id, so a copy of the in-message command picks up a stray dot, and `--bootstrap-uninstall` then refuses (safely). The UI and CLI use the structured `manual` field (`host_remove::pending_removal_prints_manual_command_for_people` ok, `green-cli.log`). ## Suites on the merge (all green) `remote_removal` 16/16, `pending_removal` 8/8, `pending_cleanup_refuses_join_token` 2/2, `already_added` 4/4, `a_machine_another_server_enrolled` 1/1, `login_revocation` 13/13 (replay), `cleanup_delivery_` 3/3, `websocket_cleanup` 1/1, `remote_recovery` 8/8, `remote_accept` 1/1, mesh `naming` 11/11 (incl. scratch), CLI `host_remove::` 2/2. The `interrupted_install` and `cleanup_lane` filters matched no lib tests here; the stuck-install case is covered by `pending_removal_cannot_resume_the_old_stuck_install`. D345 doc comments are present on every new item. Vocabulary is fine ("infraternal agent"). **Verdict: APPROVE.** Before merge: resolve the `remote_accept.rs` conflict keeping `admit_transport` + `release_cleanup_handshake()`, then run the full gate on the merge commit. Follow-up P3: a test that recovery for a pending host yields only cleanup admission.
kayg commented 2026-11-06 01:45:00 +00:00
Author
Owner
Copy link

J196 approved-round-7 follow-up started on agent-fsync-offloop-j196-r7, base f7cc793d27800d15ddde590fb51ed70ba7fe7e16. Merging current dev 43d98df1f104f602727cfa63aab2bb22f84da12a. Read J280 comment81495 and compute01 ~/j280-evidence/remote_accept-vs-dev.diff.

Plan: (1) resolve the accept conflict with dev's certificate-recovery structure, retaining cleanup-only transport admission and prompt handshake release; commit/push the merge; (2) failing-first command-suffix test and message correction, commit/push; (3) pin pending-host recovery to cleanup-only admission and its300 s cap, commit/push; (4) foreground verify quick, post exact head/output and clean target. Estimate90 minutes plus build-slot queue. No merge to dev or deployment.

Choice for P3: retain certificate recovery for pending-removed hosts so an expired host can receive its still-durable cleanup obligation. Test ordinary admission/renewal refusal and cleanup-only admission with the exact300 s cap; do not weaken revocation or final retirement. One new target only, on the selected VM after checking both queues.

J196 approved-round-7 follow-up started on `agent-fsync-offloop-j196-r7`, base `f7cc793d27800d15ddde590fb51ed70ba7fe7e16`. Merging current dev `43d98df1f104f602727cfa63aab2bb22f84da12a`. Read J280 comment81495 and compute01 `~/j280-evidence/remote_accept-vs-dev.diff`. Plan: (1) resolve the accept conflict with dev's certificate-recovery structure, retaining cleanup-only transport admission and prompt handshake release; commit/push the merge; (2) failing-first command-suffix test and message correction, commit/push; (3) pin pending-host recovery to cleanup-only admission and its300 s cap, commit/push; (4) foreground `verify quick`, post exact head/output and clean target. Estimate90 minutes plus build-slot queue. No merge to dev or deployment. Choice for P3: retain certificate recovery for pending-removed hosts so an expired host can receive its still-durable cleanup obligation. Test ordinary admission/renewal refusal and cleanup-only admission with the exact300 s cap; do not weaken revocation or final retirement. One new target only, on the selected VM after checking both queues.
kayg commented 2026-11-06 01:59:44 +00:00
Author
Owner
Copy link

J196 follow-up baseline evidence: merge f93963bf9b is pushed, taking current dev43d98df1f's recovery accept structure and retaining cleanup-only admission and prompt handshake release. Merge tests pass: remote_accept1/1, websocket_cleanup1/1, cleanup_delivery_3/3; all eight pre-existing remote_recovery tests pass.

The strengthened copy-safety test fails as intended: the in-message command has a trailing period. Artifact compute01 ~/j196-evidence/r7-followup/baseline-pending_removal_describes_one_host.log.

The new recovery fixture initially sampled the pre-recovery epoch, before the recovered certificate's actual signing time. Corrected it to the persisted issuance epoch; this was a test mistake, not a production failure. It uses a real UUID owner and pending-removal row, actual signed challenge/proof, ordinary admission/renewal refusals, cleanup classification, and paused-time299/300 s expiry assertions. Retaining recovery is intentional: an expired host can still discharge the durable cleanup obligation without regaining ordinary access. The operations note is being corrected for dev's recovery eligibility.

Corrected baseline, command green and cap/authority mutation tests are queued in one build-slot run, reusing the single compute01 target. Both queues were busy; compute01 had no queued jobs when selected. Foreground polling continues (mutations-run.log). No merge to dev or deployment.

J196 follow-up baseline evidence: merge `f93963bf9b` is pushed, taking current dev43d98df1f's recovery accept structure and retaining cleanup-only admission and prompt handshake release. Merge tests pass: `remote_accept`1/1, `websocket_cleanup`1/1, `cleanup_delivery_`3/3; all eight pre-existing `remote_recovery` tests pass. The strengthened copy-safety test fails as intended: the in-message command has a trailing period. Artifact compute01 `~/j196-evidence/r7-followup/baseline-pending_removal_describes_one_host.log`. The new recovery fixture initially sampled the pre-recovery epoch, before the recovered certificate's actual signing time. Corrected it to the persisted issuance epoch; this was a test mistake, not a production failure. It uses a real UUID owner and pending-removal row, actual signed challenge/proof, ordinary admission/renewal refusals, cleanup classification, and paused-time299/300 s expiry assertions. Retaining recovery is intentional: an expired host can still discharge the durable cleanup obligation without regaining ordinary access. The operations note is being corrected for dev's recovery eligibility. Corrected baseline, command green and cap/authority mutation tests are queued in one build-slot run, reusing the single compute01 target. Both queues were busy; compute01 had no queued jobs when selected. Foreground polling continues (`mutations-run.log`). No merge to dev or deployment.
kayg commented 2026-11-06 02:07:15 +00:00
Author
Owner
Copy link

J196 follow-up changes are pushed separately on agent-fsync-offloop-j196-r7:

  • f93963bf9: merge dev43d98df1f, using dev's certificate-recovery accept structure plus cleanup admit_transport and handshake release for TCP and QUIC. Transport checks pass (accept1, WebSocket cleanup1, cleanup delivery3).
  • 913a8241b: pending text ends with the exact uninstall command, no trailing period. Strengthened real removal-response test fails before and passes after.
  • de03ab3af: real recovery test proves a UUID-owned pending removal can recover an expired certificate for seven days but receives only cleanup admission. Ordinary admission, operations and renewal refuse; paused-time299/300 s assertions pin the exact cap. Operations documentation now describes dev's eligible recovery path rather than claiming every expired leaf makes cleanup permanently impossible.

Corrected recovery baseline and command green pass. Mutation checks are running through build-slot: raise the cleanup guard cap to301 s, then remove ordinary admission's pending-cleanup SQL fence; each must fail the new regression, followed by a restored green. The runner traps restoration of both production files. Foreground polling continues in compute01 ~/j196-evidence/r7-followup/mutations-run.log, PID489764. One target only /home/claude/worktrees/j196/target. Exact final quick will follow on the pushed head after mutations finish. No merge to dev or deployment.

J196 follow-up changes are pushed separately on `agent-fsync-offloop-j196-r7`: - `f93963bf9`: merge dev43d98df1f, using dev's certificate-recovery accept structure plus cleanup `admit_transport` and handshake release for TCP and QUIC. Transport checks pass (accept1, WebSocket cleanup1, cleanup delivery3). - `913a8241b`: pending text ends with the exact uninstall command, no trailing period. Strengthened real removal-response test fails before and passes after. - `de03ab3af`: real recovery test proves a UUID-owned pending removal can recover an expired certificate for seven days but receives only cleanup admission. Ordinary admission, operations and renewal refuse; paused-time299/300 s assertions pin the exact cap. Operations documentation now describes dev's eligible recovery path rather than claiming every expired leaf makes cleanup permanently impossible. Corrected recovery baseline and command green pass. Mutation checks are running through build-slot: raise the cleanup guard cap to301 s, then remove ordinary admission's pending-cleanup SQL fence; each must fail the new regression, followed by a restored green. The runner traps restoration of both production files. Foreground polling continues in compute01 `~/j196-evidence/r7-followup/mutations-run.log`, PID489764. One target only `/home/claude/worktrees/j196/target`. Exact final quick will follow on the pushed head after mutations finish. No merge to dev or deployment.
kayg commented 2026-11-06 02:12:13 +00:00
Author
Owner
Copy link

J196 follow-up verification: both mutations fail for the expected reasons, and the restored test passes. Raising the cap to301 seconds fails “seven-day recovery must not extend the cleanup cap”; removing ordinary admission's outbox fence fails store.admit(&fingerprint, issued).await.is_err(). The unmodified guard/authority files are restored and git diff --exit-code on both confirms no mutant remains.

Artifacts on compute01 ~/j196-evidence/r7-followup/: red-cap-mutant.log, red-admission-mutant.log, green-recovery-before-mutants.log, green-recovery.log, green-command.log, mutations-run.log.

Exact pushed head de03ab3af5d43fa446608b01e49e919f9fa98289 is now running the complete recovery suite and nix develop -c scripts/verify quick through build-slot, foreground-polled (quick.log). Both VM slot statuses were checked; compute01 had two free slots and reuses the single target. No merge to dev or deployment.

J196 follow-up verification: both mutations fail for the expected reasons, and the restored test passes. Raising the cap to301 seconds fails “seven-day recovery must not extend the cleanup cap”; removing ordinary admission's outbox fence fails `store.admit(&fingerprint, issued).await.is_err()`. The unmodified guard/authority files are restored and `git diff --exit-code` on both confirms no mutant remains. Artifacts on compute01 `~/j196-evidence/r7-followup/`: `red-cap-mutant.log`, `red-admission-mutant.log`, `green-recovery-before-mutants.log`, `green-recovery.log`, `green-command.log`, `mutations-run.log`. Exact pushed head `de03ab3af5d43fa446608b01e49e919f9fa98289` is now running the complete recovery suite and `nix develop -c scripts/verify quick` through build-slot, foreground-polled (`quick.log`). Both VM slot statuses were checked; compute01 had two free slots and reuses the single target. No merge to dev or deployment.
kayg commented 2026-11-06 02:26:20 +00:00
Author
Owner
Copy link

J196 follow-up quick reached workspace clippy and found one test-only naming lint (issued vs issuer). Renamed the certificate validity bindings to leaf_start/leaf_end, with no suppressions or behavior change. Amended/pushed the P3 test commit as f52b624de6600290d9e87a5ee801a24ef60fb1e4; merge f93963bf9 and copy fix 913a8241b are unchanged.

The complete nine-test recovery suite had passed on the prior head, as had the cap/admission mutation checks. The full recovery suite and quick are being rerun on the updated exact head. Both VMs are busy; compute01 reuses the existing single target and avoids rebuilding all feature sets and the vendored crypto library on another architecture. Foreground polling continues, compute01:~/j196-evidence/r7-followup/quick.log. Corrected clippy failure retained as quick-clippy-red.log. No merge to dev or deployment.

J196 follow-up quick reached workspace clippy and found one test-only naming lint (`issued` vs `issuer`). Renamed the certificate validity bindings to `leaf_start`/`leaf_end`, with no suppressions or behavior change. Amended/pushed the P3 test commit as `f52b624de6600290d9e87a5ee801a24ef60fb1e4`; merge `f93963bf9` and copy fix `913a8241b` are unchanged. The complete nine-test recovery suite had passed on the prior head, as had the cap/admission mutation checks. The full recovery suite and quick are being rerun on the updated exact head. Both VMs are busy; compute01 reuses the existing single target and avoids rebuilding all feature sets and the vendored crypto library on another architecture. Foreground polling continues, `compute01:~/j196-evidence/r7-followup/quick.log`. Corrected clippy failure retained as `quick-clippy-red.log`. No merge to dev or deployment.
kayg commented 2026-11-06 02:36:06 +00:00
Author
Owner
Copy link

J196 follow-up: the full9-test recovery suite passed again after the test-binding rename. Quick then caught a rustfmt wrap changed by that rename; I had missed rerunning rustfmt. Ran cargo fmt --all on compute01 and amended/pushed the P3 test commit to 2876065237c735161dc9ce823d9f16f0f608ed51. The dev merge and command fix remain separate, unchanged commits.

No behavior change or lint suppression. Both corrected failures remain in ~/j196-evidence/r7-followup/quick-clippy-red.log and quick-fmt-red.log. The recovery suite and quick are now running on the exact formatted head (PID913402, quick.log) through build-slot; foreground polling continues. Target is still the one compute01 worktree target. No merge to dev or deployment.

J196 follow-up: the full9-test recovery suite passed again after the test-binding rename. Quick then caught a rustfmt wrap changed by that rename; I had missed rerunning rustfmt. Ran `cargo fmt --all` on compute01 and amended/pushed the P3 test commit to `2876065237c735161dc9ce823d9f16f0f608ed51`. The dev merge and command fix remain separate, unchanged commits. No behavior change or lint suppression. Both corrected failures remain in `~/j196-evidence/r7-followup/quick-clippy-red.log` and `quick-fmt-red.log`. The recovery suite and quick are now running on the exact formatted head (PID913402, `quick.log`) through build-slot; foreground polling continues. Target is still the one compute01 worktree target. No merge to dev or deployment.
kayg commented 2026-11-06 02:54:06 +00:00
Author
Owner
Copy link

j196 approved-round-7 follow-up complete on agent-fsync-offloop-j196-r7.

head: 2876065237c735161dc9ce823d9f16f0f608ed51 (pushed).

three separately pushed commits:

  • f93963bf9: merged dev 43d98df1f104f602727cfa63aab2bb22f84da12a, taking its certificate-recovery accept structure and retaining admit_transport and immediate release_cleanup_handshake() for cleanup links. the attached diff records the resolution against that dev head.
  • 913a8241b: the pending-removal message ends with the exact uninstall command, without trailing punctuation. the strengthened real-response test failed before the fix and passes afterward.
  • 287606523: retain certificate recovery for pending-removed hosts so an expired host can receive its cleanup obligation. the signed recovery test pins a seven-day certificate, refusal of ordinary admission, renewal and operations, cleanup-only admission, and expiry exactly at 300 seconds under paused time. operations documentation now describes that eligibility accurately.

evidence on compute01: ~/j196-evidence/r7-followup/, with selected logs attached here. the final recovery suite passed 9/9. baseline-pending_removal_describes_one_host.log and green-command.log record command-copy red/green. red-admission-mutant.log fails when ordinary admission's pending-cleanup fence is removed; red-cap-mutant.log fails when the cleanup cap is changed to 301 seconds. both mutants were restored, and green-recovery.log passes. the earlier test-time fixture mistake and its correction are recorded in comment 81517; it was not a production defect.

requested foreground scripts/verify quick on the exact pushed head completed with QUICK_EXIT=0. changed-package suites: 1,087 passed, 64 ignored, zero failed; the separately run feature-enabled recovery suite passed 9/9. verbatim final quick output:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| scope guard               | PASS         |
| migration names           | PASS         |
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| clippy (workspace)        | PASS         |
| unit tests (changed)      | PASS         |
| unit tests (changed, bin-only) | PASS         |
| ui literals               | PASS         |
| tracer typecheck          | PASS         |
+---------------------------+--------------+

QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done.
QUICK_EXIT=0
QUICK_DONE

cleanup complete: removed this job’s 15 GiB target and test-state directory, confirmed the worktree is clean and no own gate/test process remains. cleanup.log is attached; evidence logs remain available.

no merge to dev, deployment, or issue closure performed. this is the requested quick pre-check, not a new full fast gate.

j196 approved-round-7 follow-up complete on `agent-fsync-offloop-j196-r7`. head: `2876065237c735161dc9ce823d9f16f0f608ed51` (pushed). three separately pushed commits: - `f93963bf9`: merged dev `43d98df1f104f602727cfa63aab2bb22f84da12a`, taking its certificate-recovery accept structure and retaining `admit_transport` and immediate `release_cleanup_handshake()` for cleanup links. the attached diff records the resolution against that dev head. - `913a8241b`: the pending-removal message ends with the exact uninstall command, without trailing punctuation. the strengthened real-response test failed before the fix and passes afterward. - `287606523`: retain certificate recovery for pending-removed hosts so an expired host can receive its cleanup obligation. the signed recovery test pins a seven-day certificate, refusal of ordinary admission, renewal and operations, cleanup-only admission, and expiry exactly at 300 seconds under paused time. operations documentation now describes that eligibility accurately. evidence on compute01: `~/j196-evidence/r7-followup/`, with selected logs attached here. the final recovery suite passed 9/9. `baseline-pending_removal_describes_one_host.log` and `green-command.log` record command-copy red/green. `red-admission-mutant.log` fails when ordinary admission's pending-cleanup fence is removed; `red-cap-mutant.log` fails when the cleanup cap is changed to 301 seconds. both mutants were restored, and `green-recovery.log` passes. the earlier test-time fixture mistake and its correction are recorded in comment 81517; it was not a production defect. requested foreground `scripts/verify quick` on the exact pushed head completed with `QUICK_EXIT=0`. changed-package suites: 1,087 passed, 64 ignored, zero failed; the separately run feature-enabled recovery suite passed 9/9. verbatim final quick output: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | scope guard | PASS | | migration names | PASS | | Nix Cargo vendoring | PASS | | fmt | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | clippy (workspace) | PASS | | unit tests (changed) | PASS | | unit tests (changed, bin-only) | PASS | | ui literals | PASS | | tracer typecheck | PASS | +---------------------------+--------------+ QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done. QUICK_EXIT=0 QUICK_DONE ``` cleanup complete: removed this job’s 15 GiB target and test-state directory, confirmed the worktree is clean and no own gate/test process remains. `cleanup.log` is attached; evidence logs remain available. no merge to dev, deployment, or issue closure performed. this is the requested quick pre-check, not a new full fast gate.
quick.log
134 KiB
final-recovery-suite.log
2.7 KiB
baseline-pending_removal_describes_one_host.log
14 KiB
green-command.log
1.9 KiB
red-admission-mutant.log
2.4 KiB
red-cap-mutant.log
2.4 KiB
green-recovery.log
1.9 KiB
remote_accept-vs-dev.diff.txt
2.2 KiB
cleanup.log
135 B
kayg commented 2026-11-06 16:49:12 +00:00
Author
Owner
Copy link

J319 housekeeping, authorized by kayg (#402). Closing: Durable writes and revocation cleanup are off the event loop without a host-wide latch. The approved round-7 follow-ups, command-copy correction and recovery-cap/admission tests are integrated. Final receipt 81586 records the recovery suite 9/9, mutation red/green, changed-package suites 1087 passed and target cleanup; no outstanding request or owner question remains in this issue.

Merged on dev: be64f2902e. Re-fetched the issue and complete comment history immediately before closure and confirmed unchanged since review. Operational cleanup only; no new test run claimed.

J319 housekeeping, authorized by kayg (#402). Closing: Durable writes and revocation cleanup are off the event loop without a host-wide latch. The approved round-7 follow-ups, command-copy correction and recovery-cap/admission tests are integrated. Final receipt 81586 records the recovery suite 9/9, mutation red/green, changed-package suites 1087 passed and target cleanup; no outstanding request or owner question remains in this issue. Merged on dev: https://git.kayg.org/kayg/infraternal/commit/be64f2902ed11067d31bb875bbdb77982760524e. Re-fetched the issue and complete comment history immediately before closure and confirmed unchanged since review. Operational cleanup only; no new test run claimed.
kayg closed this issue 2026-11-06 16:49:12 +00:00
kayg commented 2026-11-08 18:22:32 +00:00
Author
Owner
Copy link

Owner decision D369 (2026-11-08, J249 P3-3): hosts stuck as "retired, unreachable" get an owner-only Forget action on their row. The audit record stays. Rejected: hiding them automatically when the certificate expires, and listing them forever. Merge dba2f6424.

**Owner decision D369 (2026-11-08, J249 P3-3):** hosts stuck as "retired, unreachable" get an owner-only **Forget** action on their row. The audit record stays. Rejected: hiding them automatically when the certificate expires, and listing them forever. Merge `dba2f6424`.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
j470b-server-mesh-rejoin
mesh-compat-382
j444b-d381-key-wins-flood
j461b-macos-utun-reader-thread
j475b-remove-host-motion
fix-reconnect-new-link
j481-d380
dev
j479-mesh-soak
fix/j476-host-facts
j475-motion-pill-split
j470-mac-node-join
root-upgrade-d376
j474-review-merge
latency-targets-120hz
nightly-suite-shards
gotatun-pin-j469
macos-utun-reader-thread
fix-flood-echo-stall
wip/j416
motion-fast-ease-out
mesh-addhost-j259
takeover-close-terminals-d368
kbd-instant-followups
fix-trust-key-double-click
nightly-suite-partial-report
appwin-icons-j207
rusqlite-slice1
swipe-content-first-d374
mesh-mtu-381
addhost-instant-open
glass-contrast-d365-mock
glass-contrast-d365
signout-followups-442
fix-startup-toast-red
research-gotatun-matrix-j317
themes-sync-j304
fix-load-flakes
fix-nix-zig-patch
fix-compute01-browser-baseline
ratchet-size-baseline-419
predict-auto-d130
revoke-coverage-d367
nightly-suite-queue-timeout
nightly-browser-suite
fix-rustsec-2026-0459
fonts-ioskeley-obel
log-retention-d375
fix-kbd-no-anim-427
fix-dev-red-specs-2
takeover-proof-305-j205
wip/j410-server-pump
ui-consistency-d372
fix-osc52-clipboard
libghostty-vt-1.0
forget-retired-host-d369
j404-merge-426
fix-setup-lock-race
fix-touch-select-426
revoke-access-report-ui
fix-predictive-echo-specs
fix-selinux-agent-exec
wip/j395-checkpoint2
wip/j395
j391-review-scratch
ui-consistency-j195-deferred-rebased
fix-rustc-paths-421
decisions-wording-341
fix-install-suite-red
fix-dev-rtt-specs
fix-pending-removal-align
fix-privileged-skip
fix-install-suite-404
review-j368
mesh-dns-j254
revoke-access-report-d356b
revoke-privilege-round5
fix-removal-confirm
review-j360
gotatun-rotation-367
deps-refresh-340
review-j354
review-J351
review-j350
addhost-takeover-d359
revoke-report-items-347
revoke-end-sessions-d362
review-j348
size-mesh-ssh-346
size-trim-337
revoke-access-diff-d356
j259-pre-rebase-b06c5a6
j305-ended-at-start-spec
addhost-sudo-consent
fix-layout-real-specs-411
fix-shell-open-411
fix-console-hosts-373
fix-shell-exit
fuzz-host-name-353
fix-host-details-410
macos-filter-j197
research-gotatun-tricks
revoke-p3-j216l
stale-state-pill
revoke-followups-j216k
revoke-progress-j216e
backup/macos01-20261106/lume/j197
backup/macos01-20261106/lume/appwin-icons-j207-stash0
backup/macos01-20261106/claude/stash-0
backup/macos01-20261106/claude/stash-1
backup/macos01-20261106/claude/J80-macos-s1
backup/macos01-20261106/claude/J80
backup/macos01-20261106/claude/J80-slice1-20261102
backup/macos01-20261106/claude/J16b-appwin-on-dev
backup/macos01-20261106/claude/appwin-launch-idle-146
backup/macos01-20261106/claude/J80-s1-run
wip/j259
trust-285-p3c
runtime-lock-j216f
mesh-ssh-p3-j175
tun-proof-p3b
trust-285-p3
terminal-polish-j271
tun-proof-p3
ssh-install-flakes-j216i
fuzz-fixes-j264
trust-285-a
trust-285-b
restart-oversized-fixture-j198
backup/macos01-20261106/lume/appwin-icons-j207
restart-bound-j216h
up-preflight-p3-j198
backup/macos01-20261106/lume/j251
up-preflight-j198
wip/up-preflight-j198-j245-p2
wip/up-preflight-j198-j245-p1
agent-fsync-offloop-j196
wip/up-preflight-j198-j245
settings-offscreen-j231
wip/up-preflight-j198-j226-inspection
wip/j103f
themes-j206
wip/up-preflight-j198-j226
flakes-j216c
backup/macos01-20261106/claude/j197
flakes-j216b
backup/macos01-20261106/claude/j197-red
wip/up-preflight-j198-review
flakes-j216
backup/macos01-20261106/claude/appwin-icons-j207
engineering-j208b
appwin-mock-j209
toast-stack-j165
engineering-j208
decisions-d351
decisions-d350
decisions-d349
appearance-presets-mock-j129
decisions-d348
docs-audit-spec
doc-coverage-baseline-raise
decisions-d347
decisions-d346b
decisions-d346
j88h-profile
ui-consistency-j195-deferred
ui-consistency-j195
engineering-practices-j194
preferred-login-ui-j166
signin-pools-263-j174
macos-node-265-s1
decisions-d345
decisions-d344
proxy-cgroup-j164
verify-quick-fuzz-329
proxy-start-io-j168
timing-flakes-j163
up-263-s1
mcp-events-j147
verify-quick-fuzz-j158
proxy-p3-j159
agent-resource-j157-detect
wip/j103d-final
remote-cancel-hang-j155
proxy-traefik-281-j62d
j149-verify-quick
proxy-traefik-281-j62b
j150-drop-unnamed
wip/j79d-red-r4
wip/j103d-d332
rootless-toast-j142
wip/j103d-head
wip/j103d
j125c-mcp
macos-build-j136
backup/macos01-20261106/claude/j136
wip/j79d-red-cli
wip/j79d-red-server
j121-link-liveness
j118c-tracer-failures
wip/j79c-head
release-desktop-j131
j124c-signed-macos
wip/j79c-red-p3
wip/j79c-red-p23
wip/j79c-red-p22
merge-j128-check
perf-deferred-j128
sidebar-toggle-260
j127-shortcut-changed-dot
j125-mcp
j118-tracer-failures
j126-account-summary
settings-sections-mock-j92
j120-pinch-lock
sidebar-292
backup/macos01-20261106/claude/j120
taste-2026-11-03
row-actions-242-215
backup/macos01-20261106/claude/stash-0-untracked
backup/macos01-20261106/claude/J80-265-s1-run
backup/macos01-20261106/claude/J80-macos-s1-20261102
j112-canvas-commands
ui-no-overshoot-scroll
fix-recovery-restart
backup/macos01-20261106/claude/stash-1-untracked
taste-2026-11-02
size-baseline-ratchet
machine-cards-structure-j108
j109-boot-copy-mockups
fix-dev-red
wip/up-263-s1
j95b-settings-rows
proxy-traefik-281
j95-option-rows
docs-human-j91
docs-keys-palette
fix-flood-1013-270-j38d
mesh-ssh-s1
mesh-ssh-s0
addhost-d291-tunnel
test-stability-load
mesh-relay-220
repro-296
shim-thp-off-24
mesh-pins-258-v2
mesh-dns
signin-d311-merged
docs-pages
ui-size-283
fix-ui-batch-specs
wip/j12b-edge-update
gate-on-arm
fix-rename-sync-293
trust-285-j60b
fix-focus-ring-clip
integ-ui-1101
edge-update-v2
signin-d311
taste-2026-11-01
layout-per-tab-250
ui-approved-splitopt-224
ui-approved-splitopt-224-on-dev
icon-text-centre-282
ui-approved-189-212-216
research-reverse-proxy
transfer-budget-320
review-mesh-pins-258-r2
sidebar-pill-259
review-mesh-relay-220
wip/j12-edge-update-v2
nix-package
status-crash-loop-237
fix-dev-test-failures
fix-fixture-socket-path
wip/mesh-relay-220-pre-dev-advance-rebase
fix-canvas-side-clip
research-head-to-head
frosted-real-256
review-mesh-pins-258
wip/mesh-relay-220-pre-final-rebase
review-layout-per-tab-250-r2
broad-suite-triage
nix-devshell
review-status-crash-loop-237-r3
wip/nix-devshell-pre-p3
taste-frosted-real
wip/code-serve-demo-2026-10-30
wip/code-serve-glass-2026-10-30
wip/code-serve-clusters-2026-10-30
wip/code-merge-release94-dev-2026-10-30
wip/rproxy-delta-scratch-2026-10-30
wip/stale-note-257-staged-2026-10-30
wip/d241-size-staged-2026-10-30
review-broad-suite-207
wip/mesh-pins-258-pre-split
wip/mesh-relay-220-pre-rebase
mesh-measure-220
plan-211-update
review-edge-autoupdate-r3
perf24-reconnect
review-stale-note-r2
release-prod-live
d241-shell-marker
review-mesh-pins-server-258
review-revocation-r3
review-setup-token-r2
review-sync-journal-r3
wip/review-mesh-pins-258-prune-review-1995fee46
frosted-scroll-256
sync-journal-r3
revocation-per-account-wip
review-tokio-asyncfd-0386
review-sync-p3s
wip/stash-8by64-stash0-2026-10-30
mesh-drag-r2
setup-token-data-wip
release-9-4
auth-busy-1013
mesh-view-overlap
sync-p3s
review-auth-busy-wake
review-readd-r8-wip
mesh-drag-bounds
audit-auth-93
app-windows-stream-start
review-authority-cap-r3
app-windows-r7-measure
readd-minimal-r6
readd-minimal-r5
app-windows-r7-soak
readd-minimal-r4
app-windows-r7
app-windows-r6
readd-minimal-r3
readd-minimal-r2
readd-minimal
merge-release8-dev
review-busy-page-liveness-r2
review-busy-page-liveness
mesh-view-trim
review-authority-read-retry-r2
review-remove-uninstall-r2
release-retry-reattach
release-retry
review-remove-uninstall
review-authority-read-retry
app-windows-ship
mesh-autoswitch-p2
mesh-ssh
mesh-relay
mesh-macos-node
screen-and-control
session-labels
review-real-logins-r5
review-real-logins-r4
review-real-logins-r3
shim-memory-r2
review-real-logins-r2
shim-memory
shadow-addhost-instant
shadow-real-logins
shadow-same-session-flood
shadow-prod-perf-fix
hosts-from-server
app-windows-real
app-windows-transport
release-prod-6w
mesh-policy-verify
quiet-close-toast
dev-merge-r4
review-mesh-agent-transport
review-app-windows-transport
settings-roundtrip
review-release-pipeline-r4
mesh-agent-transport
ended-strip-fix
review-audit-ca-update-d244
settings-rows
phone-width
signout-clear
fix-restart-tests
dev-merge-rp
signer-hardening
review-audit-ca-update
terminal-fidelity
review-audit-ca-update-wip
mesh-integrate
review-mesh-agent-transport-515d3712
review-mesh-dataplane-feec1988
mesh-dataplane
signout
research-canvas-node
wip/review-renderer-cost
wip/review-terminal-fidelity
padlock-glyph
review-release-pipeline-2e95b689
phone-polish
sidebar-polish
inbox-generic
review-mesh-dataplane-378d4055
prod-batch
ui-inbox
ui-ca-unlock
ui-machine-sections
review-mesh-dataplane-a5f744fb
tls-session-prod
session-bounds-prod
fix-dead-session
session-bounds
tls-session
palette-polish
shortcuts-d222
research-appmeta
phone-gutter
ui-centre-button
wip/ui-centre-button
wip/tls-session
review-addhost-breakit
ui-platform
renderer-cost
remote-agents
mesh-api
ssh-bootstrap
prod-hotfix2
prod-fixes
review-remote-agents
review-mesh-dataplane
session-life
research-browser
review-mesh-api
research-capture
review-mesh-api-round3
review-mesh-api-round1
gate-stability
review-gate-stability
review-relay-direct
review-mesh-api-findings
proto-sessions
font-picker
proto-addhost
proto-windows
mesh-core
latency-gate
research-bsd
research-discovery
keytray-mods
mock-core
ui-tweaks
fix-motion-sel
fix-motion
integrate2
fix-robust
t-web
t-server
vt-ghostty
fix-snap-echo
t-web-auth
review-vt2
t-agent
lane-protocol
lane-shim
lane-auth
s1-base
s1-contracts
s1-workspace
No results found.
Labels
Clear labels   
launch-qa
Findings from launch QA.

No labels
launch-qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/infraternal#347
No description provided.