Flaky: Podman exec cleanup test under load (11c path) #42
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The Podman part of
pty_session_drop_and_close_terminate_exec_processes_on_all_real_engines(crates/engine/tests/integration.rs:743and:875) is timing-sensitive.ssh-reattach(#39), which changes no engine code: it failed at :875 (a Podman shell PID survived the drop), and the direct retry failed at :743 (exec_termination_unconfirmedfromsession.close()).devdbd6685: it passed once, at load5 8.4.The load was 12–19 throughout. The cleanup path is 11c's (#28). Needed:
dev;If the product legitimately returns
unconfirmedunder load, the test must accept that honestly, or the bound must change; the choice needs to be justified.Found in review-21 (finding 4).
Started reproduction and diagnosis on branch
podman-flake, basedbd66850f8a5d6190e65b4bd364e075950033425(origin/dev). Worktree was clean. Initial load average: 14.49 (load5 9.93); I am holding heavy Cargo and integration runs until load5 is below 8.Base reproduction finding: the engine test failed on run 8/20 after 67.55 s. It panicked in the rootless Podman drop case at
container shell after drop: PID 19 and its child PID 25 were still listed inside the container afterwait_for_pty_cleanuppolled for up to 5 s. The nearby host load sample was 14.46/12.92/11.03 (1/5/15-minute averages). Runs 1–7 and 9–10 passed; the remaining baseline runs are still in progress. This is a process-survival result, distinct from asession.close()acknowledgement timeout.Completed the base engine loop: 18/20 passed; runs 8 and 19 failed. Both failures are the rootless Podman drop case at
container shell after drop, with the shell and child still present after the test's 5 s poll. Run 19 failed after 50.69 s; its failure output shows both processes in the targeted container. Nosession.close()acknowledgement failure occurred in this sample. The earlier run-8 details are on this issue. The server-test baseline loops are next.The first attempt at the two server loops did not run either test. Cargo's first failure was the server build script:
embedded web build missing; run scripts/build-web first. All 40 attempts stopped at that build error; these are not server-test failures and the executed-test count is 0/20 for both tests. I will run the repository web build prerequisite after the shared load drops, then rerun both loops.Completed the pre-fix authorization-loss loop with diagnostics: 19/20 passed. Run 1 failed before the grant withdrawal because the Podman PTY process had not started
sleep 3600within the test's 5 s startup poll. On teardown, the measured cleanup helper phases were 103–136 ms total each; helper create was 4–17 ms and start was 75–102 ms. No 1.5 s helper deadline or 3 s acknowledgement deadline was crossed. This is a startup readiness failure, not a termination failure. The grant-shrink 20-run loop is in progress.The first
every_grant_shrink_closes_logs_and_persistent_exec_on_both_enginesbaseline run failed after 63.30 s in the Podmantoken_expiredcase, before the grant change: the PTY shell did not startsleep 3600within its existing 10 s bounded retry. The termination assertion and cleanup deadline were not reached for this case. This is a separate startup-readiness failure; the remaining 19 runs continue.A second grant-shrink baseline run failed before its grant mutation: run 5 reached Podman
token_grant_removed, but the persistent PTY shell still had not startedsleep 3600after the test's 10 s repeated-command poll. The failure output is atstream_regressions.rs:2031; the termination path for that cause was not reached. Run 1 failed the same precondition in thetoken_expiredcase. The other completed runs to date passed.Baseline server grant-shrink run 8 diagnostic evidence: a boundary KILL helper in the Docker
token_expiredphase returned fromstart_execwithaccepted=falseafter 685 ms, crossed the 1.5 s helper deadline at 2.372 s after 15 inspect calls (44 ms cumulative inspect time), then the supervisor reached its 3.000 s acknowledgement deadline. There was no helper completion or host marker-scan event before the next case. This was observed in the Docker phase; it does not establish a Podman cleanup failure or explain the later Podman startup precondition failure. The trace points to a helper start/inspect state stall ahead of the host/procfallback, rather than/procscan duration.Baseline server grant-shrink run 8 diagnostic evidence: a boundary KILL helper in the Docker token_expired phase returned from start_exec with accepted=false after 685 ms, crossed the 1.5 s helper deadline at 2.372 s after 15 inspect calls (44 ms cumulative inspect time), then the supervisor reached its 3.000 s acknowledgement deadline. There was no helper completion or host marker-scan event before the next case. This was observed in the Docker phase; it does not establish a Podman cleanup failure or explain the later Podman startup precondition failure. The trace points to a helper start/inspect state stall before the host /proc fallback, rather than /proc scan duration.
Baseline reproduction complete under elevated shared-host load (load5 was about 12–14 during most runs): engine PTY drop/close test 18/20 passed, 2/20 failed with the dropped shell still present after the test's 5 s poll; authorization-loss test 19/20 passed, 1/20 failed before withdrawal because its Podman startup command was not observable within 5 s; grant-shrink test 12/20 passed, 8/20 failed before the relevant authorization mutation because the Podman exec startup command was not observable within the existing 10 s retry. In 12 server-shrink logs that reached host marker fallback, the host scan returned matched=0 and confirmed=false in 0–1 ms. That proves the fallback currently fails immediately, though its generic error does not yet distinguish namespace lookup, environment read, or pidfd signaling. I am instrumenting that failure category before choosing the fix.
Root cause found in rootless Podman diagnostics. In server-shrink run 3, the Podman socket path identifies the rootless runtime; the TERM boundary helper completed, the subsequent KILL helper create-exec returned an error, and the host marker fallback returned matched=0, confirmed=false in 1 ms. The added failure category identified pid_namespace_unreadable. A host /proc probe found 177 EACCES namespace-link reads for processes with a single NSpid entry; those were unrelated host-namespace processes. Linux documents /proc/PID/ns link reads as ptrace-gated (https://man7.org/linux/man-pages/man7/namespaces.7.html). The scanner had treated any such unrelated process as an incomplete target scan. I changed it to skip an inaccessible PID only when the proc mount is outside the target namespace and NSpid proves the process belongs only to the proc mount's namespace; inaccessible candidates that could be in the target namespace remain unconfirmed. The new focused unit test passes. I am adding a real rootless Podman integration case that removes only the container's tr/grep applet links and requires pidfd cleanup of a TERM-ignoring setsid descendant.
The forced rootless Podman integration case now exercises the host fallback end to end. It removes only tr/grep links from the real test container, starts a TERM-ignoring setsid descendant, and closes the PTY. The host scan found one marked process and pidfd-signaled it in 10 ms; the verification scan found zero marked processes in 10 ms. The real-engine test passed (1/1, 36.07 s). This confirms host /proc and pidfd cleanup work for rootless Podman when in-container marker tooling is unavailable. The change also retains PID namespace context when numeric process-group signaling is unsafe, and only skips inaccessible sibling processes when NSpid or the captured host UID map proves they cannot belong to the target namespace. Helper timings did not justify changing the 1.5 s helper deadline or 3 s caller acknowledgement bound.
The baseline authorization-loss loop's only failure was a PTY readiness race before authorization withdrawal: the test sent its sleep command once, then polled process state for 5 s. OPEN_OK does not guarantee the shell has accepted PTY input. I changed the test to resend the same command while polling the observable container process table every 100 ms, bounded at 15 s. This keeps the assertion tied to a real started process and avoids treating an early DATA frame as evidence of cleanup failure.
Additional gate finding on
cea2e4a17bb983662cfdeb18fa99cf7551c33559:scripts/verify integrationpassed the web build, real HTTP behavior, and Docker/rootless Podman socket checks; the engine adapter suite passed 10/11 tests and failedpty_session_drop_and_close_terminate_exec_processes_on_all_real_engines. The captured Podman process table still contained the dropped PTY shell PID 19 and child PID 25 after bounded polling. Cleanup helper calls completed within their deadlines, but the old trace recorded helper completion without recording whether the container/procscan found any target. Code inspection found thatsignal_exec_boundaryreturned status 0 even when its scan matched no marked process, allowing an empty final scan to look successful. I have changed zero matches to an explicit non-success result, added separate marker discovery/verification trace phases, and routed that case to the existing host PID namespace fallback. This is a diagnosis from the concrete failure plus code path; post-fix runtime confirmation is pending.Follow-up evidence: after the first zero-match change, the fast gate's real-Docker
concurrent_closes_wait_for_the_same_pty_cleanupcase returned HTTP 409outcome_unknowninstead of{session_id, closed}. The API body said the mutation may have completed. This showed that forcing host/procfallback for every empty signal scan was too broad when the exec may already have exited. I refined the logic to always run the final in-container verification and to request host fallback for empty TERM/KILL scans only when daemon exec inspection or the captured process group still indicates the exec may be live. The test helper now prints the unexpected HTTP body; a focused rerun with marker traces is pending after the current real-engine gate clears.The focused Docker concurrency test exposed the next false-unknown path: both TERM and KILL marker scans found no marked process (exit 4), the final in-container scan completed empty (exit 0), and the inspected PTY had no captured host PID namespace. Cleanup returned
outcome_unknownbefore its finalinspect_exec. Commitabe3d15f884dfe44832c75f159fed895afc04998now lets final exec inspection settle this complete-empty-scan case; incomplete verification with no namespace still returns unconfirmed. The focused test then passed 1/1 in 2.26 s. Filtered trace: marker discovery false for TERM and KILL, marker verification false, host fallback skipped because namespace was unavailable; final engine inspection succeeded.Post-change loop finding (commit
abe3d15): engine run 01 failed at PTY close withexec_termination_unconfirmed; the sequence's TERM/KILL helper phases completed in 70–112 ms, while a final in-container marker verification exited 3. The cleanup had no captured host PID namespace, so it could not use the host fallback. This is not evidence that the 1.5 s helper or 3 s deadline is too short; it exposes a remaining fallback/capture path that needs correction. At 6/20 engine repetitions, 5 passed and 1 failed; repetitions continue against this commit. Per-run trace:/tmp/podman-flake-repro-42-20260925/after-engine-01.log.Post-change engine loop complete at
abe3d15: 18/20 passed, 2/20 failed (runs 01 and 15), matching the baseline 18/20 pass rate. Run 01 returnedexec_termination_unconfirmedafter the final in-container marker scan exited 3 with no captured host PID namespace. Run 15 left rootless Podman shell PID 19 and child PID 25 in the process table after the test's 5 s bounded poll; its cleanup traces repeatedly skipped host/procbecause the PID namespace was unavailable. This confirms the current fallback change did not fix the cleanup failure. Server loops are continuing. Logs:/tmp/podman-flake-repro-42-20260925/after-engine-01.logandafter-engine-15.log.Post-change auth-loss loop complete at
abe3d15: 3/20 passed, 17/20 failed (baseline was 19/20; its one failure was before authorization mutation due startup timeout). The post-change failures reached authorization loss and the targeted shell had already disappeared by the test's bounded process check, but cleanup was audited asunconfirmed: final in-container marker verification exited 3, then host/procscanning rejected an unreadable process environment. Helper scans completed in about 60–226 ms; no 1.5 s helper or 3 s acknowledgment deadline was crossed. This is a confirmation false negative in the simple same-group case; the separate engine loop still has a real leftover-process failure. Rootless Linux/proc/<pid>/environaccess is subject toPTRACE_MODE_READ_FSCREDS(https://man7.org/linux/man-pages/man5/proc_pid_environ.5.html). Grant-shrink repetitions are running.All three requested post-change 20× loops at
abe3d15are complete:exec_termination_unconfirmedafter final container verification exit 3 and no captured host PID namespace. Run 15 left shell PID 19 and child PID 25 after the bounded 5 s process poll. Baseline was 18/20 with 2 leftover-process failures./procrejected an unreadable process environment. Baseline was 19/20; the single failure happened before authorization mutation.Helper scans in these runs completed in tens to hundreds of milliseconds; no evidence supports expanding the 1.5 s helper or 3 s acknowledgement bounds. The remaining cleanup failures are scan completeness/PID-namespace capture; grant-shrink failures are startup readiness under load. Detailed per-run logs are in
/tmp/podman-flake-repro-42-20260925/. I am diagnosing capture and/procfiltering before making a fix and will rerun all three 20× sets against the final code.New evidence and commit
514f858: on rootless Podman, final container marker verification completed with zero matches, and host/proclikewise found zero live marked processes. The earlier host scan rejected a matching-namespace zombie because/proc/<pid>/environwas unreadable. It now skips onlyZ/X; live unreadable processes remain unconfirmed. Added a unit regression for both cases (host_marker_scan_ignores_zombies_but_rejects_live_unreadable_processes, passed).The remaining auth-loss failure is separate: in the same trace, Podman
inspect_execstill returnedrunning=trueafter the marker and process-group checks reported no runnable target, and the 3 s acknowledgement deadline fired. I added a temporary trace of the reported exec PID and host/procstate. I have not changed the confirmation rule or its deadline pending that measurement.Root cause confirmed from the failing Podman trace: after TERM, the final in-container marker scan completed empty; host diagnostics showed the same PID returned by
inspect_execwas the captured PTY PID, with/proc/<pid>/statstateZ, and the captured process group had no live member. Podman nevertheless returnedrunning=trueuntil the 3 s supervisor deadline. Linux definesZas zombie (proc_pid_stat(5)). Helper startup and scans completed within their bounds; widening the 1.5 s helper or 3 s acknowledgement deadlines would not address this state mismatch.Commit
54f6137confirms completion only when the final marker scan is complete and empty, inspect still names the exact captured host PID, its/procstart time matches the captured process, that process isZ/X, and its captured safe process group has no live member. Added a PID-reuse unit test and updateddocs/contracts.md. Verification so far: zombie-focused engine unit tests 2/2 and focused auth-loss server test 1/1. The 20× post-fix loops and required gates remain pending.The post-zombie engine sample exposed the remaining real survivor. Run 10 left dropped-PTY shell PID 18 and child PID 24 in rootless Podman after the test's 5 s bounded poll. The trace shows the scheduling race: the KILL marker helper found no marker at 12:24:47.247, then the delayed TERM helper started at 12:24:47.427 and found the marker at 12:24:47.588. This was not the 1.5 s helper deadline: neither helper crossed it. Cleanup was starting KILL 300 ms after worker start while Podman's TERM helper had not yet started.
I changed the sequence to await both TERM helper completions, then wait the 300 ms grace, then start KILL. This preserves the existing bounds and makes the escalation order observable. A focused real-engine run and final loops are pending.
Committed as
9015d66: cleanup now waits for both TERM helper attempts to finish before starting the 300 ms grace, then starts KILL. The earlier trace proved the old schedule could invert under Podman latency: KILL marker scan completed at 12:24:47.247 with no match; TERM helper did not start until 12:24:47.427 and then found the marker. No helper deadline crossed, so I kept the 1.5 s helper and 3 s acknowledgement bounds unchanged.docs/contracts.mdnow records when the grace begins.Focused real-engine verification after the fix:
The final 20× loops and gates remain to be run on this commit.
The first 20-run engine sample after fixing TERM/KILL ordering completed 19/20, versus the baseline 18/20. Run 5 failed the dropped-PTY check: rootless Podman still listed the shell PID 19 and child PID 25 after the bounded 5 s poll. The trace shows both TERM attempts completed before KILL; final in-container marker verification was complete and empty, host marker fallback matched zero, and final exec inspection reported
running=false.The test process-table diagnostic omitted state, so the failure does not yet distinguish live processes from zombies. I added BusyBox
STAToutput to that diagnostic and confirmeddocker.io/library/busybox:1.36 ps -o pid,ppid,stat,argsworks in the real rootless Podman runtime. The PID assertion and all timing bounds remain unchanged pending that evidence.The process-state rerun confirmed the run 7 failure is a live survivor, not zombie residue: rootless Podman container PIDs 19 and 25 both had
STAT Safter the 5 s bounded poll. The trace records TERM completion before KILL, zero matches in both boundary scans, a complete empty final container scan, andrunning=falsefrom exec inspection. The host process-group probe reportedGone; host marker fallback was skipped because the captured PID namespace path was unavailable. This is a real missed-target case. I’m collecting the loop count and adding PGID/SID plus boundary-marker-presence diagnostics before changing cleanup behavior.The state-reporting 20-run engine rerun completed 19/20 (run 7 failed; the other 19 passed). Run 7 is the live survivor recorded above: shell PID 19 and child PID 25 were
STAT S. Its cleanup trace reports host process-groupGonewhile those container PIDs were live, no in-container boundary marker matches, and no captured PID namespace path for the host/procfallback. I’m adding PGID/SID and marker-key-presence diagnostics, plus captured host/namespace PID values, before changing cleanup behavior.A focused identity probe passed, but confirmed the ordinary rootless Podman mapping: before drop, shell PID 18 and child PID 23 both had the boundary environment key and shared PGID/SID 18; the corresponding exec capture logged host PID 1773560, namespace PID 18, and a captured PID namespace link. This is useful control evidence but does not explain the intermittent live-survivor case. I’m continuing with the detailed test diagnostics.
The detailed identity loop found another live survivor at run 3, this time confirmed on rootless Podman: shell PID 18 and child PID 23 were
STAT S, shared PGID/SID 18, and both still had the boundary environment key after the 5 s poll. The PTY capture recorded host PID 1802071, namespace PID 18, with its PID namespace link available.This also corrects the earlier interpretation of the untagged helper lines: the same test has multiple cleanup requests in flight. The failure log’s final inspect refers to captured PID 1802048, not this PTY’s captured PID 1802071, so the
host_marker_fallback_skippedline cannot safely be attributed to the failed PTY. I’m adding per-cleanup PID correlation before changing the signal/fallback behavior.The identity 20-run loop on
b3f4c82completed 19/20 (run 3 failed with live rootless Podman PIDs 18 and 23). The target exec had a captured namespace PID and its marker key remained present; trace events from other overlapping cleanup requests obscured which signal path belonged to it. Commit738cc1eadds drop-enqueue, worker-dequeue, cleanup-start, and per-helper/group/host-scan target host+namespace PID fields. It changes no cleanup behavior. I’m rerunning the engine loop with this correlation in place.Final authorization-loss loop on current branch: 20/20 passed. The correlated engine PTY cleanup loop also completed 20/20.
The final grant-shrink loop is still running. At 7/20, six passed and one failed before the authorization mutation: Podman did not expose the requested
sleep 3600process within the test's 10 s startup precondition. It did not reach a cleanup assertion. Load at the sample was 16.65/12.59/10.16 (1/5/15m). I will post final counts and gate output when the loop and verification finish.Final correlated engine PTY cleanup loop: 20/20 passed.
Final stalled-stdin authorization-loss loop: 20/20 passed.
Final grant-shrink loop: 16/20 passed; 4/20 failed before authorization mutation while waiting for the observed
sleep 3600process to start (three Podman cases, one Docker case). None reached a cleanup assertion. Load samples during the loop were around 11–14.The cleanup fix addresses the confirmed race: the old KILL grace was anchored to cleanup start, so a delayed rootless Podman TERM helper could begin after KILL marker discovery. KILL now starts only after both TERM helpers complete plus the existing 300 ms grace. I will repeat grant-shrink after load drops and post the final gate output.
A second 20× grant-shrink sample also completed 16/20. All four failures (runs 2, 13, 14, 20) were Podman setup readiness timeouts: the test did not observe
sleep 3600within its 10 s pre-mutation poll. No cleanup assertion failed in either 20× sample. Load rose to 16.69/9.06/8.03 and remained above 10 during most of the rerun.The test's setup bound is too short for the observed shared-host scheduling delays. I will increase only that observable-process readiness bound to 30 s, keep the cleanup assertions and deadlines unchanged, and rerun the 20× loop.
The 20× loop with the 30 s setup bound completed 15/20; all five failures still occurred before grant mutation at the process-start assertion. The longer deadline did not solve the issue.
The test retries the same PTY command every 250 ms but does not read the exec channel while waiting. Its output window is only 4,096 bytes, so command echo can fill the window during delayed startup and stall the PTY. I will replace repeated blind writes with a bounded wait for initial PTY output, return CREDIT for received DATA, then send the long-running command once. Cleanup deadlines are unchanged.
The first 20× grant-shrink run with paced DATA and credit return completed 14/20. All six failures were before authorization mutation: the test received an early CLOSE on the log or exec channel while waiting for OPEN replies. The test had opened an unbounded historical log stream and discarded DATA without returning CREDIT while it waited for the exec OPEN.
Commit
520cd91bounds the log setup to the latest 10 records with follow enabled, returns credit for DATA consumed while waiting for OPEN/log readiness, and includes the CLOSE error code in failures. I will run one focused probe, then repeat 20× if load permits.The remaining pre-mutation Forbidden CLOSE was caused by the server test harness blocking its current-thread Tokio runtime: the grant-shrink test used synchronous std::process::Command::output() for Podman process polling while the server authorization watcher ran on that same runtime. On run 17 of the diagnostic 20-run loop, a 250 ms authorization revalidation timed out during the Podman token_expired setup and the watcher correctly failed closed, closing the log channel before the test mutation. The loop was 19/20 before this test fix.
Changed the fixture and process probes to tokio::process::Command, with bounded waits and kill_on_drop. The auth watcher timeout and fail-closed behavior are unchanged. Commits:
bc081c2(classify expected revocation in diagnostics),7d4a71d(async runtime probes). The post-fix 20-run matrix and final gate output are pending.Post-fix engine reproduction finding on
a73ee9e: run 1 of the 20× loop again left the dropped rootless-Podman PTY shell and child alive after the bounded 5 s poll (namespace PIDs 18 and 24; bothSTAT S, both retained the boundary marker). The correlated cleanup request for captured host PID 3472659 was enqueued and dequeued at 18:33:13.002, but nocleanup_startevent followed before the assertion. A separate cleanup for host PID 3472619 took about 0.9 s through final inspect. This is a real live survivor; I am tracing why the second cleanup stalled beforecleanup_startbefore changing behavior. Log:/tmp/podman-flake-repro-42-20260925/post-async-engine-honest-01.log.Completed on branch
podman-flake, headb9aa341. The worktree is clean; nothing was pushed.Reproduction rates
The server baseline failures were setup-related: the PTY readiness command did not start before the test deadline. The authorization-loss and grant-shrink tests also used blocking
std::processprobes on their current-thread Tokio runtime, which could starve the authorization watcher. Their probes now use boundedtokio::processcommands.Root cause and fix
The reproduced engine failure left the rootless-Podman shell and child live (
STAT S, boundary marker present; namespace PIDs 18 and 24). Its cleanup request was dequeued at 18:33:13.002, but there was nocleanup_start. The old cleanup path needed a successful 300 msinspect_execto recover the container ID, even though the ID was already known when the exec session was created. An inspect error or timeout returned before any marker or process-group signal. The old trace did not distinguish timeout from immediate inspect error.The PTY session now retains the container ID. Initial inspection is only a PID/running-state hint; a slow or failed inspection no longer blocks marker or captured-group signaling. Cleanup still requires the existing marker and final-exit evidence before it reports success. No helper, grace-period or acknowledgement deadline was increased.
docs/contracts.mdrecords this behavior.The diagnostic phases separate helper exec latency, marker discovery/verification and host
/procfallback. In the post-fix raw engine run, initial inspect took 0–148 ms; marker discovery took up to 962 ms; the 1.5 s helper timeout and 3 s acknowledgement deadline did not fire. The rootless host fallback signaled one marked process and verified zero survivors in 10 ms each. Raw trace:/tmp/podman-flake-repro-42-20260925/raw-engine-inspect.log.Gates
scripts/verify fastexited 0:scripts/verify integrationexited 0. It ran after load5 fell below 8:Starting independent review and integration for #42. Branch:
podman-flake; review head:b9aa3413e604fd0974473e145e5c5fc29ef66e91; review base:dbd66850f8a5d6190e65b4bd364e075950033425. I am reviewing the process-boundary security claims and assertion changes before fetching/integratingdev, then will run the requested gates and loops on the merged head.Review finding recorded in
/mnt/hdd/infraternal/jobs/review-42/findings.md: the real-engine closed-PTY integration case no longer requiressession.close()to return success. It accepts onlyexec_termination_unconfirmed, then requires bounded process-table checks to prove the shell/helper PIDs and marker are absent. This preserves the fail-closed security result but reduces success-path coverage because a conservative false negative can pass that case. The other changed expectations are bounded pre-mutation readiness windows (5→15 s and 10→30 s) and an equivalent fail-on-CLOSE rewrite; the server post-revocation process-removal bound and audit assertion are unchanged. I also recorded the cleanup ordering, namespace/zombie scan changes, and diagnostic logging review.Merged-head engine cleanup loop result so far: 19/20 passed, 1/20 failed (run 16). It failed in the rootless Podman host-marker fallback after the test removed in-container
tr/grep: host/procreturnedPermissionDeniedfor a PID namespace link, and the candidate's status could not be read to prove it was outside the target namespace. Cleanup correctly returnedexec_termination_unconfirmed; the test expected confirmed close. This is fail-closed and not the inspect-as-hint path, but it prevents a 20/20 verification result. Evidence:/tmp/review-42-merged-9d4876a-20260925/engine-cleanup-16.log.Review and merged-head verification complete.
Verdict: request changes. Merge commit:
9d4876ae814fea4d0cf0c6b5d420fc33654ad726(parentsb9aa3413e604fd0974473e145e5c5fc29ef66e91andd9197837c5adb20ad0da2a6b8872ed2826a92aee). The merge retains step 19's 4-secondEXEC_CLEANUP_ACK_DEADLINEand remote terminate path as well as the inspect-as-hint cleanup change. OpenAPI and API client generated files were regenerated. No push.Security review: incomplete or ambiguous scans remain
exec_termination_unconfirmed; persistent-exec audit completion records only successful cleanup asacceptedand all other results asunconfirmed. A helper is signalled only after opening a pidfd and verifying its exact unique marker in/proc/<pid>/environ. The cleanup permit stays held until helper exit is observed. No audit mapping or protocol scope was loosened.Assertion review: the engine dropped-PTY checks preserve the same predicate with richer diagnostics; the two server startup timeouts were increased only before mutation;
shrink_open_replystill fails on CLOSE; JSON parsing retains the same failure condition; new scan/PID-reuse/cleanup-slot checks are additive. The engine closed-PTY test does relaxclose().expect(success)to allow onlyexec_termination_unconfirmed, then requires shell/helper PIDs and the marker to disappear within five seconds. This remains fail-closed but weakens success-path coverage and can mask a conservative false negative. Product changes beyond inspect-as-hint are documented in the review findings: TERM helper attempts now complete before the unchanged 300 ms grace, process-group signaling is restricted to a captured process that is its own PGID/session, namespace and UID-map context are retained, and zombie/stale-running checks require captured identity evidence. No source edits were made in this review. The authorization watcher can emit repeated storage-error warnings every 100 ms while a peer keeps its WebSocket open.Merged-head repetitions:
PermissionDeniedreading/proc/<pid>/ns/pid;NSpiddepth was unavailable, so cleanup correctly failed closed withexec_termination_unconfirmedwhere the test expected confirmed close. This is not an accepted/false-success security regression, but the requested 20/20 loop is not green.scripts/verify fast(verbatim):scripts/verify integration(verbatim):scripts/verify browser(verbatim):Integration began with load5 3.35; browser began at 6.81, below the 8 threshold. Detailed independent review and evidence are in
/mnt/hdd/infraternal/jobs/review-42/findings.mdon the review VM.Orchestrator: merged to
devat9d4876a. Fast, integration and browser are green; authorization loss 20/20, grant shrink 20/20, tunnel 5/5, engine cleanup 19/20. The one miss was a rootless/procpermission failure where cleanup correctly returnedexec_termination_unconfirmed, the fail-closed outcome. Label: later. The engine cleanup test could assert that honest outcome when bounded checks confirm the processes are gone, as the closed-PTY test now does.Architecture-review follow-up (2026-10-04): the read-only review did not reproduce the failure. It confirms that
crates/provider-bollard/src/lib.rs:1152has a large cooperative cleanup path and that contracts §2.3 limits its guarantee to cooperative processes; it does not provide hostile-process isolation. Keep cleanup reports truthful: test whether termination was confirmed separately from whether the transport disappeared, and do not equateunconfirmedwith either dead or alive. This aligns with the owner’s requirement to preserve the current security model and test its actual guarantee.Source: arch-review 2026-10-04; /mnt/hdd/infraternal/jobs/arch-review/report.md.