Step 11c: per-exec process boundary and remaining withdrawal gaps #28
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Carried over from step 11b (#22) after its one fix round (
ce77555). 11b fixed dispatch fencing, remove-and-restore, CLOSE ordering and reaping of TERM-ignoring children in the process group. These remain, and each existed before 11b:setsidchildren leave the exec's process group, so a group signal misses them. This needs a per-exec process boundary (a cgroup per exec, or an engine-side kill of the whole exec tree). Research the options for Docker and rootless Podman and cite them.crates/agent's control path.docs/contracts.md§2.3 and close the window.Security surface: Sol for the build and the review. Reproducers are in
/mnt/hdd/infraternal/artifacts/review-grant-shrink/.Starting step 11c on branch
process-boundaryat base1337efde31f52d0c717029c1760ff7d3c7eb2940. Reading issue #28 and its reproducers, then implementing and testing the four withdrawal gaps. No push.Step 11c findings: Docker's exec create/start interface has no per-exec cgroup placement field (https://docs.docker.com/reference/api/engine/). Linux cgroup migration requires write access to both the destination and the common ancestor's
cgroup.procs;cgroup.killis the kernel-enforced option (https://cdn.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html). On this VM the rootless Podman scope is owned bykayg, while the rootful Docker scope is owned by root, so a host-side cgroup move is not portable to the unprivileged infraternal agent. The current branch tags each exec environment and scans/procfrom a helper inside its container. A new real-engine test reaped bothsetsidand double-fork descendants on rootful Docker and rootless Podman (1 passed, 0 failed, 14.22s). This is cooperative: a child can remove the marker or deny/procinspection. The DATA-after-commit guarantee still needs a commit-synchronized write fence; watcher polling cannot provide it.The
/procboundary's limit is confirmed by the Linuxproc_pid_environ(5)manual: it exposes the environment supplied atexecveand access is subject toPTRACE_MODE_READ_FSCREDS(https://man7.org/linux/man-pages/man5/proc_pid_environ.5.html). Therefore a descendant that starts a new program with a scrubbed environment, or cannot be inspected under the container's proc policy, is outside this cooperative tracker. The real Docker/rootless Podman fixture proves ordinary inherited-environmentsetsidand double-fork descendants are reaped; it is not proof against hostile scrubbing.DATA fence implementation: the server reserves bounded relay capacity, then
SqliteStorage::admit_stream_datausesBEGIN IMMEDIATEto compare both attachment epochs and enqueue the frame synchronously beforeCOMMIT. Grant mutations use SQLite's same writer lock. Therefore a DATA frame sent after a withdrawal transaction commits cannot pass the old-epoch check, including when the watcher has not polled yet. A frame enqueued before commit may still drain to the process afterward; the updated §2.3 states this exact limit. Added a concurrent commit/admission test, a Docker/rootless Podman continuous-writer process test, and a Criterion benchmark for the input fence. These new tests are pending verification while the integration build is running.Integration finding on
f852f36:stalled_pty_stdin_cannot_hold_exec_terminationpassed on Docker (397 ms) but rootless Podman returnedexec_signal_timeoutfrom the cleanup helper after 244 s of engine tests. The added per-exec/prochelper and original-group helper are currently serialized, and helper create/start has a 750 ms deadline. I am making KILL helpers concurrent and allowing more startup time, then will rerun the targeted regression and both required gates.Cleanup helper correction committed as
c8522e0. The Podman failure was two races: a 250 ms helper-inspection timeout under rootless Podman load, and the marker helper reaping a process group before the concurrent group helper signaled it. Helpers now have a bounded retry window for inspection; TERM/KILL group and marker helpers run without making one wait for the other; signaling an already vanished group succeeds. Targeted real-enginestalled_pty_stdin_cannot_hold_exec_terminationpassed: Docker 470 ms, rootless Podman 1.354 s; 1 passed, 0 failed. Full final-head gates are running.Final-head integration run on
c8522e0passed the real engine suite, log snapshots, TypeScript stream client, and the newclient_data_written_after_grant_commit_never_reaches_execplusfailed_dispatch_commit_never_leaves_a_persistent_exectests. It then failed the existingevery_grant_shrink_closes_logs_and_persistent_exec_on_both_enginestest because the Podmantag_changecase did not see its shell command start within the 10 s test deadline. A direct rerun of that full regression passed all 16 Docker/Podman causes in 47.63 s, including Podmantag_change(CLOSE 19 ms, process reaped 725 ms). No source change was made; the full integration gate is being rerun to verify the combined suite.Second final-head integration run passed the real engine and stream regression sections, including the new 11c cases, then timed out in the existing compiled CLI/MCP SSH replay PTY harness after the remote command printed its expected BEFORE/AFTER output. A direct rerun of the complete
cli_and_mcp_use_the_public_http_contracttest passed (122.23 s, 1 passed, 0 failed). No source change was made. A third full integration gate began with load5 3.03; the first full run's older Podman grant-shrink test also passed in this second run.Third full integration run again passed the 11c engine and stream sections, then timed out in the existing compiled CLI/MCP SSH PTY harness. This time the initial SSH command printed
41 111and__CLI_SSH_COMMAND_OK__, then echoedexit 37, but the CLI process did not exit within the harness's 15 s wait. The direct complete CLI/MCP test passed earlier on this same head. The failure is intermittent and occurs after remote input is delivered; I am checking the SSH close path before deciding whether another gate run is warranted. No code change has been made.While investigating the intermittent full-gate timeouts, I found a cancellation safety issue in the new inbound DATA fence: it used raw
BEGIN IMMEDIATEon a pooled SQLx connection. A cancelled async admission could leave transaction cleanup dependent on pool behavior. I changed it to SQLxConnection::begin_with("BEGIN IMMEDIATE"), whose transaction object starts rollback on drop (SQLx Connection, SQLx Transaction). The injected interruption regression passes and verifies a grant write can proceed afterward. New head2a79eeb; final gates are being rerun. This safeguard may or may not explain the earlier timing failures; no causal claim yet.Step 11c finished on branch
process-boundaryat head2a79eeb65ec8982609be9da7859a2283d45e3783(base1337efde31f52d0c717029c1760ff7d3c7eb2940). No push; working tree clean.Boundary choice: each exec receives a unique inherited environment marker. Cleanup scans
/proc/*/environinside the container PID namespace and signals matching PIDs, includingsetsidand double-fork descendants; it also signals the original process group. This works without host cgroup delegation under rootless Podman and was exercised on real rootful Docker and rootless Podman. Docker's exec API does not offer a per-exec cgroup, and a host cgroup move for rootful Docker requires delegation the unprivileged server does not have. Limits: a process can scrub its marker, deny/procinspection, or fork during the final scan; the image needs/bin/sh,tr, andgrep. This is cooperative cleanup, not kernel containment. Sources: Docker Exec API, Linux proc_pid_environ(5), Linux cgroup v2.Other fixes: nonpersistent PTY writes have a 250 ms bound so cleanup cannot wait indefinitely; failed dispatch commits fence late persistent exec/SSH creation and terminate any already reserved session; inbound DATA enqueue is serialized with grant commits under an immediate SQLite transaction. SQLx owns rollback on interrupted admission. Exact ingress guarantee: DATA sent after the withdrawal commit is not forwarded; data admitted before the commit may already be buffered and can be consumed afterward.
docs/contracts.md§2.3 states this distinction. The 11b forbidden CLOSE bound remains covered by real-engine tests. Added a Criterion benchmark for the admission path; performance numbers are reserved for the perf-test VM under the repo contract.Targeted evidence: real Docker/rootless Podman descendant test passed; stalled PTY cleanup test passed (Docker 470 ms, Podman 1.354 s); failed commit and continuous post-commit DATA tests passed on both engines; injected interrupted DATA admission released the grant write lock. Earlier full integration runs exposed intermittent pre-existing Podman process-start and CLI SSH PTY timing failures. Direct reruns of those suites passed, and the final full integration run below passed without a source or test-timeout change to those paths. Their failed outputs and analysis are in prior issue comments.
scripts/verify fastoutput (verbatim):scripts/verify integrationoutput (verbatim):Review-11c started on branch
process-boundary, base1337efd, head2a79eeb65ec8982609be9da7859a2283d45e3783. Independent read-only defensive audit; targeted tests pending.Independent review-11c complete. Branch
process-boundary, base1337efd, head2a79eeb65ec8982609be9da7859a2283d45e3783. Verdict: request changes. Full notes:/mnt/hdd/infraternal/jobs/review-11c/findings.md.Findings:
crates/engine/src/lib.rs:943-972,1015-1031): if the marker helper cannot scan/procor the image lackstr/grep, detachedsetsiddescendants remain outside the original group. The code reports an error, but the requested descendant-termination property is not met for that configuration. Gate exec creation on a usable scanner or provide another boundary; qualify any guarantee accordingly.crates/engine/src/lib.rs:1136-1189): a helper can be created or started before its request times out, then continue after the cleanup future releases its 256-slot permit. Repeated timeout cycles can accumulate detached helper processes. Retain and supervise helper IDs until confirmed exited.crates/engine/src/lib.rs:946-968,1041-1106): scan failure does not always escalate to the original group. Group capture and the host/procliveness check are prerequisites; unreadable host proc or a non-group-leading exec skips it. Narrow the claim and test this failure path.The cooperative marker limitation is documented substantially honestly.
/proc/<pid>/environshows the initial environment for the current exec image and has ptrace visibility checks (https://man7.org/linux/man-pages/man5/proc_pid_environ.5.html). Docker and Podman process listings alone do not establish exec ancestry (https://docs.docker.com/reference/cli/docker/container/top/ ; https://docs.podman.io/en/latest/markdown/podman-top.1.html). A per-exec cgroup is stronger but not a cheap universal substitute because cgroup migration requires delegated write access (https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html).Checks run, real output:
cargo test -p infraternal-engine --test integration pty_close_reaps_setsid_and_double_fork_descendants -- --ignored:test pty_close_reaps_setsid_and_double_fork_descendants ... ok/test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 9 filtered out; finished in 16.19scargo test -p infraternal-agent stalled_nonpersistent_pty_input_does_not_hold_cleanup --lib -- --ignored:test tests::stalled_nonpersistent_pty_input_does_not_hold_cleanup ... ok/test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 9 filtered out; finished in 5.64scargo test -p infraternal-server --test storage continuous_stream_data_is_fenced_at_revocation_commit -- --exact:test continuous_stream_data_is_fenced_at_revocation_commit ... ok/test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 20 filtered out; finished in 0.21scargo test -p infraternal-server --test storage interrupted_stream_data_admission_releases_grant_write_lock:test interrupted_stream_data_admission_releases_grant_write_lock ... ok/test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 20 filtered out; finished in 0.05scargo test -p infraternal-server --test stream_regressions failed_dispatch_commit_never_leaves_a_persistent_exec -- --ignored:test failed_dispatch_commit_never_leaves_a_persistent_exec ... ok/test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 18 filtered out; finished in 15.86sThrowaway mutation checks: disabling
PersistentExecSessions::reservecancellation made its new unit test fail; disabling theadmit_stream_dataepoch check made the continuous revocation test fail with 500 post-commit admissions. A focused throwaway test deletingcontainers:execrole grant passed: existing migration0010_authorization_epochs.sqlbumps user and token epochs on DELETE, so DATA admission correctly rejected it. The throwaway worktree was removed. The review worktree is clean. No commit or push was made; issue remains open.Fix round started on branch
process-boundaryat base2a79eeb65ec8982609be9da7859a2283d45e3783. I am addressing audit Findings 1–3: explicit unconfirmed termination outcomes, owned helper lifecycle, and host/procdependency. DATA fencing is unchanged. I will post regression sensitivity and final gate output here.Fix round evidence on
process-boundary(current head2c6e335):/procscan scoped by the captured PID namespace and uses pidfds to signal matching processes. An incomplete scan returnsoutcome_unknown, and persistent-session completion audits now sayunconfirmed. In a real rootless Podman container withtrandgrepunavailable and restricted host/proc, the new regression observes an unconfirmed result instead of success. The existing real Docker/Podmansetsidand double-fork regression passes./procinjection returnsUnknownrather thanGone; the unit regression passes.The fast gate passed all 15 checks on this head. Regression sensitivity checks and final integration gate remain.
Gate note: the first integration run (started before the later helper-identity commits) passed the release build, real HTTP, Docker/Podman sockets, engine adapters, log snapshots, TypeScript stream, stream authorization, persistent PTY blocked-send, and resource HTTP checks. It failed the compiled CLI/MCP SSH PTY harness: the CLI printed
__CLI_SSH_COMMAND_OK__andexitbut did not exit within the harness deadline. The direct one-time rerun ofcli_and_mcp_use_the_public_http_contractpassed (1 passed, 126.15s). Full failing and passing outputs are retained in/tmp/fix-11c-integration.logand/tmp/fix-11c-cli-mcp-rerun.log; final-head fast and integration are being run separately.Regression sensitivity confirmed on the fix branch. I reversed one fix at a time locally, ran only its regression, then restored the committed code and reran the unit suite:
unknown_outcomeconstructor with an ordinary engine error made the real rootless Podman missing-tools test fail:incomplete scan must be explicit ... outcome_unknown: false(log/tmp/fix-11c-regression-1.log).left: 1 right: 0available permits (log/tmp/fix-11c-regression-2.log)./procasGonemade the injected-proc test fail:left: Gone right: Unknown(log/tmp/fix-11c-regression-3.log).All three reversions were restored. Final-head
scripts/verify fastpassed all 15 checks at15cd8694140d5347670bfab70c4da6d12a7b5775(full gate output will be quoted in the final comment). The final integration run started with load5=4.57.Fix round complete on
process-boundaryat head15cd8694140d5347670bfab70c4da6d12a7b5775. No push; working tree clean. DATA fencing was left unchanged.Finding 1 — fixed within the documented cooperative boundary. Cleanup now runs a final post-KILL verification scan. A failed/incomplete container scan uses a host
/procPID-namespace marker fallback when visibility allows; signals use pidfds. The real rootless Podman missing-tools regression confirms an explicitoutcome_unknownwhen host/procis restricted. Persistent-session completion audits recordunconfirmed, and the caller receives an error. The existing real Docker/Podman setsid/double-fork test passes. A marker removed by a hostile process or a fork after verification remains outside the guarantee, as documented.Finding 2 — fixed. A cleanup helper ID is retained immediately after creation, the helper is supervised through start and inspect, and a timeout attempts host pidfd KILL only after checking its unique marker. The exec cleanup slot remains held until an exit code is observed, even after the caller receives an unconfirmed result. The timeout/slot regression passes. Docker's exec API does not expose an exec-delete endpoint, so a helper ID cannot be explicitly removed; the implementation confirms the process exited before releasing capacity.
Finding 3 — fixed/documented. Original-group escalation is limited to groups captured from readable host
/procand verifiably live. An injected missing host/proctest returnsUnknown, notGone, and prevents a false success. The contract states this conditional guarantee.Regression sensitivity was proved by reversing each fix locally, one at a time: missing-tools test failed with
outcome_unknown: false; supervisor test failed with available permitsleft: 1 right: 0; missing-proc test failed withleft: Gone right: Unknown. Restoring the code made all pass. The pidfd test was also synchronized with its owned child environment after an intermittent fast-gate failure; final fast passed.Final-head
scripts/verify fastoutput (verbatim):Final-head
scripts/verify integrationoutput (verbatim):The final integration run began at load5=4.57; load5 rose above 12 during the CLI/MCP SSH timing check. That check was rerun directly once after load dropped. Direct rerun output (verbatim):
The earlier integration run on the pre-safety-commit head also failed the CLI/MCP SSH exit-wait harness after 12 PASS checks; its one direct rerun passed. Those exact logs remain in
/tmp/fix-11c-integration.logand/tmp/fix-11c-cli-mcp-rerun.log, and that flake was noted above.Verification started for issue #28. Branch:
process-boundary, current head15cd8694140d5347670bfab70c4da6d12a7b5775, based on1337efd. I will mergedevatce194d4as the only permitted repository commit, verify Findings 1–3 (including throwaway reversion worktrees), then run the requested gates when load permits.Regression sensitivity results, isolated in
/home/kayg/wt/verify-11c-tmpat15cd869; all temporary edits are confined to that worktree.cleanup_tests::host_marker_scan_detects_survivors_and_reports_incomplete_inspectionpasses at the fix head. Revertingtermination_unconfirmedfromEngineError::unknown_outcometo an ordinary engine error fails withassertion failed: error.outcome_unknown.host_marker_scan_signals_only_the_marked_pid_through_pidfdpassed with the live child carrying a different marker from the synthetic/procentry. Removing the marker read/check afterpidfd_openfails witha reused PID without the marker must not be signaled: 1.cleanup_tests::timed_out_cleanup_reports_unknown_but_retains_its_slot_until_exitpasses at the fix head. Dropping the permit in the timeout branch fails withleft: 1 right: 0athelper still owns cleanup capacity.I will remove the throwaway worktree after the remaining checks.
Verdict: verified for Findings 1–3 on merge head
dbd66850f8a5d6190e65b4bd364e075950033425.origin/devatce194d4bf53bf370a6bcccb76745c74c74240494merged cleanly; this is the only commit on the branch after15cd869. No push.Finding 1 — incomplete scans do not report successful termination.
crates/engine/src/lib.rs::terminate_execreturnstermination_unconfirmedif the fallback scan or final verification is incomplete. The persistent-session completion paths writeunconfirmedincrates/server/src/personal.rsandcrates/server/src/stream.rs. The unit testhost_marker_scan_detects_survivors_and_reports_incomplete_inspectionpasses; when I revertedtermination_unconfirmedto an ordinary engine error in the throwaway worktree, it failed withassertion failed: error.outcome_unknown. The real Podmanmissing_container_scan_tools_report_unconfirmed_when_host_proc_is_restrictedregression also ran in the integration gate.Finding 2 — helper timeout signaling is identity checked, and cleanup capacity remains owned through observed exit.
run_exec_helpersupplies a unique helper marker;signal_host_processopens a pidfd, checks/proc/<pid>/environfor that marker, then sends through the pidfd. Linux documents the pidfd as a stable reference to the selected process, avoiding signals to a different process after PID reuse: https://man7.org/linux/man-pages/man2/pidfd_send_signal.2.html. A temporary strengthenedhost_marker_scan_signals_only_the_marked_pid_through_pidfdtest passed with a mismatched live-process marker; removing the post-pidfd marker check made it fail witha reused PID without the marker must not be signaled: 1.timed_out_cleanup_reports_unknown_but_retains_its_slot_until_exitpassed; explicitly dropping the permit on timeout made it fail withleft: 1 right: 0athelper still owns cleanup capacity.Finding 3 — missing host
/procdoes not prove the original group is gone.host_exec_group_status_atreturnsUnknownwhen/proccannot be read.missing_host_proc_is_unknown_not_a_dead_original_grouppassed; reverting that branch toGonefailed withleft: Gone right: Unknown.Audit presentation check: the server records the outcome as the literal
unconfirmed. There is no current audit read endpoint or CLI audit command. The web/audit-logroute is a placeholder, and the dashboard says activity will appear when the audit-log API lands. So there is no API, CLI, or web audit query/render path on this head to mis-map the value; presentation is not yet implemented.scripts/verify fastfirst failed onlysteady_state_framing_does_not_allocate_per_chunkat load56.23(load at 06:11:56 UTC:load average: 7.44, 6.23, 6.40). Verbatim failure excerpt:Direct rerun, verbatim:
A second full fast run passed. Verbatim output:
scripts/verify integrationstarted with load52.95and passed. Verbatim output: