Step 11b: prove the one-second grant-shrink bound #22

Open

opened 2026-09-23 07:13:50 +00:00 by kayg · 20 comments

kayg commented 2026-09-23 07:13:50 +00:00

Owner

Copy link

From the step 15 verify (#11, item 3). docs/contracts.md §2.3 says that when effective grants shrink, the server, within one second, bumps the authorization epochs and closes every channel and attachment no longer covered, with CLOSE {error: forbidden}.

Today that bound is not proven. The regression test allows 7 s, and the persistent-exec kill helper (crates/engine/src/lib.rs, touched in 54b57a1) can wait longer than 1 s. No run was observed over 1 s.

Done when: a real-engine test on Docker and rootless Podman asserts close and termination within 1 s (with a written dev-VM tolerance, if one is needed) for every shrink cause in §2.3, and the helper's worst case is bounded by design. Security surface, Sol.

From the step 15 verify (#11, item 3). `docs/contracts.md` §2.3 says that when effective grants shrink, the server, **within one second**, bumps the authorization epochs and closes every channel and attachment no longer covered, with `CLOSE {error: forbidden}`. Today that bound is not proven. The regression test allows 7 s, and the persistent-exec kill helper (`crates/engine/src/lib.rs`, touched in `54b57a1`) can wait longer than 1 s. No run was observed over 1 s. **Done when:** a real-engine test on Docker and rootless Podman asserts close and termination within 1 s (with a written dev-VM tolerance, if one is needed) for every shrink cause in §2.3, and the helper's worst case is bounded by design. Security surface, Sol.

kayg commented 2026-09-23 07:15:20 +00:00

Author

Owner

Copy link

Starting step 11b on branch grant-shrink at base/head 1830177ac5. Scope: crates/engine persistent-exec termination, crates/server authorization revalidation and tests, and docs/contracts.md only if the process-reap timing needs clarification. I will map worst-case timing before changing code and run scripts/verify fast, integration and bench.

Starting step 11b on branch grant-shrink at base/head 1830177ac53bb17160b8b038a6016eacd76d40d2. Scope: crates/engine persistent-exec termination, crates/server authorization revalidation and tests, and docs/contracts.md only if the process-reap timing needs clarification. I will map worst-case timing before changing code and run scripts/verify fast, integration and bench.

kayg commented 2026-09-23 07:16:46 +00:00

Author

Owner

Copy link

Code-path timing before changes (base 1830177): mutation transaction commits the grant/state change and sometimes the epoch; each client stream independently polls at 500 ms. The poll then makes sequential token/grant and host database reads before finding uncovered pairs. For each pair it awaits tunnel CLOSE enqueue, client registry update, client CLOSE enqueue, then release. There is no deadline on either enqueue or database read, and an inactive token takes a WebSocket 4003 path without channel CLOSE. Detached persistent sessions use another 500 ms sweep, sequential host/owner/audit queries, then enqueue termination. The engine cleanup worker is serial and performs an unbounded initial inspect, up to 500 ms stalled stdin interrupt/flush/shutdown, up to 4 s helper create/start, then helper inspections with no per-call timeout inside a 1 s nominal loop, followed by a 5 s process/exec inspection loop also without per-call timeout. Thus the observed subsecond cases do not establish the contract: the channel path has unbounded waits, the token-inactive path misses CLOSE, and reaping can exceed 10.5 s plus engine API stalls and worker queueing on both Docker and rootless Podman. A process ignoring SIGTERM is currently handled by immediate SIGKILL helper after PTY interrupt, but its launch can itself stall.

Code-path timing before changes (base 1830177): mutation transaction commits the grant/state change and sometimes the epoch; each client stream independently polls at 500 ms. The poll then makes sequential token/grant and host database reads before finding uncovered pairs. For each pair it awaits tunnel CLOSE enqueue, client registry update, client CLOSE enqueue, then release. There is no deadline on either enqueue or database read, and an inactive token takes a WebSocket 4003 path without channel CLOSE. Detached persistent sessions use another 500 ms sweep, sequential host/owner/audit queries, then enqueue termination. The engine cleanup worker is serial and performs an unbounded initial inspect, up to 500 ms stalled stdin interrupt/flush/shutdown, up to 4 s helper create/start, then helper inspections with no per-call timeout inside a 1 s nominal loop, followed by a 5 s process/exec inspection loop also without per-call timeout. Thus the observed subsecond cases do not establish the contract: the channel path has unbounded waits, the token-inactive path misses CLOSE, and reaping can exceed 10.5 s plus engine API stalls and worker queueing on both Docker and rootless Podman. A process ignoring SIGTERM is currently handled by immediate SIGKILL helper after PTY interrupt, but its launch can itself stall.

kayg referenced this issue 2026-09-23 07:18:40 +00:00

Handover: start here (orchestrator on the dev VM) #16

kayg commented 2026-09-24 05:29:26 +00:00

Author

Owner

Copy link

Starting continuation grant-shrink2 on branch grant-shrink. Base head a6364b9b29; fetched and merged dev 7bda001e5b. I own crates/engine termination, crates/server authorization revalidation and tests, and §2.3 timing clarification. No push; required gates are scripts/verify fast and integration.

Starting continuation grant-shrink2 on branch grant-shrink. Base head a6364b9b2978b9e2f969d5fc88a6ed3d2d48ff70; fetched and merged dev 7bda001e5bd8ff42c3768fb1e6cbba317d22789e. I own crates/engine termination, crates/server authorization revalidation and tests, and §2.3 timing clarification. No push; required gates are scripts/verify fast and integration.

kayg commented 2026-09-24 05:30:54 +00:00

Author

Owner

Copy link

Worst-case timeline from the pre-fix code, starting at the committing shrink transaction (t=0):

• t=0: the mutation commits. Some paths update an epoch in the same transaction, but the stream does not subscribe to it. The stream poll can have just missed the change, adding 500 ms. A detached persistent exec has a separate 500 ms sweep.
• t>=500 ms: the client stream makes sequential token/grant and host reads. There is no time limit. The relay then sends tunnel CLOSE, updates the channel registry, sends client CLOSE, and releases the pair sequentially; either bounded queue send can wait indefinitely. On an inactive token the old path sent WebSocket 4003 without channel CLOSE. Queued DATA could pass the writer after revocation.
• t>=500 ms: the detached-session sweep reads host, owner grants, and audit state sequentially before enqueueing termination. The engine worker is serial across live execs. For one exec it can wait on an initial inspect indefinitely, then up to 500 ms on a stalled stdin interrupt/flush/shutdown, up to 4 s on helper create/start, 1 s nominal helper inspection with unbounded API calls, and 5 s process inspection with unbounded API calls. Docker and rootless Podman share this bollard path. An ignored SIGTERM is not presently scheduled: the fallback sends SIGKILL immediately after PTY interrupt, but only after the preceding waits. Hence >10.5 s plus API stalls and worker queueing is possible. The 7 s regression allowance does not prove the one-second claim.

Fix direction: fence DATA and close the client channel before tunnel/engine cleanup; bound revalidation reads and queue sends, and make engine signal escalation concurrent per exec with fixed deadlines. The one-second promise must cover withdrawal of access and CLOSE delivery on a responsive client connection; process reaping has a separately stated escalation deadline because engine API scheduling and a process ignoring SIGTERM cannot be included in that one second.

Worst-case timeline from the pre-fix code, starting at the committing shrink transaction (t=0): - t=0: the mutation commits. Some paths update an epoch in the same transaction, but the stream does not subscribe to it. The stream poll can have just missed the change, adding 500 ms. A detached persistent exec has a separate 500 ms sweep. - t>=500 ms: the client stream makes sequential token/grant and host reads. There is no time limit. The relay then sends tunnel CLOSE, updates the channel registry, sends client CLOSE, and releases the pair sequentially; either bounded queue send can wait indefinitely. On an inactive token the old path sent WebSocket 4003 without channel CLOSE. Queued DATA could pass the writer after revocation. - t>=500 ms: the detached-session sweep reads host, owner grants, and audit state sequentially before enqueueing termination. The engine worker is serial across live execs. For one exec it can wait on an initial inspect indefinitely, then up to 500 ms on a stalled stdin interrupt/flush/shutdown, up to 4 s on helper create/start, 1 s nominal helper inspection with unbounded API calls, and 5 s process inspection with unbounded API calls. Docker and rootless Podman share this bollard path. An ignored SIGTERM is not presently scheduled: the fallback sends SIGKILL immediately after PTY interrupt, but only after the preceding waits. Hence >10.5 s plus API stalls and worker queueing is possible. The 7 s regression allowance does not prove the one-second claim. Fix direction: fence DATA and close the client channel before tunnel/engine cleanup; bound revalidation reads and queue sends, and make engine signal escalation concurrent per exec with fixed deadlines. The one-second promise must cover withdrawal of access and CLOSE delivery on a responsive client connection; process reaping has a separately stated escalation deadline because engine API scheduling and a process ignoring SIGTERM cannot be included in that one second.

kayg commented 2026-09-24 05:37:20 +00:00

Author

Owner

Copy link

Finding during real-engine matrix: Docker role change and role-grant removal delivered both logs and persistent-exec CLOSE forbidden in 40.3 ms and 44.2 ms after commit. Token-grant removal delivered both CLOSE in 95.8 ms, but the process remained: §3.11 makes persistent sessions user-owned, and §2.3 ends them only when the owner no longer holds the role grant. I adjusted the matrix to verify channel closure for token-only shrink, then disable the owner and verify bounded process termination. For tag withdrawal the test now filters the owner's role grant, so that cause must also terminate the process. The matrix is still running through all seven causes on Docker and rootless Podman.

Finding during real-engine matrix: Docker role change and role-grant removal delivered both logs and persistent-exec CLOSE forbidden in 40.3 ms and 44.2 ms after commit. Token-grant removal delivered both CLOSE in 95.8 ms, but the process remained: §3.11 makes persistent sessions user-owned, and §2.3 ends them only when the owner no longer holds the role grant. I adjusted the matrix to verify channel closure for token-only shrink, then disable the owner and verify bounded process termination. For tag withdrawal the test now filters the owner's role grant, so that cause must also terminate the process. The matrix is still running through all seven causes on Docker and rootless Podman.

kayg commented 2026-09-24 05:45:01 +00:00

Author

Owner

Copy link

Real-engine seven-cause matrix after queue fencing (targeted test, before final gates): Docker CLOSE forbidden for both logs and persistent exec after commit: role change 37.3 ms, role-grant removal 36.0 ms, token-grant removal 2.8 ms, tag change 1.3 ms, user disabled 1.5 ms, token revoked 68.7 ms, token expired 68.5 ms. Rootless Podman: 34.4, 10.9, 3.9, 51.4, 13.5, 2.0, 5.8 ms respectively. Every case asserted both CLOSE frames by 1 s and no DATA afterward. Owner-grant loss reaped the process after observed CLOSE in 0.07–0.62 s on Docker and 0.31–1.19 s on rootless Podman; token-only shrink preserved the user-owned session per §3.11, then disabling the owner reaped it in 0.40–0.47 s on Docker and 0.80–1.14 s on Podman. The separate stalled-stdin test with HUP/TERM ignored reaped on Docker in 353 ms and Podman in 503 ms.

The first fast gate failed at clippy because the new matrix needed a documented large-test allowance and two checked u128→i64 conversions. Those are fixed; the full gate is running again. I also caught and corrected an expiry implementation that would have marked access tokens revoked and broken refresh. Natural expiry now bumps epochs through a dedicated marker while retaining the token's refresh eligibility.

Real-engine seven-cause matrix after queue fencing (targeted test, before final gates): Docker CLOSE forbidden for both logs and persistent exec after commit: role change 37.3 ms, role-grant removal 36.0 ms, token-grant removal 2.8 ms, tag change 1.3 ms, user disabled 1.5 ms, token revoked 68.7 ms, token expired 68.5 ms. Rootless Podman: 34.4, 10.9, 3.9, 51.4, 13.5, 2.0, 5.8 ms respectively. Every case asserted both CLOSE frames by 1 s and no DATA afterward. Owner-grant loss reaped the process after observed CLOSE in 0.07–0.62 s on Docker and 0.31–1.19 s on rootless Podman; token-only shrink preserved the user-owned session per §3.11, then disabling the owner reaped it in 0.40–0.47 s on Docker and 0.80–1.14 s on Podman. The separate stalled-stdin test with HUP/TERM ignored reaped on Docker in 353 ms and Podman in 503 ms. The first fast gate failed at clippy because the new matrix needed a documented large-test allowance and two checked u128→i64 conversions. Those are fixed; the full gate is running again. I also caught and corrected an expiry implementation that would have marked access tokens revoked and broken refresh. Natural expiry now bumps epochs through a dedicated marker while retaining the token's refresh eligibility.

kayg commented 2026-09-24 05:58:32 +00:00

Author

Owner

Copy link

Gate finding: repeated scripts/verify fast stopped in workspace tests, while the wrapper printed only the first 240 compile/test lines. A full-output cargo test --workspace run identified the regression: canonical_host_lookup_failure_is_audited_before_open_error sometimes received the new authorization watcher's transport CLOSE before the OPEN handler could return its audited OPEN_ERR. The watcher now skips host revalidation when no attachment exists, leaving a new OPEN to perform its own audited lookup. Existing attachments remain fail-closed on host lookup failure. Targeted test and full gate are being rerun.

Gate finding: repeated `scripts/verify fast` stopped in workspace tests, while the wrapper printed only the first 240 compile/test lines. A full-output `cargo test --workspace` run identified the regression: `canonical_host_lookup_failure_is_audited_before_open_error` sometimes received the new authorization watcher's transport CLOSE before the OPEN handler could return its audited OPEN_ERR. The watcher now skips host revalidation when no attachment exists, leaving a new OPEN to perform its own audited lookup. Existing attachments remain fail-closed on host lookup failure. Targeted test and full gate are being rerun.

kayg commented 2026-09-24 06:13:10 +00:00

Author

Owner

Copy link

Step 11b finished on branch grant-shrink; head 7835c0efb5. No push. The worktree is clean.

Timing design and worst-case path:

• At t=0 the shrinking mutation commits; SQLite triggers bump affected user/token epochs in that transaction. Natural token expiry is materialized by a 100 ms sweep without revoking refresh eligibility.
• Each live stream has an independent 100 ms authorization watcher, so an OPEN, relay send, or stalled stdin cannot postpone revalidation. The token/grant and host reads each have a 250 ms deadline. On failed reads the watcher fails closed. Thus the scheduled check plus read deadlines totals at most 600 ms before local close work, subject to normal task scheduling and a responsive peer.
• Before attempting engine or tunnel cleanup, the server fences DATA at the writer and places CLOSE forbidden in a dedicated priority queue with 256 reserved slots, one for every permitted open channel. The writer discards queued DATA on revoked channels. Tests assert both logs and persistent-exec CLOSE within 1 s of the transaction, and no DATA after CLOSE.
• Persistent exec cleanup is independent per session, so one stalled engine operation cannot queue behind another. It drops the PTY transport immediately, attempts SIGTERM, then schedules SIGKILL at t+300 ms if the process remains. Initial inspect has a 300 ms timeout; each signal helper has 750 ms for create/start and 750 ms for inspection; post-signal process inspection is bounded to 2 s with per-call deadlines. A stuck or unresponsive engine can still prevent confirmed process death; it cannot postpone channel closure. The contract now states that the one-second bound covers access withdrawal/CLOSE on responsive connections, while engine reaping follows the bounded escalation and reports failures.
• Token-only withdrawal closes that token's attachments but retains a user-owned persistent session while the owner's role grant remains valid, as §3.11 requires. The matrix then disables the owner and proves termination. Role changes, role-grant removal, tag withdrawal and user disable remove the owner grant and trigger termination directly.

Measured final real-engine matrix (both logs and persistent exec CLOSE forbidden after commit):
docker role_change both CLOSE forbidden in 14.860291ms
docker role_change owner process reaped in 551.874136ms
docker role_grant_removed both CLOSE forbidden in 1.488915ms
docker role_grant_removed owner process reaped in 592.210147ms
docker token_grant_removed both CLOSE forbidden in 87.109143ms
docker token_grant_removed owner process reaped in 471.169309ms
docker tag_change both CLOSE forbidden in 8.607126ms
docker tag_change owner process reaped in 606.852118ms
docker user_disabled both CLOSE forbidden in 85.38084ms
docker user_disabled owner process reaped in 497.175266ms
docker token_revoked both CLOSE forbidden in 83.975224ms
docker token_revoked owner process reaped in 409.660077ms
docker token_expired both CLOSE forbidden in 69.438538ms
docker token_expired owner process reaped in 418.582002ms
podman role_change both CLOSE forbidden in 77.406584ms
podman role_change owner process reaped in 265.702105ms
podman role_grant_removed both CLOSE forbidden in 63.497254ms
podman role_grant_removed owner process reaped in 407.132405ms
podman token_grant_removed both CLOSE forbidden in 99.20952ms
podman token_grant_removed owner process reaped in 908.496023ms
podman tag_change both CLOSE forbidden in 66.213345ms
podman tag_change owner process reaped in 1.249088018s
podman user_disabled both CLOSE forbidden in 33.245697ms
podman user_disabled owner process reaped in 882.135961ms
podman token_revoked both CLOSE forbidden in 77.887526ms
podman token_revoked owner process reaped in 872.151382ms
podman token_expired both CLOSE forbidden in 77.97409ms
podman token_expired owner process reaped in 1.067657689s

Additional real Docker/rootless Podman stalled-stdin regression: CLOSE forbidden in 43.804/47.673 ms; the engine test with HUP and SIGTERM ignored reaped in 353.021/503.420 ms. Both passed. The seven-cause matrix's maximum observed CLOSE latency was 99.210 ms (rootless Podman token-grant removal). All fourteen cases asserted a 1 s deadline and no post-CLOSE DATA. These are dev-VM observations, not a network or engine-API worst-case guarantee.

Required gate output, verbatim:
PASS fast: migrations are append-only
PASS fast: JavaScript dependencies
PASS fast: build web app
PASS fast: cargo fmt --check
PASS fast: cargo clippy
PASS fast: cargo test
PASS fast: svelte-check
PASS fast: UI package svelte-check
PASS fast: stream package typecheck
PASS fast: Vitest
PASS fast: UI package Vitest
PASS fast: stream package Vitest
PASS fast: OpenAPI and TypeScript client drift
PASS fast: terminal TypeScript
PASS fast: terminal Vitest
PASS integration: JavaScript dependencies
PASS integration: build web app
PASS integration: build release binary
PASS integration: real HTTP behavior
PASS integration: Docker socket _ping
PASS integration: rootless Podman socket _ping
PASS integration: engine adapters against Docker and Podman
PASS integration: TypeScript stream client against Docker and Podman
PASS integration: stream authorization and CLOSE regressions against Docker
PASS integration: persistent PTY termination preempts blocked send
PASS integration: resource HTTP API against Docker and rootless Podman
PASS integration: compiled CLI and MCP against real server and engines
PASS integration: SSH contract against real sshd

Step 11b finished on branch grant-shrink; head 7835c0efb52a8d0546b82f3c66543527000d3036. No push. The worktree is clean. Timing design and worst-case path: - At t=0 the shrinking mutation commits; SQLite triggers bump affected user/token epochs in that transaction. Natural token expiry is materialized by a 100 ms sweep without revoking refresh eligibility. - Each live stream has an independent 100 ms authorization watcher, so an OPEN, relay send, or stalled stdin cannot postpone revalidation. The token/grant and host reads each have a 250 ms deadline. On failed reads the watcher fails closed. Thus the scheduled check plus read deadlines totals at most 600 ms before local close work, subject to normal task scheduling and a responsive peer. - Before attempting engine or tunnel cleanup, the server fences DATA at the writer and places CLOSE forbidden in a dedicated priority queue with 256 reserved slots, one for every permitted open channel. The writer discards queued DATA on revoked channels. Tests assert both logs and persistent-exec CLOSE within 1 s of the transaction, and no DATA after CLOSE. - Persistent exec cleanup is independent per session, so one stalled engine operation cannot queue behind another. It drops the PTY transport immediately, attempts SIGTERM, then schedules SIGKILL at t+300 ms if the process remains. Initial inspect has a 300 ms timeout; each signal helper has 750 ms for create/start and 750 ms for inspection; post-signal process inspection is bounded to 2 s with per-call deadlines. A stuck or unresponsive engine can still prevent confirmed process death; it cannot postpone channel closure. The contract now states that the one-second bound covers access withdrawal/CLOSE on responsive connections, while engine reaping follows the bounded escalation and reports failures. - Token-only withdrawal closes that token's attachments but retains a user-owned persistent session while the owner's role grant remains valid, as §3.11 requires. The matrix then disables the owner and proves termination. Role changes, role-grant removal, tag withdrawal and user disable remove the owner grant and trigger termination directly. Measured final real-engine matrix (both logs and persistent exec CLOSE forbidden after commit): docker role_change both CLOSE forbidden in 14.860291ms docker role_change owner process reaped in 551.874136ms docker role_grant_removed both CLOSE forbidden in 1.488915ms docker role_grant_removed owner process reaped in 592.210147ms docker token_grant_removed both CLOSE forbidden in 87.109143ms docker token_grant_removed owner process reaped in 471.169309ms docker tag_change both CLOSE forbidden in 8.607126ms docker tag_change owner process reaped in 606.852118ms docker user_disabled both CLOSE forbidden in 85.38084ms docker user_disabled owner process reaped in 497.175266ms docker token_revoked both CLOSE forbidden in 83.975224ms docker token_revoked owner process reaped in 409.660077ms docker token_expired both CLOSE forbidden in 69.438538ms docker token_expired owner process reaped in 418.582002ms podman role_change both CLOSE forbidden in 77.406584ms podman role_change owner process reaped in 265.702105ms podman role_grant_removed both CLOSE forbidden in 63.497254ms podman role_grant_removed owner process reaped in 407.132405ms podman token_grant_removed both CLOSE forbidden in 99.20952ms podman token_grant_removed owner process reaped in 908.496023ms podman tag_change both CLOSE forbidden in 66.213345ms podman tag_change owner process reaped in 1.249088018s podman user_disabled both CLOSE forbidden in 33.245697ms podman user_disabled owner process reaped in 882.135961ms podman token_revoked both CLOSE forbidden in 77.887526ms podman token_revoked owner process reaped in 872.151382ms podman token_expired both CLOSE forbidden in 77.97409ms podman token_expired owner process reaped in 1.067657689s Additional real Docker/rootless Podman stalled-stdin regression: CLOSE forbidden in 43.804/47.673 ms; the engine test with HUP and SIGTERM ignored reaped in 353.021/503.420 ms. Both passed. The seven-cause matrix's maximum observed CLOSE latency was 99.210 ms (rootless Podman token-grant removal). All fourteen cases asserted a 1 s deadline and no post-CLOSE DATA. These are dev-VM observations, not a network or engine-API worst-case guarantee. Required gate output, verbatim: PASS fast: migrations are append-only PASS fast: JavaScript dependencies PASS fast: build web app PASS fast: cargo fmt --check PASS fast: cargo clippy PASS fast: cargo test PASS fast: svelte-check PASS fast: UI package svelte-check PASS fast: stream package typecheck PASS fast: Vitest PASS fast: UI package Vitest PASS fast: stream package Vitest PASS fast: OpenAPI and TypeScript client drift PASS fast: terminal TypeScript PASS fast: terminal Vitest PASS integration: JavaScript dependencies PASS integration: build web app PASS integration: build release binary PASS integration: real HTTP behavior PASS integration: Docker socket _ping PASS integration: rootless Podman socket _ping PASS integration: engine adapters against Docker and Podman PASS integration: TypeScript stream client against Docker and Podman PASS integration: stream authorization and CLOSE regressions against Docker PASS integration: persistent PTY termination preempts blocked send PASS integration: resource HTTP API against Docker and rootless Podman PASS integration: compiled CLI and MCP against real server and engines PASS integration: SSH contract against real sshd

kayg commented 2026-09-24 06:15:03 +00:00

Author

Owner

Copy link

Review start: branch grant-shrink, base 7bda001, head 7835c0efb5. Scope: independent defensive audit of step 11b; source tree remains read-only.

Review start: branch grant-shrink, base 7bda001, head 7835c0efb52a8d0546b82f3c66543527000d3036. Scope: independent defensive audit of step 11b; source tree remains read-only.

kayg commented 2026-09-24 06:30:49 +00:00

Author

Owner

Copy link

Independent review of 7bda001..7835c0e on grant-shrink (7835c0efb52a8d0546b82f3c66543527000d3036). Source tree was not edited. Findings, ordered by severity:

1. blocks-merge — SIGKILL can be skipped while a child still runs. terminate_exec sends TERM to the process group but tests only the original leader PID at 300 ms before deciding whether to send KILL (crates/engine/src/lib.rs:832-843). A TERM-sensitive leader can exit while a same-group child ignores TERM; inspect_exec then reports the leader done. Scratch group-repro.sh produced child_survived_TERM=yes on both real Docker and rootless Podman. detached-repro.sh also showed a child in a new process group survives KILL of the original group on both engines. Test case: an exec shell spawns a TERM-ignoring same-group child and a detached child, then exits on TERM; withdrawal must reap both, or the contract must explicitly exclude escaped process groups. The existing integration test runs only a single exec sleep 3600. Linux kill(2) defines a negative PID as exactly one process group: https://www.man7.org/linux/man-pages/man2/kill.2.html.

2. blocks-merge — forced CLOSE is lost on a full bounded queue. close_forbidden_pairs ignores both sender.try_send(CLOSE) and output.try_send(CLOSE) errors and then releases the pair (crates/server/src/stream.rs:1736-1770). A one-slot infraternal-agent inbound queue already holding DATA can reject the only termination frame; a full revocation queue can reject the only client CLOSE. No retry remains after pair release. Test case: nonpersistent exec with PTY stdin stalled and its inbound slot occupied, then remove its grant; assert forbidden CLOSE and process exit. The current stalled-stdin test uses a persistent exec whose separate owner sweep can hide lost attachment cleanup. Tokio documents try_send returning Full immediately: https://docs.rs/tokio/latest/tokio/sync/mpsc/struct.Sender.html#method.try_send.

3. blocks-merge — an in-flight OPEN can dispatch after withdrawal. open_channel authorizes, awaits target lookup and audit, then calls embedded_agent.open_frame_at without an epoch check at dispatch (crates/server/src/stream.rs:727-1057). stream_authorization also reads token state, role grants and token grants in separate queries (crates/server/src/storage/mod.rs:1217-1240), so a concurrent grant transaction can yield a mixed snapshot. The watcher skips while no pair exists and can only close the new pair on a later tick. Test case: pause target lookup after authorization, commit role/token grant removal, then release lookup; no exec or reattach may start after commit. Also race a transaction between the role- and token-grant reads. This violates §2.3's effective-permission rule at OPEN dispatch.

4. blocks-merge — a brief withdrawal can be missed entirely. The watcher compares current grants but never the user/token epoch bound to an existing pair (crates/server/src/stream.rs:418-491,1662-1707). DELETE and INSERT of a grant between 100 ms polls bump epochs twice yet leave the old attachment and persistent session active. Test case: open exec, synchronize just after a poll, commit removal and restoration before the next poll, then require forbidden CLOSE and termination of the old session. This also covers a disable/re-enable sequence. §2.3 says the shrink itself closes uncovered attachments.

5. blocks-merge — priority forbidden CLOSE can overtake earlier control frames. The writer drains revocation_rx before interactive_rx but filters only DATA for revoked channels (crates/server/src/stream.rs:367-392). An OPEN_OK or CREDIT already queued for the channel can therefore be sent after its forbidden CLOSE, violating §3.4's per-channel order/retirement rules. Test case: stall the socket writer with OPEN_OK queued, withdraw the grant, release the writer, and assert no same-channel frame follows CLOSE.

6. later — forced attachment closures have no audit record. close_forbidden_pairs creates a fresh request id for the error but never calls storage.audit; stream.session.terminate covers only persistent-exec cleanup. Test case: open a channel, withdraw its grant, then query audit records for the forced channel close, including target and actor. Existing OPEN_ERR paths without an attached stream are audited by send_audited_open_error, and their regression tests passed; this gap concerns the subsequent forced CLOSE. §4 should explicitly name the closure event if that is the intended audit requirement.

7. later — queued-data fencing is not exercised, and the new wording needs an observation point. The new matrix confirms OPEN_OK, process start, and forbidden CLOSE within one second, so its close assertion is not vacuous. It permits DATA after the epoch bump until CLOSE and creates no deliberate writer/socket backlog (crates/server/tests/stream_regressions.rs:1169-1266). A DATA frame already in awaited sink.send or the TCP send buffer can arrive after the bump; the writer fence cannot retract it. Test case: saturate logs, stop reading, commit withdrawal, then drain and count DATA arriving after commit and before CLOSE; assert relay-queued DATA is dropped after the writer observes the fence. Clarify that §2.3's DATA fence applies after watcher detection at the writer, while already accepted socket bytes may still arrive. On an unresponsive peer there is no bounded physical receipt, as the revised wording correctly states.

Contract/timing assessment. The revised one-second guarantee is reasonable for server-side withdrawal and channel CLOSE on a responsive connection; process reaping must remain separate. This run measured all 14 real Docker/rootless Podman grant-shrink cases: maximum forbidden CLOSE 83.989786 ms, mean 37.644247 ms; stalled-stdin CLOSE 19.386622 ms Docker and 45.446256 ms Podman. Complete process reaping took 350.27418 ms and 507.454767 ms in the stalled-stdin engine test; one rootless Podman role-change case took 2.640139361 s. These are dev-VM diagnostic samples under concurrent work, not perf-test VM A/B results. Scratch reproducers and captured timing log: /mnt/hdd/infraternal/artifacts/review-grant-shrink/.

Gate output (verbatim). scripts/verify fast reported:

PASS fast: migrations are append-only
PASS fast: JavaScript dependencies
PASS fast: build web app
PASS fast: cargo fmt --check
PASS fast: cargo clippy
PASS fast: cargo test
PASS fast: svelte-check
PASS fast: UI package svelte-check
PASS fast: stream package typecheck
PASS fast: Vitest
PASS fast: UI package Vitest
PASS fast: stream package Vitest
PASS fast: OpenAPI and TypeScript client drift
PASS fast: terminal TypeScript
PASS fast: terminal Vitest

scripts/verify integration exited 0 and reported:

PASS integration: JavaScript dependencies
PASS integration: build web app
PASS integration: build release binary
PASS integration: real HTTP behavior
PASS integration: Docker socket _ping
PASS integration: rootless Podman socket _ping
PASS integration: engine adapters against Docker and Podman
PASS integration: TypeScript stream client against Docker and Podman
PASS integration: stream authorization and CLOSE regressions against Docker
PASS integration: persistent PTY termination preempts blocked send
PASS integration: resource HTTP API against Docker and rootless Podman
PASS integration: compiled CLI and MCP against real server and engines
PASS integration: SSH contract against real sshd

Independent review of `7bda001..7835c0e` on `grant-shrink` (`7835c0efb52a8d0546b82f3c66543527000d3036`). Source tree was not edited. Findings, ordered by severity: 1. **blocks-merge — SIGKILL can be skipped while a child still runs.** `terminate_exec` sends TERM to the process group but tests only the original leader PID at 300 ms before deciding whether to send KILL (`crates/engine/src/lib.rs:832-843`). A TERM-sensitive leader can exit while a same-group child ignores TERM; `inspect_exec` then reports the leader done. Scratch `group-repro.sh` produced `child_survived_TERM=yes` on both real Docker and rootless Podman. `detached-repro.sh` also showed a child in a new process group survives KILL of the original group on both engines. Test case: an exec shell spawns a TERM-ignoring same-group child and a detached child, then exits on TERM; withdrawal must reap both, or the contract must explicitly exclude escaped process groups. The existing integration test runs only a single `exec sleep 3600`. Linux `kill(2)` defines a negative PID as exactly one process group: https://www.man7.org/linux/man-pages/man2/kill.2.html. 2. **blocks-merge — forced CLOSE is lost on a full bounded queue.** `close_forbidden_pairs` ignores both `sender.try_send(CLOSE)` and `output.try_send(CLOSE)` errors and then releases the pair (`crates/server/src/stream.rs:1736-1770`). A one-slot infraternal-agent inbound queue already holding DATA can reject the only termination frame; a full revocation queue can reject the only client CLOSE. No retry remains after pair release. Test case: nonpersistent exec with PTY stdin stalled and its inbound slot occupied, then remove its grant; assert forbidden CLOSE and process exit. The current stalled-stdin test uses a persistent exec whose separate owner sweep can hide lost attachment cleanup. Tokio documents `try_send` returning `Full` immediately: https://docs.rs/tokio/latest/tokio/sync/mpsc/struct.Sender.html#method.try_send. 3. **blocks-merge — an in-flight OPEN can dispatch after withdrawal.** `open_channel` authorizes, awaits target lookup and audit, then calls `embedded_agent.open_frame_at` without an epoch check at dispatch (`crates/server/src/stream.rs:727-1057`). `stream_authorization` also reads token state, role grants and token grants in separate queries (`crates/server/src/storage/mod.rs:1217-1240`), so a concurrent grant transaction can yield a mixed snapshot. The watcher skips while no pair exists and can only close the new pair on a later tick. Test case: pause target lookup after authorization, commit role/token grant removal, then release lookup; no exec or reattach may start after commit. Also race a transaction between the role- and token-grant reads. This violates §2.3's effective-permission rule at OPEN dispatch. 4. **blocks-merge — a brief withdrawal can be missed entirely.** The watcher compares current grants but never the user/token epoch bound to an existing pair (`crates/server/src/stream.rs:418-491,1662-1707`). DELETE and INSERT of a grant between 100 ms polls bump epochs twice yet leave the old attachment and persistent session active. Test case: open exec, synchronize just after a poll, commit removal and restoration before the next poll, then require forbidden CLOSE and termination of the old session. This also covers a disable/re-enable sequence. §2.3 says the shrink itself closes uncovered attachments. 5. **blocks-merge — priority forbidden CLOSE can overtake earlier control frames.** The writer drains `revocation_rx` before `interactive_rx` but filters only DATA for revoked channels (`crates/server/src/stream.rs:367-392`). An OPEN_OK or CREDIT already queued for the channel can therefore be sent after its forbidden CLOSE, violating §3.4's per-channel order/retirement rules. Test case: stall the socket writer with OPEN_OK queued, withdraw the grant, release the writer, and assert no same-channel frame follows CLOSE. 6. **later — forced attachment closures have no audit record.** `close_forbidden_pairs` creates a fresh request id for the error but never calls `storage.audit`; `stream.session.terminate` covers only persistent-exec cleanup. Test case: open a channel, withdraw its grant, then query audit records for the forced channel close, including target and actor. Existing OPEN_ERR paths without an attached stream are audited by `send_audited_open_error`, and their regression tests passed; this gap concerns the subsequent forced CLOSE. §4 should explicitly name the closure event if that is the intended audit requirement. 7. **later — queued-data fencing is not exercised, and the new wording needs an observation point.** The new matrix confirms OPEN_OK, process start, and forbidden CLOSE within one second, so its close assertion is not vacuous. It permits DATA after the epoch bump until CLOSE and creates no deliberate writer/socket backlog (`crates/server/tests/stream_regressions.rs:1169-1266`). A DATA frame already in awaited `sink.send` or the TCP send buffer can arrive after the bump; the writer fence cannot retract it. Test case: saturate logs, stop reading, commit withdrawal, then drain and count DATA arriving after commit and before CLOSE; assert relay-queued DATA is dropped after the writer observes the fence. Clarify that §2.3's DATA fence applies after watcher detection at the writer, while already accepted socket bytes may still arrive. On an unresponsive peer there is no bounded physical receipt, as the revised wording correctly states. **Contract/timing assessment.** The revised one-second guarantee is reasonable for server-side withdrawal and channel CLOSE on a responsive connection; process reaping must remain separate. This run measured all 14 real Docker/rootless Podman grant-shrink cases: maximum forbidden CLOSE 83.989786 ms, mean 37.644247 ms; stalled-stdin CLOSE 19.386622 ms Docker and 45.446256 ms Podman. Complete process reaping took 350.27418 ms and 507.454767 ms in the stalled-stdin engine test; one rootless Podman role-change case took 2.640139361 s. These are dev-VM diagnostic samples under concurrent work, not perf-test VM A/B results. Scratch reproducers and captured timing log: `/mnt/hdd/infraternal/artifacts/review-grant-shrink/`. **Gate output (verbatim).** `scripts/verify fast` reported: ``` PASS fast: migrations are append-only PASS fast: JavaScript dependencies PASS fast: build web app PASS fast: cargo fmt --check PASS fast: cargo clippy PASS fast: cargo test PASS fast: svelte-check PASS fast: UI package svelte-check PASS fast: stream package typecheck PASS fast: Vitest PASS fast: UI package Vitest PASS fast: stream package Vitest PASS fast: OpenAPI and TypeScript client drift PASS fast: terminal TypeScript PASS fast: terminal Vitest ``` `scripts/verify integration` exited 0 and reported: ``` PASS integration: JavaScript dependencies PASS integration: build web app PASS integration: build release binary PASS integration: real HTTP behavior PASS integration: Docker socket _ping PASS integration: rootless Podman socket _ping PASS integration: engine adapters against Docker and Podman PASS integration: TypeScript stream client against Docker and Podman PASS integration: stream authorization and CLOSE regressions against Docker PASS integration: persistent PTY termination preempts blocked send PASS integration: resource HTTP API against Docker and rootless Podman PASS integration: compiled CLI and MCP against real server and engines PASS integration: SSH contract against real sshd ```

kayg commented 2026-09-24 06:32:56 +00:00

Author

Owner

Copy link

Starting step 11b security fix round in branch grant-shrink at base 7835c0efb5. Scope: crates/server stream and storage authorization, crates/engine persistent exec termination, associated tests, and docs/contracts.md §2.3. Reviewing the independent findings and scratch reproducers, then implementing regression-tested fixes. Gates planned: targeted tests, scripts/verify fast, scripts/verify integration. No push.

Starting step 11b security fix round in branch grant-shrink at base 7835c0efb52a8d0546b82f3c66543527000d3036. Scope: crates/server stream and storage authorization, crates/engine persistent exec termination, associated tests, and docs/contracts.md §2.3. Reviewing the independent findings and scratch reproducers, then implementing regression-tested fixes. Gates planned: targeted tests, scripts/verify fast, scripts/verify integration. No push.

kayg commented 2026-09-24 06:45:30 +00:00

Author

Owner

Copy link

Root-cause evidence from this fix round: close_forbidden_pairs dropped CLOSE on full bounded queues (try_send returned Full), reproduced by the new unit test before the queue change. OPEN read identity/grants in separate SQLite statements and had no dispatch epoch check; the new storage regression failed to compile before the API was added and now checks that a revoked snapshot cannot run the dispatch callback. The writer previously filtered only DATA; a queued OPEN_OK/CREDIT could follow priority forbidden CLOSE. The new writer regression covers all same-channel frames. The engine's leader/group cleanup missed a detached TERM-ignoring child: the new real-Docker regression failed with that child still present, then passed after cleanup began signaling all processes carrying a per-exec inherited marker. Rootless Podman passed the same regression. One remaining cross-lane limit: while the infraternal agent is blocked writing PTY stdin it does not poll its inbound queue, so a server-only queued CLOSE or sender drop cannot guarantee prompt nonpersistent reaping; crates/agent is outside this job's file ownership.

Root-cause evidence from this fix round: `close_forbidden_pairs` dropped CLOSE on full bounded queues (`try_send` returned Full), reproduced by the new unit test before the queue change. OPEN read identity/grants in separate SQLite statements and had no dispatch epoch check; the new storage regression failed to compile before the API was added and now checks that a revoked snapshot cannot run the dispatch callback. The writer previously filtered only DATA; a queued OPEN_OK/CREDIT could follow priority forbidden CLOSE. The new writer regression covers all same-channel frames. The engine's leader/group cleanup missed a detached TERM-ignoring child: the new real-Docker regression failed with that child still present, then passed after cleanup began signaling all processes carrying a per-exec inherited marker. Rootless Podman passed the same regression. One remaining cross-lane limit: while the infraternal agent is blocked writing PTY stdin it does not poll its inbound queue, so a server-only queued CLOSE or sender drop cannot guarantee prompt nonpersistent reaping; crates/agent is outside this job's file ownership.

kayg commented 2026-09-24 07:21:35 +00:00

Author

Owner

Copy link

Security fix round handoff (branch grant-shrink, base 7835c0e, head 296e9602e0118139cd273483491fb2940bb07ea5). Commits: 18083ac marker experiment superseded by eb8de35 safe process-group correction; 296e960 stream/storage/docs. No push. Worktree clean.

Implemented: one SQLite read snapshot for stream identity/grants/epochs; final OPEN dispatch check serialized with grant transactions; each pair bound to both epochs so remove→restore still closes its original attachment; all queued same-channel frames filtered after priority forbidden CLOSE; forced attachment closure audit with actor, target, channel and forbidden error code; bounded full-queue CLOSE retry. §2.3 now states the writer observation point and that already kernel-accepted bytes may arrive after commit. A real Docker/rootless Podman matrix assertion counted 3 pre-CLOSE log DATA frames after commit on each engine and rejected every same-channel frame after CLOSE.

Engine: a real Docker regression first showed a TERM-ignoring same-group child alive after its leader exited; the final code captures the exec process-group identity at start and checks surviving host PGID/session members before KILL. Targeted regression passed on Docker (575.454475ms reaping) and rootless Podman (1.107028149s). These are dev-VM diagnostics, not perf-test VM A/B numbers. The final grant-shrink matrix passed on both engines: maximum observed forbidden CLOSE 119.980881ms; transient remove→restore CLOSE 46.749248ms Docker and 2.179982ms Podman; owner process reaping varied 94.893161ms–1.681728827s.

UNRESOLVED BLOCKERS: (1) Detached setsid/double-fork descendants leave the original process group. A marker-based experiment was removed from the final tree after env -i detached children survived on both engines and independent review found an unsafe root helper PATH surface. Default Docker and rootless Podman containers have read-only /sys/fs/cgroup, so strict full-descendant containment needs a per-exec kernel/runtime boundary. (2) In crates/agent, nonpersistent exec handling awaits PTY stdin writes without polling inbound CLOSE; while that write is stalled, server-only queue retry or sender drop cannot guarantee prompt process exit. That crate was explicitly outside this job's ownership. (3) A peer stalled longer than the 250ms priority retry can miss physical CLOSE even though its channel is retired and further egress is fenced. These are not claimed fixed.

Verification output below is quoted verbatim. Final scripts/verify fast exited 0. scripts/verify integration exited 0 before the final test-only DATA assertion; the final targeted Docker/rootless Podman matrix passed afterward.

scripts/verify fast:

PASS fast: migrations are append-only
PASS fast: JavaScript dependencies
PASS fast: build web app
PASS fast: cargo fmt --check
PASS fast: cargo clippy
PASS fast: cargo test
PASS fast: svelte-check
PASS fast: UI package svelte-check
PASS fast: stream package typecheck
PASS fast: Vitest
PASS fast: UI package Vitest
PASS fast: stream package Vitest
PASS fast: OpenAPI and TypeScript client drift
PASS fast: terminal TypeScript
PASS fast: terminal Vitest

scripts/verify integration:

PASS integration: JavaScript dependencies
PASS integration: build web app
PASS integration: build release binary
PASS integration: real HTTP behavior
PASS integration: Docker socket _ping
PASS integration: rootless Podman socket _ping
PASS integration: engine adapters against Docker and Podman
PASS integration: TypeScript stream client against Docker and Podman
PASS integration: stream authorization and CLOSE regressions against Docker
PASS integration: persistent PTY termination preempts blocked send
PASS integration: resource HTTP API against Docker and rootless Podman
PASS integration: compiled CLI and MCP against real server and engines
PASS integration: SSH contract against real sshd

Targeted final matrix result:

test every_grant_shrink_closes_logs_and_persistent_exec_on_both_engines ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 12 filtered out; finished in 45.94s

Security fix round handoff (branch `grant-shrink`, base `7835c0e`, head `296e9602e0118139cd273483491fb2940bb07ea5`). Commits: `18083ac` marker experiment superseded by `eb8de35` safe process-group correction; `296e960` stream/storage/docs. No push. Worktree clean. Implemented: one SQLite read snapshot for stream identity/grants/epochs; final OPEN dispatch check serialized with grant transactions; each pair bound to both epochs so remove→restore still closes its original attachment; all queued same-channel frames filtered after priority forbidden CLOSE; forced attachment closure audit with actor, target, channel and forbidden error code; bounded full-queue CLOSE retry. §2.3 now states the writer observation point and that already kernel-accepted bytes may arrive after commit. A real Docker/rootless Podman matrix assertion counted 3 pre-CLOSE log DATA frames after commit on each engine and rejected every same-channel frame after CLOSE. Engine: a real Docker regression first showed a TERM-ignoring same-group child alive after its leader exited; the final code captures the exec process-group identity at start and checks surviving host PGID/session members before KILL. Targeted regression passed on Docker (575.454475ms reaping) and rootless Podman (1.107028149s). These are dev-VM diagnostics, not perf-test VM A/B numbers. The final grant-shrink matrix passed on both engines: maximum observed forbidden CLOSE 119.980881ms; transient remove→restore CLOSE 46.749248ms Docker and 2.179982ms Podman; owner process reaping varied 94.893161ms–1.681728827s. UNRESOLVED BLOCKERS: (1) Detached `setsid`/double-fork descendants leave the original process group. A marker-based experiment was removed from the final tree after `env -i` detached children survived on both engines and independent review found an unsafe root helper PATH surface. Default Docker and rootless Podman containers have read-only `/sys/fs/cgroup`, so strict full-descendant containment needs a per-exec kernel/runtime boundary. (2) In `crates/agent`, nonpersistent exec handling awaits PTY stdin writes without polling inbound CLOSE; while that write is stalled, server-only queue retry or sender drop cannot guarantee prompt process exit. That crate was explicitly outside this job's ownership. (3) A peer stalled longer than the 250ms priority retry can miss physical CLOSE even though its channel is retired and further egress is fenced. These are not claimed fixed. Verification output below is quoted verbatim. Final `scripts/verify fast` exited 0. `scripts/verify integration` exited 0 before the final test-only DATA assertion; the final targeted Docker/rootless Podman matrix passed afterward. `scripts/verify fast`: ``` PASS fast: migrations are append-only PASS fast: JavaScript dependencies PASS fast: build web app PASS fast: cargo fmt --check PASS fast: cargo clippy PASS fast: cargo test PASS fast: svelte-check PASS fast: UI package svelte-check PASS fast: stream package typecheck PASS fast: Vitest PASS fast: UI package Vitest PASS fast: stream package Vitest PASS fast: OpenAPI and TypeScript client drift PASS fast: terminal TypeScript PASS fast: terminal Vitest ``` `scripts/verify integration`: ``` PASS integration: JavaScript dependencies PASS integration: build web app PASS integration: build release binary PASS integration: real HTTP behavior PASS integration: Docker socket _ping PASS integration: rootless Podman socket _ping PASS integration: engine adapters against Docker and Podman PASS integration: TypeScript stream client against Docker and Podman PASS integration: stream authorization and CLOSE regressions against Docker PASS integration: persistent PTY termination preempts blocked send PASS integration: resource HTTP API against Docker and rootless Podman PASS integration: compiled CLI and MCP against real server and engines PASS integration: SSH contract against real sshd ``` Targeted final matrix result: ``` test every_grant_shrink_closes_logs_and_persistent_exec_on_both_engines ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 12 filtered out; finished in 45.94s ```

kayg commented 2026-09-24 07:41:08 +00:00

Author

Owner

Copy link

Orchestrator: migration number collision. grant-shrink (11b) adds 0010_authorization_epochs.sql, and ui-apis (12f) adds its own 0010. Whichever branch merges to dev first keeps 0010. The other renumbers its unmerged migration to 0011 before merging, re-runs its upgrade test from the new dev head, and passes the append-only gate.

Orchestrator: **migration number collision.** `grant-shrink` (11b) adds `0010_authorization_epochs.sql`, and `ui-apis` (12f) adds its own `0010`. Whichever branch merges to `dev` first keeps `0010`. The other renumbers its **unmerged** migration to `0011` before merging, re-runs its upgrade test from the new `dev` head, and passes the append-only gate.

kayg commented 2026-09-24 07:58:02 +00:00

Author

Owner

Copy link

Step 11b follow-up on branch grant-shrink (base 7835c0e), final head ce77555a860fb98dcf5329caf307ee729983019f. No push. Worktree clean.

Implemented and verified within this job's server/engine scope:

• OPEN reads identity, grants, and user/token epochs in one SQLite snapshot; final epoch check and dispatch callback run under BEGIN IMMEDIATE, serialized with grant mutation. A failed commit returns the started channel for explicit close/compensation.
• Each attachment retains both epochs, so remove→restore still retires the original attachment. Withdrawal audits stream.attachment.close, fences queued writer frames and incoming DATA once observed, and prioritizes the forbidden CLOSE. An already marked ordinary CLOSE no longer suppresses the forced CLOSE. Full priority/inbound queues or audit failure signal bounded fail-closed socket shutdown (1011); writer sends are interruptible.
• Persistent exec captures its original process group. Closing it reaps a TERM-ignoring same-group child after the leader exits on real Docker and rootless Podman. Capture is bounded and group identity is checked against PID reuse.
• docs/contracts.md §2.3 describes epoch binding, in-flight OPEN, writer ordering, and the limit that bytes already accepted by the kernel can arrive after withdrawal.

Unresolved required guarantees:

• Detached setsid/double-fork descendants escape process-group signaling. A marker approach was removed because env -i evades it on both engines and a root helper could execute container-controlled programs. Both default engine cgroup mounts were read-only; a per-exec kernel/runtime boundary is needed to guarantee reaping every descendant.
• The nonpersistent infraternal-agent task can stall in PTY stdin write without polling its inbound CLOSE. This server lane has no abort hook, so bounded queue retry and socket shutdown cannot guarantee prompt process exit in that case. This needs the separately owned crates/agent control path.
• A persistent session created after the one-second OPEN_OK compensation wait on an uncertain SQLite COMMIT failure can remain untracked. The current AgentChannel API lacks an abort handle that can guarantee cancellation.
• The inbound DATA queue fence begins when the watcher observes the epoch bump (100 ms poll), not exactly at commit. The writer fences queued frames at that observation, but strict no-enqueue-after-commit is not established. In the real remove→restore matrix, three log DATA frames appeared after commit but before CLOSE on each engine; none appeared after CLOSE.

Diagnostic measurements on the shared dev VM (not official perf-test numbers): forbidden CLOSE 2.18–119.98 ms in the Docker/rootless Podman grant-shrink matrix; owner process reaping 94.89 ms–1.68 s. Same-group leader-exit child reaping: Docker 575.45 ms, rootless Podman 1.107 s. env -i setsid child survived on both engines.

Final-head fast gate, exit 0:

PASS fast: migrations are append-only
PASS fast: JavaScript dependencies
PASS fast: build web app
PASS fast: cargo fmt --check
PASS fast: cargo clippy
PASS fast: cargo test
PASS fast: svelte-check
PASS fast: UI package svelte-check
PASS fast: stream package typecheck
PASS fast: Vitest
PASS fast: UI package Vitest
PASS fast: stream package Vitest
PASS fast: OpenAPI and TypeScript client drift
PASS fast: terminal TypeScript
PASS fast: terminal Vitest

cargo test -p infraternal-server --lib forced_close_survives_full_queues_and_already_marked_local_close -- --nocapture: exit 0, 1 passed. Prior real Docker/rootless Podman grant-shrink matrix: exit 0, 1 passed, 0 failed.

Final-head integration rerun, exit 1 because the shared localhost sshd had an accumulated failed-authentication source penalty (OpenSSH documents this behavior: https://www.openssh.org/releasenotes.html):

PASS integration: JavaScript dependencies
PASS integration: build web app
PASS integration: build release binary
PASS integration: real HTTP behavior
PASS integration: Docker socket _ping
PASS integration: rootless Podman socket _ping
PASS integration: engine adapters against Docker and Podman
PASS integration: TypeScript stream client against Docker and Podman
PASS integration: stream authorization and CLOSE regressions against Docker
PASS integration: persistent PTY termination preempts blocked send
PASS integration: resource HTTP API against Docker and rootless Podman
PASS integration: compiled CLI and MCP against real server and engines
FAIL integration: SSH contract against real sshd
    Finished `test` profile [unoptimized] target(s) in 0.26s
     Running tests/real_sshd.rs (target/debug/deps/real_sshd-f889221c50c2cd15)

running 11 tests
test concurrent_same_open_key ... 
thread 'concurrent_same_open_key' (4183769) panicked at crates/ssh/tests/real_sshd.rs:318:37:
iteration 1 first open failed: Ssh { stage: Connect, message: "Disconnected" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
FAILED
test direct_terminate_terminates_shell ... 
thread 'direct_terminate_terminates_shell' (4183867) panicked at crates/ssh/tests/real_sshd.rs:642:9:
expected an explicit needs-trust outcome
FAILED
test idle_expiry_terminates_shell ... 
thread 'idle_expiry_terminates_shell' (4183896) panicked at crates/ssh/tests/real_sshd.rs:642:9:
expected an explicit needs-trust outcome
FAILED
test nonpersistent_attachment_drop_terminates_shell ... 
thread 'nonpersistent_attachment_drop_terminates_shell' (4183911) panicked at crates/ssh/tests/real_sshd.rs:642:9:
expected an explicit needs-trust outcome
FAILED
test owner_loss_terminates_shell ... 
thread 'owner_loss_terminates_shell' (4183926) panicked at crates/ssh/tests/real_sshd.rs:642:9:
expected an explicit needs-trust outcome
FAILED
test persistent_limits_allow_ephemeral ... 
thread 'persistent_limits_allow_ephemeral' (4183945) panicked at crates/ssh/tests/real_sshd.rs:642:9:
expected an explicit needs-trust outcome
FAILED
test plain_thread_termination_terminates_shell ... 
thread 'plain_thread_termination_terminates_shell' (4183964) panicked at crates/ssh/tests/real_sshd.rs:642:9:
expected an explicit needs-trust outcome
FAILED
test pty_resize_and_replay ... 
thread 'pty_resize_and_replay' (4183982) panicked at crates/ssh/tests/real_sshd.rs:642:9:
expected an explicit needs-trust outcome
FAILED
test simultaneous_actual_persistent_limits ... 
thread 'simultaneous_actual_persistent_limits' (4183997) panicked at crates/ssh/tests/real_sshd.rs:642:9:
expected an explicit needs-trust outcome
FAILED
test strict_host_key_change ... ok
test takeover_while_output_flows ... 
thread 'takeover_while_output_flows' (4184083) panicked at crates/ssh/tests/real_sshd.rs:642:9:
expected an explicit needs-trust outcome
FAILED

failures:

failures:
    concurrent_same_open_key
    direct_terminate_terminates_shell
    idle_expiry_terminates_shell
    nonpersistent_attachment_drop_terminates_shell
    owner_loss_terminates_shell
    persistent_limits_allow_ephemeral
    plain_thread_termination_terminates_shell
    pty_resize_and_replay
    simultaneous_actual_persistent_limits
    takeover_while_output_flows

test result: FAILED. 1 passed; 10 failed; 0 ignored; 0 measured; 1 filtered out; finished in 1.51s

error: test failed, to rerun pass `-p infraternal-ssh --test real_sshd`

Earlier full integration run before the last two stream corrections, exit 0:

PASS integration: JavaScript dependencies
PASS integration: build web app
PASS integration: build release binary
PASS integration: real HTTP behavior
PASS integration: Docker socket _ping
PASS integration: rootless Podman socket _ping
PASS integration: engine adapters against Docker and Podman
PASS integration: TypeScript stream client against Docker and Podman
PASS integration: stream authorization and CLOSE regressions against Docker
PASS integration: persistent PTY termination preempts blocked send
PASS integration: resource HTTP API against Docker and rootless Podman
PASS integration: compiled CLI and MCP against real server and engines
PASS integration: SSH contract against real sshd

After the sshd penalty cleared, targeted real-sshd recovery, exit 0:

    Finished `test` profile [unoptimized] target(s) in 0.24s
     Running tests/real_sshd.rs (target/debug/deps/real_sshd-f889221c50c2cd15)

running 11 tests
test concurrent_same_open_key ... ok
test direct_terminate_terminates_shell ... ok
test idle_expiry_terminates_shell ... ok
test nonpersistent_attachment_drop_terminates_shell ... ok
test owner_loss_terminates_shell ... ok
test persistent_limits_allow_ephemeral ... ok
test plain_thread_termination_terminates_shell ... ok
test pty_resize_and_replay ... ok
test simultaneous_actual_persistent_limits ... ok
test strict_host_key_change ... ok
test takeover_while_output_flows ... ok

test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out; finished in 21.94s

Step 11b follow-up on branch `grant-shrink` (base `7835c0e`), final head `ce77555a860fb98dcf5329caf307ee729983019f`. No push. Worktree clean. Implemented and verified within this job's server/engine scope: - OPEN reads identity, grants, and user/token epochs in one SQLite snapshot; final epoch check and dispatch callback run under `BEGIN IMMEDIATE`, serialized with grant mutation. A failed commit returns the started channel for explicit close/compensation. - Each attachment retains both epochs, so remove→restore still retires the original attachment. Withdrawal audits `stream.attachment.close`, fences queued writer frames and incoming DATA once observed, and prioritizes the forbidden CLOSE. An already marked ordinary CLOSE no longer suppresses the forced CLOSE. Full priority/inbound queues or audit failure signal bounded fail-closed socket shutdown (1011); writer sends are interruptible. - Persistent exec captures its original process group. Closing it reaps a TERM-ignoring same-group child after the leader exits on real Docker and rootless Podman. Capture is bounded and group identity is checked against PID reuse. - `docs/contracts.md` §2.3 describes epoch binding, in-flight OPEN, writer ordering, and the limit that bytes already accepted by the kernel can arrive after withdrawal. Unresolved required guarantees: - Detached `setsid`/double-fork descendants escape process-group signaling. A marker approach was removed because `env -i` evades it on both engines and a root helper could execute container-controlled programs. Both default engine cgroup mounts were read-only; a per-exec kernel/runtime boundary is needed to guarantee reaping every descendant. - The nonpersistent infraternal-agent task can stall in PTY stdin write without polling its inbound CLOSE. This server lane has no abort hook, so bounded queue retry and socket shutdown cannot guarantee prompt process exit in that case. This needs the separately owned `crates/agent` control path. - A persistent session created after the one-second OPEN_OK compensation wait on an uncertain SQLite COMMIT failure can remain untracked. The current AgentChannel API lacks an abort handle that can guarantee cancellation. - The inbound DATA queue fence begins when the watcher observes the epoch bump (100 ms poll), not exactly at commit. The writer fences queued frames at that observation, but strict no-enqueue-after-commit is not established. In the real remove→restore matrix, three log DATA frames appeared after commit but before CLOSE on each engine; none appeared after CLOSE. Diagnostic measurements on the shared dev VM (not official perf-test numbers): forbidden CLOSE 2.18–119.98 ms in the Docker/rootless Podman grant-shrink matrix; owner process reaping 94.89 ms–1.68 s. Same-group leader-exit child reaping: Docker 575.45 ms, rootless Podman 1.107 s. `env -i setsid` child survived on both engines. Final-head fast gate, exit 0: ``` PASS fast: migrations are append-only PASS fast: JavaScript dependencies PASS fast: build web app PASS fast: cargo fmt --check PASS fast: cargo clippy PASS fast: cargo test PASS fast: svelte-check PASS fast: UI package svelte-check PASS fast: stream package typecheck PASS fast: Vitest PASS fast: UI package Vitest PASS fast: stream package Vitest PASS fast: OpenAPI and TypeScript client drift PASS fast: terminal TypeScript PASS fast: terminal Vitest ``` `cargo test -p infraternal-server --lib forced_close_survives_full_queues_and_already_marked_local_close -- --nocapture`: exit 0, 1 passed. Prior real Docker/rootless Podman grant-shrink matrix: exit 0, 1 passed, 0 failed. Final-head integration rerun, exit 1 because the shared localhost sshd had an accumulated failed-authentication source penalty (OpenSSH documents this behavior: https://www.openssh.org/releasenotes.html): ``` PASS integration: JavaScript dependencies PASS integration: build web app PASS integration: build release binary PASS integration: real HTTP behavior PASS integration: Docker socket _ping PASS integration: rootless Podman socket _ping PASS integration: engine adapters against Docker and Podman PASS integration: TypeScript stream client against Docker and Podman PASS integration: stream authorization and CLOSE regressions against Docker PASS integration: persistent PTY termination preempts blocked send PASS integration: resource HTTP API against Docker and rootless Podman PASS integration: compiled CLI and MCP against real server and engines FAIL integration: SSH contract against real sshd Finished `test` profile [unoptimized] target(s) in 0.26s Running tests/real_sshd.rs (target/debug/deps/real_sshd-f889221c50c2cd15) running 11 tests test concurrent_same_open_key ... thread 'concurrent_same_open_key' (4183769) panicked at crates/ssh/tests/real_sshd.rs:318:37: iteration 1 first open failed: Ssh { stage: Connect, message: "Disconnected" } note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace FAILED test direct_terminate_terminates_shell ... thread 'direct_terminate_terminates_shell' (4183867) panicked at crates/ssh/tests/real_sshd.rs:642:9: expected an explicit needs-trust outcome FAILED test idle_expiry_terminates_shell ... thread 'idle_expiry_terminates_shell' (4183896) panicked at crates/ssh/tests/real_sshd.rs:642:9: expected an explicit needs-trust outcome FAILED test nonpersistent_attachment_drop_terminates_shell ... thread 'nonpersistent_attachment_drop_terminates_shell' (4183911) panicked at crates/ssh/tests/real_sshd.rs:642:9: expected an explicit needs-trust outcome FAILED test owner_loss_terminates_shell ... thread 'owner_loss_terminates_shell' (4183926) panicked at crates/ssh/tests/real_sshd.rs:642:9: expected an explicit needs-trust outcome FAILED test persistent_limits_allow_ephemeral ... thread 'persistent_limits_allow_ephemeral' (4183945) panicked at crates/ssh/tests/real_sshd.rs:642:9: expected an explicit needs-trust outcome FAILED test plain_thread_termination_terminates_shell ... thread 'plain_thread_termination_terminates_shell' (4183964) panicked at crates/ssh/tests/real_sshd.rs:642:9: expected an explicit needs-trust outcome FAILED test pty_resize_and_replay ... thread 'pty_resize_and_replay' (4183982) panicked at crates/ssh/tests/real_sshd.rs:642:9: expected an explicit needs-trust outcome FAILED test simultaneous_actual_persistent_limits ... thread 'simultaneous_actual_persistent_limits' (4183997) panicked at crates/ssh/tests/real_sshd.rs:642:9: expected an explicit needs-trust outcome FAILED test strict_host_key_change ... ok test takeover_while_output_flows ... thread 'takeover_while_output_flows' (4184083) panicked at crates/ssh/tests/real_sshd.rs:642:9: expected an explicit needs-trust outcome FAILED failures: failures: concurrent_same_open_key direct_terminate_terminates_shell idle_expiry_terminates_shell nonpersistent_attachment_drop_terminates_shell owner_loss_terminates_shell persistent_limits_allow_ephemeral plain_thread_termination_terminates_shell pty_resize_and_replay simultaneous_actual_persistent_limits takeover_while_output_flows test result: FAILED. 1 passed; 10 failed; 0 ignored; 0 measured; 1 filtered out; finished in 1.51s error: test failed, to rerun pass `-p infraternal-ssh --test real_sshd` ``` Earlier full integration run before the last two stream corrections, exit 0: ``` PASS integration: JavaScript dependencies PASS integration: build web app PASS integration: build release binary PASS integration: real HTTP behavior PASS integration: Docker socket _ping PASS integration: rootless Podman socket _ping PASS integration: engine adapters against Docker and Podman PASS integration: TypeScript stream client against Docker and Podman PASS integration: stream authorization and CLOSE regressions against Docker PASS integration: persistent PTY termination preempts blocked send PASS integration: resource HTTP API against Docker and rootless Podman PASS integration: compiled CLI and MCP against real server and engines PASS integration: SSH contract against real sshd ``` After the sshd penalty cleared, targeted real-sshd recovery, exit 0: ``` Finished `test` profile [unoptimized] target(s) in 0.24s Running tests/real_sshd.rs (target/debug/deps/real_sshd-f889221c50c2cd15) running 11 tests test concurrent_same_open_key ... ok test direct_terminate_terminates_shell ... ok test idle_expiry_terminates_shell ... ok test nonpersistent_attachment_drop_terminates_shell ... ok test owner_loss_terminates_shell ... ok test persistent_limits_allow_ephemeral ... ok test plain_thread_termination_terminates_shell ... ok test pty_resize_and_replay ... ok test simultaneous_actual_persistent_limits ... ok test strict_host_key_change ... ok test takeover_while_output_flows ... ok test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out; finished in 21.94s ```

kayg commented 2026-09-24 08:09:16 +00:00

Author

Owner

Copy link

Final-head gate recovery on clean ce77555a860fb98dcf5329caf307ee729983019f: scripts/verify integration exited 0. The shared localhost sshd penalty had cleared; this rerun passed all 13 steps. Exact output:

PASS integration: JavaScript dependencies
PASS integration: build web app
PASS integration: build release binary
PASS integration: real HTTP behavior
PASS integration: Docker socket _ping
PASS integration: rootless Podman socket _ping
PASS integration: engine adapters against Docker and Podman
PASS integration: TypeScript stream client against Docker and Podman
PASS integration: stream authorization and CLOSE regressions against Docker
PASS integration: persistent PTY termination preempts blocked send
PASS integration: resource HTTP API against Docker and rootless Podman
PASS integration: compiled CLI and MCP against real server and engines
PASS integration: SSH contract against real sshd

Final-head gate recovery on clean `ce77555a860fb98dcf5329caf307ee729983019f`: `scripts/verify integration` exited 0. The shared localhost sshd penalty had cleared; this rerun passed all 13 steps. Exact output: ``` PASS integration: JavaScript dependencies PASS integration: build web app PASS integration: build release binary PASS integration: real HTTP behavior PASS integration: Docker socket _ping PASS integration: rootless Podman socket _ping PASS integration: engine adapters against Docker and Podman PASS integration: TypeScript stream client against Docker and Podman PASS integration: stream authorization and CLOSE regressions against Docker PASS integration: persistent PTY termination preempts blocked send PASS integration: resource HTTP API against Docker and rootless Podman PASS integration: compiled CLI and MCP against real server and engines PASS integration: SSH contract against real sshd ```

kayg referenced this issue 2026-09-24 08:10:08 +00:00

Step 11c: per-exec process boundary and remaining withdrawal gaps #28

kayg commented 2026-09-24 08:15:51 +00:00

Author

Owner

Copy link

Step 11b verification started on grant-shrink at ce77555a860fb98dcf5329caf307ee729983019f. Fetched origin/dev at 7bda001e5bd8ff42c3768fb1e6cbba317d22789e; merging FETCH_HEAD reported Already up to date. I’m re-running the review reproducers and regression tests, then the required fast and integration gates. No push.

Step 11b verification started on `grant-shrink` at `ce77555a860fb98dcf5329caf307ee729983019f`. Fetched `origin/dev` at `7bda001e5bd8ff42c3768fb1e6cbba317d22789e`; merging `FETCH_HEAD` reported `Already up to date.` I’m re-running the review reproducers and regression tests, then the required fast and integration gates. No push.

kayg commented 2026-09-24 08:35:33 +00:00

Author

Owner

Copy link

Verification finding: the role_grant_removed_then_restored timing case recorded committed only after restoring the grants, so its one-second assertion did not start at the withdrawal commit. I moved the timer to immediately after the deletion transaction commits; restoration writes now fall inside the measured interval. The matrix rationale now says eight causes. Commit: 4c15772f562dc572514f6a908ecc4095b89de35f. scripts/verify fast passed after the change.

Verification finding: the `role_grant_removed_then_restored` timing case recorded `committed` only after restoring the grants, so its one-second assertion did not start at the withdrawal commit. I moved the timer to immediately after the deletion transaction commits; restoration writes now fall inside the measured interval. The matrix rationale now says eight causes. Commit: `4c15772f562dc572514f6a908ecc4095b89de35f`. `scripts/verify fast` passed after the change.

kayg commented 2026-09-24 09:07:23 +00:00

Author

Owner

Copy link

Step 11b verification: PASS

Branch head: 4c15772f562dc572514f6a908ecc4095b89de35f. Fetched origin/dev at 7bda001e5bd8ff42c3768fb1e6cbba317d22789e; git merge --no-edit FETCH_HEAD reported Already up to date. The branch still keeps migration 0010_authorization_epochs.sql.

1. PASS — an in-flight OPEN cannot dispatch after withdrawal. stream_open_dispatch_rejects_a_withdrawn_snapshot_before_running_callback passed: after the snapshot is read and token removal commits, dispatch returns no channel and never invokes the callback. The dispatch/revocation race test is also in the passing fast suite.

2. PASS — remove and restore within one poll retires the original attachment. The Docker and rootless Podman matrix closed both original channels after role grants were removed and restored. The timing starts at the removal commit; regrant writes are included.

3. PASS — forced CLOSE delivery and per-channel retirement. writer_discards_queued_control_and_data_after_forbidden_close, forced_close_survives_full_queues_and_already_marked_local_close, full_priority_queue_forces_connection_shutdown_after_retry_bound, and full_inbound_queue_forces_connection_shutdown_after_retry_bound passed. The writer discards stale same-channel frames after forced CLOSE; queue delivery failure signals fail-closed, and crates/server/src/stream.rs sends WebSocket close code 1011 (§3.10).

4. PASS — TERM-ignoring process-group child is reaped after escalation. pty_close_reaps_term_ignoring_group_child_after_leader_exits passed on both real runtimes: Docker 532.890089 ms; rootless Podman 918.784267 ms from PTY close to child reaping.

5. PASS — every shrink cause closes within one second. Real-engine matrix, load averages at start 3.00 / 6.69 / 6.29 (1 / 5 / 15 minutes):

   Shrink cause	Docker CLOSE	Rootless Podman CLOSE
   Role change	73.743127 ms	101.740078 ms
   Role grant removed	5.142329 ms	43.440661 ms
   Role grant removed then restored	68.573525 ms	87.13569 ms
   Token grant removed	89.498948 ms	40.755202 ms
   Tag change	61.874792 ms	39.586389 ms
   User disabled	103.322206 ms	52.947859 ms
   Token revoked	68.771144 ms	55.191383 ms
   Token expired	49.322033 ms	38.666393 ms

   Maximum: 103.322206 ms. The transient case also observed 2 Docker and 5 Podman log DATA frames accepted before the writer observed retirement and before CLOSE; it observed no same-channel frames after CLOSE.

6. PASS — audits exist for forced closes and OPEN errors without a stream. forced_close_survives_full_queues_and_already_marked_local_close, failed_open_is_audited_with_its_error_before_relay, and canonical_host_lookup_failure_is_audited_before_open_error passed and queried the expected audit rows.

7. PASS — docs/contracts.md §2.3 matches the implementation. It states the commit-to-close bound, dispatch fence and permanent epoch retirement, audit failure behavior, observation point for DATA fencing, already-accepted WebSocket/kernel bytes that can arrive before CLOSE, and the responsive-peer assumption.

Step 11c / #28: all four deferred findings remain listed and were not changed by this verification: detached setsid/double-fork descendants can escape the process group; stalled nonpersistent PTY writes can block cleanup in crates/agent; persistent creation after failed commit can escape compensation; inbound DATA fencing starts at writer observation. The detached reproducer still reported child_survived_group_KILL=yes, as tracked there.

The review artifacts were rerun: same-group child survived TERM; process_group_repro.py reported leader_exited=True child_same_group_survives_TERM=True algorithm_would_skip_KILL=True; detached child survived the original group KILL. The real-engine test above confirms the shipped escalation reaps same-group children.

The one-second test's timing origin was corrected in commit 4c15772 to start immediately after the shrink commit and before restoration writes.

scripts/verify fast (exit 0):

PASS fast: migrations are append-only
PASS fast: JavaScript dependencies
PASS fast: build web app
PASS fast: cargo fmt --check
PASS fast: cargo clippy
PASS fast: cargo test
PASS fast: svelte-check
PASS fast: UI package svelte-check
PASS fast: stream package typecheck
PASS fast: Vitest
PASS fast: UI package Vitest
PASS fast: stream package Vitest
PASS fast: OpenAPI and TypeScript client drift
PASS fast: terminal TypeScript
PASS fast: terminal Vitest

scripts/verify integration (exit 0):

PASS integration: JavaScript dependencies
PASS integration: build web app
PASS integration: build release binary
PASS integration: real HTTP behavior
PASS integration: Docker socket _ping
PASS integration: rootless Podman socket _ping
PASS integration: engine adapters against Docker and Podman
PASS integration: TypeScript stream client against Docker and Podman
PASS integration: stream authorization and CLOSE regressions against Docker
PASS integration: persistent PTY termination preempts blocked send
PASS integration: resource HTTP API against Docker and rootless Podman
PASS integration: compiled CLI and MCP against real server and engines
PASS integration: SSH contract against real sshd

Step 11b verification: PASS Branch head: `4c15772f562dc572514f6a908ecc4095b89de35f`. Fetched `origin/dev` at `7bda001e5bd8ff42c3768fb1e6cbba317d22789e`; `git merge --no-edit FETCH_HEAD` reported `Already up to date.` The branch still keeps migration `0010_authorization_epochs.sql`. 1. **PASS — an in-flight OPEN cannot dispatch after withdrawal.** `stream_open_dispatch_rejects_a_withdrawn_snapshot_before_running_callback` passed: after the snapshot is read and token removal commits, dispatch returns no channel and never invokes the callback. The dispatch/revocation race test is also in the passing fast suite. 2. **PASS — remove and restore within one poll retires the original attachment.** The Docker and rootless Podman matrix closed both original channels after role grants were removed and restored. The timing starts at the removal commit; regrant writes are included. 3. **PASS — forced CLOSE delivery and per-channel retirement.** `writer_discards_queued_control_and_data_after_forbidden_close`, `forced_close_survives_full_queues_and_already_marked_local_close`, `full_priority_queue_forces_connection_shutdown_after_retry_bound`, and `full_inbound_queue_forces_connection_shutdown_after_retry_bound` passed. The writer discards stale same-channel frames after forced CLOSE; queue delivery failure signals fail-closed, and `crates/server/src/stream.rs` sends WebSocket close code 1011 (§3.10). 4. **PASS — TERM-ignoring process-group child is reaped after escalation.** `pty_close_reaps_term_ignoring_group_child_after_leader_exits` passed on both real runtimes: Docker 532.890089 ms; rootless Podman 918.784267 ms from PTY close to child reaping. 5. **PASS — every shrink cause closes within one second.** Real-engine matrix, load averages at start 3.00 / 6.69 / 6.29 (1 / 5 / 15 minutes): | Shrink cause | Docker CLOSE | Rootless Podman CLOSE | |---|---:|---:| | Role change | 73.743127 ms | 101.740078 ms | | Role grant removed | 5.142329 ms | 43.440661 ms | | Role grant removed then restored | 68.573525 ms | 87.13569 ms | | Token grant removed | 89.498948 ms | 40.755202 ms | | Tag change | 61.874792 ms | 39.586389 ms | | User disabled | 103.322206 ms | 52.947859 ms | | Token revoked | 68.771144 ms | 55.191383 ms | | Token expired | 49.322033 ms | 38.666393 ms | Maximum: 103.322206 ms. The transient case also observed 2 Docker and 5 Podman log DATA frames accepted before the writer observed retirement and before CLOSE; it observed no same-channel frames after CLOSE. 6. **PASS — audits exist for forced closes and OPEN errors without a stream.** `forced_close_survives_full_queues_and_already_marked_local_close`, `failed_open_is_audited_with_its_error_before_relay`, and `canonical_host_lookup_failure_is_audited_before_open_error` passed and queried the expected audit rows. 7. **PASS — `docs/contracts.md` §2.3 matches the implementation.** It states the commit-to-close bound, dispatch fence and permanent epoch retirement, audit failure behavior, observation point for DATA fencing, already-accepted WebSocket/kernel bytes that can arrive before CLOSE, and the responsive-peer assumption. **Step 11c / #28:** all four deferred findings remain listed and were not changed by this verification: detached `setsid`/double-fork descendants can escape the process group; stalled nonpersistent PTY writes can block cleanup in `crates/agent`; persistent creation after failed commit can escape compensation; inbound DATA fencing starts at writer observation. The detached reproducer still reported `child_survived_group_KILL=yes`, as tracked there. The review artifacts were rerun: same-group child survived TERM; `process_group_repro.py` reported `leader_exited=True child_same_group_survives_TERM=True algorithm_would_skip_KILL=True`; detached child survived the original group KILL. The real-engine test above confirms the shipped escalation reaps same-group children. The one-second test's timing origin was corrected in commit `4c15772` to start immediately after the shrink commit and before restoration writes. `scripts/verify fast` (exit 0): ```text PASS fast: migrations are append-only PASS fast: JavaScript dependencies PASS fast: build web app PASS fast: cargo fmt --check PASS fast: cargo clippy PASS fast: cargo test PASS fast: svelte-check PASS fast: UI package svelte-check PASS fast: stream package typecheck PASS fast: Vitest PASS fast: UI package Vitest PASS fast: stream package Vitest PASS fast: OpenAPI and TypeScript client drift PASS fast: terminal TypeScript PASS fast: terminal Vitest ``` `scripts/verify integration` (exit 0): ```text PASS integration: JavaScript dependencies PASS integration: build web app PASS integration: build release binary PASS integration: real HTTP behavior PASS integration: Docker socket _ping PASS integration: rootless Podman socket _ping PASS integration: engine adapters against Docker and Podman PASS integration: TypeScript stream client against Docker and Podman PASS integration: stream authorization and CLOSE regressions against Docker PASS integration: persistent PTY termination preempts blocked send PASS integration: resource HTTP API against Docker and rootless Podman PASS integration: compiled CLI and MCP against real server and engines PASS integration: SSH contract against real sshd ```

kayg referenced this issue from a commit 2026-09-24 09:52:19 +00:00

Merge one-second revocation bound (step 11b, #22)

kayg commented 2026-09-24 09:52:21 +00:00

Author

Owner

Copy link

Orchestrator: step 11b is merged to dev at efa1bac and pushed. The merge is conflict-free and file-disjoint with the deploy tooling (9688a24). Gates on the exact merge head:

• fast: 15 of 15 PASS.
• integration: 13 of 13 PASS, after one re-run. The first run hit a load-induced timeout in stalled_pty_stdin_cannot_hold_exec_termination at load5 18. At low load it passed 3 of 3, reaping in 389–400 ms on Docker and 502–732 ms on Podman. Filed as #31.

The path was build, Sol audit (5 blocks-merge findings), one Sol fix round, verify: every item PASS. The remaining process-boundary gaps are step 11c (#28).

Orchestrator: **step 11b is merged to `dev` at `efa1bac`** and pushed. The merge is conflict-free and file-disjoint with the deploy tooling (`9688a24`). Gates on the exact merge head: - `fast`: 15 of 15 PASS. - `integration`: 13 of 13 PASS, after one re-run. The first run hit a load-induced timeout in `stalled_pty_stdin_cannot_hold_exec_termination` at load5 18. At low load it passed 3 of 3, reaping in 389–400 ms on Docker and 502–732 ms on Podman. Filed as #31. The path was build, Sol audit (5 blocks-merge findings), one Sol fix round, verify: every item PASS. The remaining process-boundary gaps are step 11c (#28).

kayg referenced this issue 2026-10-03 23:28:21 +00:00

UI: replace shadcn with calternal's design system (owner decision 2026-09-26) #46

Commenting is not possible because the repository is archived.

No Branch/Tag specified

Branches Tags

main

dev

perf-gates

provider-docker-podman

fix-networkidle

fix-apierror-panics

fix-revoke-panic

wal-race

fix-private-storage

provider-runtime

plugin-design

server-channels

scope-v01

deploy-git-fix

sec-hardening

gates-tiering

sidebar-sync

multi-tab

cli-eof

podman-flake

ssh-reattach

agent-tunnel

process-boundary

shadcn

stream-params

cli-ssh

ssh-product

container-live

flaky-tests

s1-e2e

ssh-test-isolation

cli-stream

research-lxc

container-live-agent

ui-apis

grant-shrink

deploy-o2

ui-redesign

migration-fix

cli

stream

No results found.

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kayg/infraternal-v0#22

No description provided.