Step 11: Stream protocol #7
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Stream tickets, WebSocket framing, flow control, priority queues, in-process infraternal agent, server relay.
Step 11 in docs/PLAN.md. Done when: merged to dev
Starting step 11 implementation.
stream8e98a4679ec6506476679fe042f4250dd3e63e7bscripts/verify fast,scripts/verify integration,scripts/verify benchStep 11 stream implementation started.
stream8e98a4679ec6506476679fe042f4250dd3e63e7bFinding: live signed-grant tests exposed an intermittent decoder rejection. The proto base64url padding validator had the unused-bit counts reversed: 32-byte nonces require two zero padding bits in the 43rd symbol, while 64-byte Ed25519 signatures require four in the 86th. Commit
aebc00acorrects both and adds exact boundary tests; the real WebSocket grant/relay/revocation test then passed repeatedly.Checkpoint
1b9641fwires the bounded same-frame in-process transport through server channel pairing into embedded Docker/Podman logs/events/exec dispatch, adds aggregate credit accounting and grant-shrink attachment closure, and adds TypeScript send-credit backpressure and serialized consumption. Focused evidence: infraternal-agent 2/2, server stream 4/4, packages/stream 3/3 plus tsc.Implementation checkpoint at
42ace5e80d51d8a11aad7afa1b58ed231b46c6fe.Delivered: atomic one-shot stream tickets; strict WebSocket handshake/origin/subprotocol/revocation behavior; bounded channel registry, per-channel and aggregate credit accounting, priority queues and liveness; fresh authorization and audit at each OPEN; signed Ed25519 grants with persisted server key, canonical payload binding, replay/tunnel/ownership checks; same-frame bounded in-process relay; Docker/Podman engine dispatch for logs/events/nonpersistent exec; TypeScript multiplexing client with consumer-driven credit, send backpressure, per-channel receive ordering, CLOSE replies and reconnect; OpenAPI/client generation; focused conformance/security tests.
Gate output:
The fast failure is in existing
packages/ui/src/primitives.test.tsmatcher typings (toHaveClass,toBeVisible, etc.); this stream branch did not modify that package.Open blockers: no required TypeScript-client-to-real-container combined e2e; no keystroke p50/p99-under-log-load or bounded-memory log-throughput measurement; persistent exec remains unavailable; OPEN authorization does not yet resolve target visibility/tag filters before scope; adaptive link-sized bulk framing/socket buffer caps and TypeScript liveness/aggregate budget remain incomplete; infraternal-agent grant rejection is not separately audited by the server.
Build job
stream(gpt-5.6-sol/medium) stopped with gaps; its report:Starting finish job for stream protocol step 11.
stream42ace5escripts/verify fast,integration, andbenchCLAUDE.mdanddocs/contracts.md§1.4, §2.2–2.3, §3, and §4 read before implementationStep 11 completion findings and decisions:
packages/streamnow resolves the workspace Vitest 5.0.1 and TypeScript toolchain;bun.lockcontains one Vitest package, and the UI Svelte check is green.OPENbut were closed by the 500 ms reauthorization loop because that loop required unfiltered grants. Stream pairs now retain resolved host tags and recheck both visibility and action scopes witheffective_permission. The real Docker/Podman test proves allowed tags, while an existing hidden target and a missing target return identicalnot_foundpayloads after request-id removal.OPEN_ERR forbiddenreports are audited asstream.grant.rejectbefore relay; failure to persist that audit closes the connection with 1011. A storage-backed unit test verifies the audit record.verify bench.Podman's Docker-compatible follow behavior was checked against the upstream report while diagnosing rootless log delivery: https://github.com/containers/podman/issues/23712. The observed test failure was ultimately the server's tag reauthorization bug above, not a Podman workaround.
Finished step 11 on branch
streamat182eb06c9ecd2dfc2613514eb017903a158c0630(base42ace5eplus merge of currentdev).Delivered:
verify benchSmoke on this dev VM (not reportable performance data):
STREAM_BENCH keystroke_p50_ms=5.984 keystroke_p99_ms=7.038 log_mib_per_s=19.718 peak_rss_kib=24832Final gate output:
Open problems: none for step 11. The dev-VM smoke p99 is above the 5 ms LAN budget; per the repository contract it is not reportable and the authoritative A/B remains the perf-test VM review gate.
Continuation job
stream2(gpt-5.6-sol/medium) finished (28 gate steps PASS, none FAIL):Orchestrator: branch
streampushed to Forgejo at182eb06so it survives the dev VM rebuild. Next (handover): one independent review (Sol medium, defensive-audit wording), one fix round, one verify, then merge. Note the smoke keystroke p99 (7.0 ms) is above the 5 ms budget; the reportable number must come from the perf-test VM.Starting
stream-mergefor step 11.stream182eb06c9ecd2dfc2613514eb017903a158c0630/mnt/hdd/infraternal/.worktrees/streamdevatbfa8af7intostreamwith a merge commit, preserving both stream and Containers/Hosts behavior; regenerate lock/generated artifacts and run the required fast, integration, and browser gates.Merge resolution evidence:
crates/server/src/lib.rs: reviewed both histories withgit log -p 8d5000d..origin/dev -- crates/server/src/lib.rsand the correspondingstreamrange. The resolution keeps the stream ticket route, embedded relay state, and per-OPENauthorization path, plus the Hosts/Containers routes,ResourceState, and resource authorization service. All routers remain wrapped by the shared request/audit middleware.Cargo.toml: kept the dependency union and newertime = 0.3.45with bothformattingandparsingfeatures.crates/server/Cargo.toml: the auto-merge duplicated bothfutures-util.workspaceandtime.workspace;cargo metadata --format-version 1reproduced the parse failure (duplicate key, line 17). Both branches independently added those keys. Removed only the duplicate entries, thencargo metadata --format-version 1succeeded and regeneratedCargo.lockfrom theorigin/devside rather than hand-merging it.docs/openapi.jsonandpackages/api-client/src/schema.ts: replaced the conflicted generated files using the same generator commands ascheck_openapi_drift; regeneratedclient.tsfrom the combined OpenAPI document too.scripts/verify: reviewed both histories and retained both stream and resource integration checks, plus both stream and container benchmark checks.bun.lock:bun install --frozen-lockfileexited successfully. All three packages requestvitest ^5.0.1, and the lock contains one resolved Vitest package,vitest@5.0.1.Fast-gate merge failure and root cause:
scripts/verify fastfailed insetup_banner_uses_stdout_and_database_uses_the_data_directory: the banner was absent. A redacted startup probe showed the server exited first with SQLite code 1555,UNIQUE constraint failed: _sqlx_migrations.version; the database existed but the stream signing key did not.Both branches independently introduced migration version
0007(0007_hosts_and_plan_selector.sqlondev,0007_stream_sessions.sqlonstream). SQLx requires unique migration versions. Because the integration target isdev, its existing version 0007 and checksum remain stable; the stream-session migration is renamed to0008_stream_sessions.sqlwithout changing its SQL. The focused startup test then passed: 1 passed, 0 failed.Independent review found one Important merge interaction: stream OPEN resolution used the pre-step-12
stream_host_tagsbridge and did not enforce the canonical local host ID, while REST resource resolution usedLocalInventory. That allowed an invented host ID to reach the embedded engine and left attachment authorization tied to copied tags.Fix commit:
a4cde1cc5f32b8f01a5929482876adf017efe986(fix(stream): share canonical host authorization).Server::opennow constructs oneLocalInventoryand shares it with both REST and stream services. OPEN resolves host identity/tags through that inventory; paired channels retain the host ID and revalidate against the inventory rather than cached tag copies. The obsolete stream-only tag table and storage bridge were removed, and the real-engine stream E2E now uses the persisted local host ID.RED→GREEN evidence:
Before the fix: failed, received
EngineErrorfor the invented host instead ofNotFound(0 passed, 1 failed). After the fix: 1 passed, 0 failed.Additional verification before commit:
cargo test -p infraternal-serverpassed all non-ignored server tests; the real Docker/Podman TypeScript stream E2E passed 1/1; workspace clippy with-D warningspassed.Finished
stream-mergefor step 11.679940cff994cda3f8865e4d7b9e1b7e7eda64c1182eb06c9ecd2dfc2613514eb017903a158c0630andbfa8af7ffcc146d0d87a14e8bdcf5ed8434eefa4a4cde1cc5f32b8f01a5929482876adf017efe986streamConflict resolution:
crates/server/src/lib.rs: kept both route/state graphs. One sharedLocalInventoryis now constructed inServer::openand passed to the stream and resource routers. Stream tickets, embedded relay state, signed grants, and per-OPEN authorization remain; Hosts/Containers routes,ResourceState, and resource authorization remain. All routes stay under the common request/audit middleware.Cargo.toml: kept one of every workspace dependency, the union of both branches' dependencies/features, and the newertime = 0.3.45withformattingandparsing. The auto-mergedcrates/server/Cargo.tomlduplicatedfutures-util.workspaceandtime.workspace; only the duplicate entries were removed.Cargo.lock: took one side as the regeneration seed and refreshed it withcargo metadata --format-version 1; it was not hand-merged.docs/openapi.jsonandpackages/api-client/src/schema.ts: regenerated from the combined Rust API with the commands used bycheck_openapi_drift; the generated client was refreshed too.scripts/verify: retained both stream and resource integration checks and both stream and container benchmark checks.bun.lock:bun install --frozen-lockfilecompleted with no changes at the final head. The lock has exactly one Vitest resolution:vitest@5.0.1.Merge-caused fixes after the merge commit:
0c394e9f08b978bc84d01013f77cc92fe4147e77: both branches independently added SQLx migration version 0007. Preserveddev's existing0007_hosts_and_plan_selector.sqlchecksum/order and renamed the incoming stream-session migration to0008_stream_sessions.sql. The startup regression changed from 0 passed/1 failed to 1 passed/0 failed.a4cde1cc5f32b8f01a5929482876adf017efe986: independent review found stream OPEN resolution still used the old stream-only host/tag bridge. Stream and REST now share the canonical inventory; invented host IDs returnnot_found, and attachment revalidation reads canonical host tags instead of cached copies. The focused WebSocket regression changed fromEngineError(0 passed/1 failed) toNotFound(1 passed/0 failed); the real Docker/Podman TypeScript stream E2E passed 1/1.Independent review: one Important finding (canonical host/tag authorization mismatch), fixed in
a4cde1c; no Critical or Minor findings were reported.Final gate output at
a4cde1cc5f32b8f01a5929482876adf017efe986:scripts/verify fastscripts/verify integrationscripts/verify browserIndependent review
review-streamstarted.streamdeva4cde1cc5f32b8f01a5929482876adf017efe986git diff dev...a4cde1c/mnt/hdd/infraternal/artifacts/review-stream/.I read the operating contract and governing design/plan/UI/protocol contracts before inspecting the diff or issue history.
review-stream findings
Review head:
a4cde1cc5f32b8f01a5929482876adf017efe986Findings are appended as they are confirmed. Each entry includes severity, contract impact, code evidence, and a reproducing test or command.
F1 — TypeScript client does not enforce the channel state machine (blocks-merge)
packages/stream/src/index.ts:389-421handles only a subset of frame types and does not maintain EOF/open-ack state or validate JSON control payload schemas. A secondOPEN_OKresets channel credit,EOFis silently ignored so laterDATAis accepted, and malformedCLOSEJSON is echoed and accepted. Contract §3.4 requires invalid ordering to close 1002; §3.5 requires malformedOPEN_OK,OPEN_ERR, andCLOSEJSON to close 1002.Reproducer:
client_protocol_failures.test.ts, first three tests.Command:
F2 — A failed disconnected
open()leaks channel slots and aggregate receive budget (blocks-merge)StreamClient.open()inserts the channel and incrementsreservedReceiveWindowbeforesendFrame()checks socket state. The thrownstream is not connectederror leaves both mutations behind. Repeating a recoverable caller error permanently exhausts the 256-channel cap and retains objects.Reproducer:
client_protocol_failures.test.ts,does not retain channels and receive budget when open is attempted disconnected.F3 — Retired channel tracking grows without bound for the life of a socket (blocks-merge)
StreamClient.retiredChannelsis aSet<number>and every completed channel id is retained until reconnect. Sequential ids can be represented as parity intervals (the Rust codec already does this), but the browser client consumes memory proportional to total historical channels. This violates the bounded-memory requirement for long-running stream connections.Reproducer:
client_protocol_failures.test.ts,keeps retired-id tracking bounded across sequential channel churn.F4 —
connect()never settles if the WebSocket closes beforeopen(blocks-merge)StreamClient.connect()resolves ononopenand rejects ononerror, but itsonclosehandler only callsstopConnection; it does not reject the pending connection promise. A ticket can be consumed and the connection then close during handshake, leaving every caller awaiting forever instead of retrying with a fresh ticket.Reproducer:
client_protocol_failures.test.ts,rejects connect when the socket closes before OPEN.F5 — Tag changes are not observed by OPEN authorization or the 500 ms recheck (blocks-merge)
StreamServiceauthorizes againstservice.inventory.host().tags(crates/server/src/stream.rs:446-463andclose_uncovered_pairs), butLocalInventory.hostis a startup snapshot (crates/server/src/resources.rs:65-86,181-183). It is never refreshed from the durable host row. A committed tag change therefore does not apply to the next request as §2.2 requires, does not bump the effective stream decision, and is invisible to the 500 ms recheck.Runnable repro:
server-repros/src/lib.rs::tag_change_applies_to_the_next_open_without_restartupdates the durable host tags to match the token, then opens aneventschannel without restarting.F6 — Authorization withdrawal detaches but does not terminate a persistent exec session (blocks-merge)
The 500 ms recheck calls
close_uncovered_pairs, which sends an ordinary channelCLOSEto the embedded infraternal agent (crates/server/src/stream.rs:1141-1204). Indrive_persistent_exec,FrameType::Closeonly removes the current attachment (crates/agent/src/lib.rs:1288-1293); it deliberately leaves the PTY and process alive. Contract §2.3 instead requires persistent sessions to end when their owner loses the grant, not merely detach.Runnable real-Docker repro:
server-repros/src/lib.rs::authorization_loss_terminates_persistent_exec_processopens a persistent/bin/sh, removes the owner'scontainers:execrole grant and bumps its epoch, waits for the recheck, then inspects the real container process table.F7 — OPEN failures after authorization are recorded as accepted, without the failure error (blocks-merge)
open_channelwritesstream.open / acceptedas soon as authorization passes (crates/server/src/stream.rs:538-576). The relay later forwards infraternal-agentOPEN_ERRresponses, but only the specialforbiddengrant-rejection case writes another audit record (audit_agent_grant_rejection). Engine, host-offline, resource-exhausted, not-found-on-session, and other post-authorization failures therefore have no failed/rejected audit record or error code, contrary to §4.Runnable repro:
server-repros/src/lib.rs::failed_open_is_audited_with_its_error_before_relayuses the implementedstatsfailure path, observesOPEN_ERR engine_error, then queries its request id.F8 — TypeScript client has no inbound control-frame rate limit (blocks-merge)
StreamClient.dispatchprocesses every control frame with no token bucket or unprocessed-control accounting. A peer can exceed the contract's 200/s, burst-400 limit without a 1008 close; channel-zeroPINGalso forces an immediatePONGfor every input, amplifying work.Reproducer:
client_protocol_failures.test.ts,closes 1008 when the peer exceeds the 400-control-frame burst.F9 — Grant issuer identity and key rotation are not implemented (later, before remote tunnels)
The server emits the constant
iss: "infraternal-server"(crates/server/src/stream.rs:583) rather than a persisted server id, andGrantVerifierstores only the verifying key, infraternal-agent id, and tunnel nonce; it never checksclaims.iss(crates/agent/src/lib.rs:143-211). There is also no rotation/key-id mechanism:GrantSigner::load_or_generateloads one raw 32-byte file, silently creates a new key if it is missing, and fails startup if it is corrupt. That is adequate for the current embedded infraternal agent, but it does not deliver the claimed rotation behavior or the §3.9 server-id binding needed by enrolled remote infraternal agents.Runnable evidence:
server-repros/src/lib.rs::grant_verifier_rejects_a_signed_claim_from_the_wrong_issuersigns a structurally valid grant whoseissis a different server id; verification succeeds.F10 — Logs/events channels do not reply to client CLOSE, so clients cannot retire them (blocks-merge)
run_logsandrun_eventspass every inbound frame toFramedChannel::accept_control, which accepts onlyCREDIT(crates/agent/src/lib.rs:641-658,700-704,786-790). A valid clientCLOSEtherefore just ends the infraternal-agent task and drops the transport. The server's relay task releases its pair when output ends but emits no peerCLOSE, leaving the client-side channel live/local-closing forever and retaining its receive-budget reservation. Contract §3.4 requires the peer to replyCLOSE.Runnable real-Docker repro:
server-repros/src/lib.rs::logs_channel_replies_to_client_closeopens a live log channel, sends a validCLOSE, and waits one second for the reply.F11 — The claimed socket-send cap is not an OS socket cap; the WebSocket/server hop dominates latency (blocks-merge)
upgrade_streamsets Tungstenite'swrite_buffer_size(0)andmax_write_buffer_size(MAX_FRAME_SIZE * 2)(crates/server/src/stream.rs:157-163), but never setsSO_SNDBUFon the TCP socket. Axum/Tungstenite documentswrite_buffer_sizeas the userspace threshold before writing to the underlying stream, andmax_write_buffer_sizeas backpressure only when underlying writes fail; neither caps the kernel socket send buffer: https://docs.rs/axum/latest/axum/extract/ws/struct.WebSocketUpgrade.html#method.write_buffer_size and https://docs.rs/axum/latest/axum/extract/ws/struct.WebSocketUpgrade.html#method.max_write_buffer_size. Contract §3.8 explicitly requires the kernel send buffer to be capped at 2 ms of throughput (4–64 KiB).Diagnostic hop measurements from this dev VM (not reportable performance results), using
server-repros/src/hop_timing.rs,server-repros/src/websocket_timing.rs, andwebsocket_timing.ts:The infraternal-agent framing adds about 0.04 ms at p50; roughly 6.1 ms of the unloaded median appears only when the Bun/browser WebSocket and server relay are added. Thus the browser↔server/relay hop, not Docker/PTY or the embedded infraternal agent, dominates. The implementation provides no per-hop timing in its benchmark, so the exact split inside that dominant hop is unmeasured.
The hot path also performs avoidable per-frame copies despite the backend rule:
packages/stream/src/index.ts:58usesUint8Array.slice,crates/agent/src/lib.rs:920converts each PTYBytesslice withto_vec, andFrame::encodeallocates/copies a new contiguous buffer on every hop. These allocations are not reported by the stream benchmark.F12 — The performance gate passes regardless of the stream latency or memory budgets (blocks-merge)
packages/stream/src/bench.tsonly prints its samples.stream_priority_and_memory_smokeparses and prints them, then unconditionally succeeds; itsVmHWMis the Rust test process's lifetime high-water mark and has no limit or before/after comparison.scripts/verify benchtherefore reports PASS for any p99, throughput, or RSS. There is also no perf-test-VM A/B comparison or hop instrumentation in the gate.Direct command from this review (diagnostic dev VM only, not reportable) demonstrates the false-green behavior:
Even a 103 ms measured p99—over twenty times the ≤5 ms contract budget—returns success. Host load makes this number nonreportable, but it proves the gate does not enforce the budget on any host.
F13 — Stream session metadata has two divergent database sources of truth (later)
Migration
0003_restart_state.sqlalready created the contract'ssession_metadatatable with foreign keys and timestamps, andcrates/server/tests/storage.rs:609-681verifies that table. This change adds a secondstream_sessionstable in migration 0008, and all new persistence code reads/writes only the second table (crates/server/src/storage/mod.rs:1174-1217). The actual stream table has no owner foreign key or timestamps and is not covered by the restart-schema test. This duplicates settled restart state and makes future cleanup, backup/restore evolution, and ownership migrations ambiguous. Consolidate on the existing contract table and add behavior-level restart tests rather than a second schema.F14 — The TypeScript
reconnect()does not resume any channel or persistent session (later if manual resume is the intended API; otherwise blocks-merge)StreamClient.reconnect()finishes every channel, clears all channel/open results and offsets, resets ids, and only opens a fresh socket (packages/stream/src/index.ts:291-304). It never recreates anOPEN, suppliessession_id, or advancesresume_from. The real e2e exercises detach/reattach on one uninterrupted socket, not socket loss. Therefore the implementation supports a caller manually building a new reattach request, but does not deliver the stated TypeScript-client “reconnect and resume” behavior. The API contract should either make manual recovery explicit or the client must retain persistent-channel descriptors and last-consumed offsets across reconnect.Reproducer:
client_protocol_failures.test.ts,reopens a persistent channel with session_id and resume_from after reconnect.Verification summary
Repository gates at review head:
Additional evidence:
The source worktree remained clean, and all Docker fixtures created by the repros were removed.
Starting fix round for step 11.
streama4cde1cc5f32b8f01a5929482876adf017efe986(includesdevat65ba959)scripts/verify fast,scripts/verify integration, andscripts/verify bench, with all review repros retained in the normal suites.Fix-round investigation findings (evidence retained here):
LocalInventory.host, a startup snapshot.SqliteStorage::hostalready returns committed tags; OPEN and the 500 ms recheck will load it and fail closed.forbiddenbefore termination, and call the PTY close path. Owner role/tag loss or user disable terminates; ordinary access-token refresh/expiry does not destroy a user-owned persistent session.stream.open/acceptedis currently written before the infraternal agent replies. Final OPEN audit must beacceptedon OPEN_OK orfailedwith the exact OPEN_ERR code before relay.send; no permessage-deflate is negotiated; the 500 ms auth tick and 6 ms CREDIT scheduler do not gate DATA.iss, and there is no key rotation.session_metadataandstream_sessionsneed a data-preserving consolidation and restart behavior tests.session_idand the last consumedresume_from; the contract will state this.Official references used for the socket/flush finding: Axum
ListenerExt::tap_ioand WebSocket buffer docs, futuresSinkExt::send, and Linux socket(7).Security review blocker requiring ownership decision:
The stream/infraternal-agent termination path now preempts blocked command I/O and its real-Docker full-outbound-queue regression passes. However, the final authoritative
PtySession::close()enterscrates/engine/src/lib.rs::terminate_exec, whose cleanup worker performs unboundedstdin.write_all,flush, andshutdownbefore starting its five-second inspect deadline. A stalled PTY input can therefore hang forever after the persistent session has been removed from the stream registry, leaving the process alive and no completion audit.A reliable fix and stalled-PTY failure injection belong in
crates/engine, which is outside this job's explicit ownership. I requested a narrow ownership extension in the session and will not edit that crate without it. Tokio's cancellation-safety documentation also sayswrite_allis not cancellation-safe, so wrapping only the caller future is not sufficient: https://docs.rs/tokio/latest/tokio/macro.select.html#cancellation-safetyFix-round verification update at
3a07873:scripts/verify fastpassed every stage (web build, fmt, clippy, Rust tests, Svelte/type checks, all Vitest suites, OpenAPI/client drift).scripts/verify integrationwas run twice. Both attempts reached the real engine adapter gate and failed onlydocker_and_podman_share_the_real_engine_contractatcrates/engine/tests/integration.rs:761withevent stream timeout: Elapsed(()); its other 6 real-engine tests passed both times. The second run was made after competing container load cleared. This repeated gate failure is outside fix-stream's owned files (crates/engineis not in scope), so it is now an ownership blocker rather than a hidden retry.Fix-round handoff — BLOCKED outside assigned ownership
Head:
5f4c8738554edfc7c5e17b80e15a220f1875db6c(branchstream, worktree clean)Commits:
58358d8 fix(stream): enforce browser protocol state10a7389 fix(stream): enforce authorization lifecycle3a07873 fix(stream): enforce latency budgets680ef36 fix(verify): expose stream benchmark evidence796d2af fix(verify): run latency gate before criterion5f4c873 fix(stream): await browser benchmark modulePer requested finding:
open()cannot retain a channel slot or receive window.connect().crates/engine::terminate_execperforms unbounded stdin write/flush/shutdown before its 5 s inspect deadline, so a genuinely stalled PTY can still strand the cleanup worker. A cancellation-safe implementation/test belongs incrates/engine, which was not assigned to this job; this was separately reported with evidence.SO_SNDBUF; Linux reports the effective doubled value. The previous adaptive 2 ms ceiling was not implemented, so the contract now states the actual requested OS buffer and makes the benchmark authoritative. Production accepted sockets still haveTCP_NODELAY=0; the benchmark explicitly sets and verifies it, but the production listener is in unassignedcrates/infraternal.scripts/verify benchhas asserteddev|perfprofiles and fails on p99, log throughput, or high-water RSS. Dev: ≤10 ms, ≥25 MiB/s, ≤96 MiB; perf VM: ≤5 ms, ≥50 MiB/s, ≤64 MiB. Metrics print on success. Chromium is the contractual client; Bun remains diagnostic.Later-only items recorded: grant issuer/key rotation and duplicate session metadata tables remain deferred; §3.11 now decides that
reconnect()restores transport only and callers explicitly re-open persistent attachments withsession_id/resume_from.Final-head latency evidence (
scripts/verify bench, dev profile):The paced ~55 MiB/s source is the simultaneous budget gate. The unshaped result is a visible capacity/backlog diagnostic, not a substitute. The major owned latency causes were a self-ready bulk log loop starving interactive work (now yields between bulk frames) and Promise-queue overhead in the TypeScript synchronous DATA path (now fast-paths synchronous consumers). Tungstenite already eagerly flushes (
write_buffer_size=0), permessage-deflate is not negotiated, no relay timer gates DATA, and the 6 ms CREDIT scheduler does not delay DATA.Final gates, verbatim summaries:
Final-head integration result (same failure reproduced on three full attempts):
This integration failure and the two implementation blockers above need explicit ownership extension to
crates/engineandcrates/infraternal; no issue was closed and no branch was pushed.Orchestrator decision on fix-stream (
5f4c873): the in-scope findings are fixed. The unloaded browser keystroke round trip went from p50 6.5 ms / p99 11.8 ms to p50 1.0 ms / p99 1.6 ms. Under paced load it is p99 9.7 ms at 55.26 MiB/s with 18.3 MiB RSS; these are smoke numbers, reportable only on the perf-test VM.The three remaining blockers get one final fix with no new review round, per the PLAN step rules:
fix2-stream: security surface (terminating processes when authorization is withdrawn) →gpt-6-sol/medium.crates/engineand to the listener setup incrates/infraternal/src/main.rs.The Podman event-stream failure at
crates/engine/tests/integration.rs:761also appears on branchcli-mcp(#11), so its root cause is investigated ondevfirst. A verify job follows.Starting final step 11 fix (
fix2-stream). Branchstream, base head5f4c8738554edfc7c5e17b80e15a220f1875db6c. Scope: engine termination and Podman event test, production listener TCP_NODELAY, test-container bounds and VM provisioning for #19. Gates:scripts/verify fast,integration,bench; browser latency remeasurement. No push or issue closure.Final-fix findings (before final gates):
devatbadd796in the requested scratch worktree:docker_and_podman_share_the_real_engine_contractfailed at integration.rs:761 after 72.61 s. Rootless Podman 5.4.2 reportedeventLogger=journald;podman events --since 10m --stream=falsereturned no entries, including after a fresh labelled probe container. Withevents_logger="file"in the rootless user's containers.conf, a fresh probe emittedinit,start,attach,died, andremove. The scratch worktree was removed. The provisioning script now configures file events and Docker's bounded json-file defaults.PtySession::close()returned success whilesleep 3600(PID 7 inside the container) remained alive. Docker's exec inspection describes an attachment that has closed, so termination must target the process group itself after the PTY write stalls.ListenerExt::tap_ioper accepted socket: https://docs.rs/axum/latest/axum/serve/trait.ListenerExt.html .Podman event root cause narrowed further: changing the rootless CLI's
events_loggeralone was insufficient while the socket-activatedpodman.serviceremained on its prior configuration. After restarting that service, a live Docker-compatible/v1.41/eventssubscription received create/start/die records for a fresh probe, anddocker_and_podman_share_the_real_engine_contractadvanced beyond the event assertion. It then exposed that this VM's default journald container log driver yields no records through the rootless Docker-compatible logs endpoint. The real test containers now use boundedk8s-filelogs (Docker uses boundedjson-file); the adapter gate passed the engine suite with those fixture settings.scripts/provision-dev-vminstalls file-backed events and restarts the active Podman service so the fix survives reprovisioning.Final step 11 fix completed on branch
streamat976323658df9f6f65c025b6a7de0d60cb82ff465(base5f4c8738554edfc7c5e17b80e15a220f1875db6c). Worktree clean; no push. Commits:a4e9c90(exec/process and real-engine coverage),0b38521(accepted-socket TCP_NODELAY),2621962(bounded test logs and VM provisioning),8acf0a7/05b25bf(Clippy cleanup),9763236(auto-remove eligible engine fixtures). Issue #19 has the incident path/size and mitigation evidence; it remains open for the orchestrator.Security: cleanup now bounds PTY stdin interrupt/flush/shutdown to 500 ms and drops the transport. It identifies the Linux host PID from engine exec inspection, verifies the PTY process group and namespace PID, and signals that group with SIGKILL through a second exec in the same container when it remains alive. It waits for the host process and engine inspection to finish before reporting success. A cancelled close cannot cancel the dedicated cleanup worker. The real Docker/rootless Podman test starts
sleep 3600, confirms a 16 MiB stdin write is stalled, closes the session, and checks the process is gone within seven seconds. A second full server regression withdraws the owner'scontainers:execgrant while the persistent PTY has large input, then checks Docker and rootless Podman process exit and the acceptedstream.session.terminateaudit. Existing attached/detached authorization tests pass too. Docker's exec API exposes create/start/inspect/resize but no exec-kill operation; Podman's Docker-compatible service offers that API surface. Sources: https://docs.docker.com/reference/api/engine/version/v1.40/ and https://docs.podman.io/en/latest/markdown/podman-system-service.1.html .TCP_NODELAY: production uses axum 0.8
ListenerExt::tap_ioon each accepted TCP socket, with a real accepted-socket option test. Source: https://docs.rs/axum/latest/axum/serve/trait.ListenerExt.html . The final dev-VM browser remeasurement is below (not reportable perf-test-VM A/B): no logs p50 1.0 ms / p99 1.8 ms; paced logs p50 1.2 ms / p99 9.1 ms at 55.12 MiB/s and 18.04 MiB harness high-water RSS. The unshaped diagnostic was p99 40.7 ms at 98.32 MiB/s. The prior dev-VM smoke was no logs p99 1.6 ms and paced logs p99 9.7 ms; the harness already set TCP_NODELAY then, so these samples do not isolate a TCP_NODELAY effect.Podman event root cause: the same timeout reproduced at
devbadd796in the requested scratch worktree, which was removed. This VM's rootless journald events backend yielded zero CLI events; a file backend yielded events after restarting the socket-activated Podman service. The test then revealed that default journald container logs yielded zero records through this rootless Docker-compatible API, so its fixtures now use boundedk8s-filelogging. Provisioning makes file-backed events persistent and restarts the service. The event failure now names the runtime, target, backend, and diagnostic command instead of a bare timeout. Podman documents the events backend setting: https://docs.podman.io/en/v4.4/markdown/podman-events.1.html .Final-head gate output, verbatim:
Starting verify-stream on branch
streamat base head976323658df9f6f65c025b6a7de0d60cb82ff465. I will mergedevat the requested6270e0fhead, independently verify the stream review findings and final blockers, run the required gates, and report evidence and dev-VM smoke latency here.Verify-stream result
Step 11 verified on branch
streamat head9c9fb476be7efb41f729142dbcf6564bc5ef2803. This is the requested merge commit, with parents976323658df9f6f65c025b6a7de0d60cb82ff465and6270e0fa162e6072134d6c9c82489ca71cbcc8ed. No product code fix was needed; the merge resolution retained both branches' gate steps and regeneratedbun.lockandCargo.lock. No push.The 11 blocks-merge findings
OPEN_OK, data after EOF, malformed CLOSE, invalid schemas, and ordering/credit cases.open()slot and receive-budget leak. Disconnected OPEN and socket-send rollback tests passed in the normal stream Vitest suite.connect()hang. The pre-OPEN socket-close test passed; connect rejects.tag_change_applies_to_the_next_open_without_restartpassed in the normal Rust suite.authorization_loss_terminates_stalled_stdin_on_both_enginespassed for Docker and rootless Podman.failed_open_is_audited_with_its_error_before_relaypassed in the normal Rust suite.SO_SNDBUFon its listener; this Linux VM reports 131072 bytes on both the listener and an accepted socket. The contract correctly describes this as an OS buffer request, not a 2 ms cap. The accepted-socket TCP_NODELAY test passed.keystroke p99 14.100 ms > 0 ms. The threshold source was restored byte-for-byte. The unmodified final bench gate then passed its p99, throughput, and RSS assertions.The TypeScript stream suite passed in the fast gate. The real stream regression suites passed in the normal and ignored integration runs. The wrong-issuer Rust repro is the later finding recorded below, not one of these 11 blockers.
Final blockers and #19
accepted_server_socket_has_tcp_nodelaypassed (1/1); production configures each accepted socket.docker_and_podman_share_the_real_engine_contractpassed (1/1). Podman is 5.4.2; the live rootless service reportsEventLogger=fileandActiveState=active. This matches the root cause already recorded here: journald returned no rootless event records, and changing the client config alone was insufficient until the socket-activated service was restarted.docker inspectreturnedLabels={"infraternal.test":"1"} AutoRemove=true LogDriver=json-file LogOpts={"max-file":"1","max-size":"10m"}. Test fixtures also use--rm; source bounds Docker and Podman test logs. The stale sweep inspectsinfraternal.demoand skips value1./etc/docker/daemon.jsonhasmax-size=50mandmax-file=2.Latency (dev-VM smoke only; not perf-test-VM evidence)
p50/p99 0.282/0.602 ms; embedded infraternal agent0.284/0.620 ms.0.509/3.346 ms; embedded framed PTY0.588/4.700 ms(the same test's no-log control was0.597/2.366 ms).1.1/1.6 msp50/p99. Internal server hop was0.476/0.791 ms; agent-output-to-client-queue was0.002/0.019 ms.1.5/7.7 msp50/p99 at55.018 MiB/s; internal server hop was0.527/2.474 ms, queue was0.002/0.018 ms. Harness VmHWM was18.25 MiBand is not real-server RSS.14.3/36.0 msp50/p99 at83.746 MiB/s; it is not a budget gate.One earlier unmodified bench attempt coincided with concurrent Rust compilation and reported shaped p99
31.0 ms; after those compiler processes cleared, the final bench run passed at7.7 ms. The threshold was unchanged.Later findings — record only
issdiffers from the enrolled server id. No code change in this step.session_metadata(migration 0003) andstream_sessions(migration 0008) remain; current persistence usesstream_sessions. No schema change in this step.docs/contracts.md§3.11 saysreconnect()restores transport only and callers explicitly reopen withsession_idandresume_from. The in-suite TypeScript test verifies that manual reattach behavior.Final gate output (verbatim)
scripts/verify fastscripts/verify integrationscripts/verify browserscripts/verify benchOrchestrator: step 11 is merged to
devatbba250fand pushed. The merged tree is identical to the verified head9c9fb47(the diff is empty). The path was:stream,stream2), merge ofdev;fix-stream, then one final fixfix2-streamfor the three out-of-ownership blockers, under the PLAN step rules;Dev-VM smoke, not reportable: keystroke through the browser path p50 1.1 ms / p99 1.6 ms with no load, and p50 1.5 ms / p99 7.7 ms under paced logs at 55 MiB/s. The reportable ≤5 ms p99 check under saturating logs is step 16 on the perf-test VM.
Later, recorded:
reconnect()is manual; the decision is documented indocs/contracts.md§3.11.Now unblocked: 12b (logs view), 13b (exec in the UI), 14b (SSH in the product) and the CLI's
logs/exec/ssh(15).