Step 11d: stream params for logs and exec, and stdin EOF #29
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found by step 15c (
cli-stream, #11). The stream protocol'sOPENparams lack:tail(last N lines) andfollow(keep streaming, or stop at the end), plussinceandtimestampsif the engines support them. Today the CLI and web can't ask the engine for these.argv(the command to run) andtty(true or false), plusenvandworkdirif sensible, per the Docker and Podman exec APIs.catstays open until Ctrl-C. Define EOF in the contract (for example a half-close frame) and deliver it: close the exec's stdin for non-TTY, and send^Dor close for TTY, per the engine docs.Contract first (
docs/contracts.md§3.7 and the protoExecParamsandLogsParams), then the server, agent and engine, then the CLI and web use them. Backend step;packages/streamand the CLI adopt it after.Starting step 11d on branch
stream-params, based ondevHEADee62153839982d068586e3890bfe8749db8f1dab. I’m reading the settled stream contract and tracing logs/exec through proto, server, infraternal agent, engine, andpackages/stream. I’ll commit and post the §3.7/§3.11 plus proto contract first, then implement the changes and run the requested Docker/rootless Podman tests and gates. I’ll keep changes confined to logs and exec paths.Finding from the existing paths and engine references:
run_logsincrates/agent/src/lib.rscurrently forcesfollow: true, andBollardEngine::stream_logsincrates/engine/src/bollard_adapter.rsalways asks the engine fortimestamps: true. The exec adapter always createsTty: true, withrun_execdefaulting to/bin/sh. The new optional defaults will preserve those behaviors (follow=true,timestamps=true,tty=true, omittedargvuses/bin/sh).Docker's logs endpoint exposes
follow,tail,since, andtimestamps, while exec create acceptsCmd,Env,WorkingDir,AttachStdin, andTty: https://docs.docker.com/reference/api/engine/version/v1.40/ and https://docs.docker.com/reference/cli/docker/container/logs/.Podman's current system service documents its Docker v1.40 compatibility API; its logs docs confirm
follow,since,tail, andtimestamps: https://docs.podman.io/en/latest/markdown/podman-system-service.1.html and https://docs.podman.io/en/latest/markdown/podman-logs.1.html. The Podman compatibility API schema documents the matching exec create fields: https://docs.podman.io/en/latest/_static/api.html.Contract-first slice committed:
46f6c64833b4f715b3d6a84a450eb1f43cb6b439.Contract excerpt (§3.7/§3.11): logs accept bounded
tail,follow,sinceandtimestamps; omittedfollowandtimestampspreserve their currenttruedefaults. Exec accepts boundedargv,tty,envandworkdir; omittedttypreserves the web terminal's PTY behavior.EOFshuts down stdin for non-TTY exec and sends EOT (0x04) to TTY exec while output continues. The contract maps these to the Docker and Podman APIs with citations.Proto gate:
cargo test: 56 passed (3 suites, 1.66s).Real Docker and rootless Podman engine adapter fixtures pass. The stream E2E then exposed a Docker edge case: a short
tty:truecommand exits before the initial PTY resize and Docker returns HTTP 500. The adapter now treats a resize error as benign only if exec inspection confirms the process has exited; running execs still return the resize error. I also added stage and target labels to stream E2E open failures so later integration evidence identifies the failing case.Step 11d complete.
Branch:
stream-paramsHead:
46763d786fe34fd7d51ecf100d8963e34614f398Contract excerpt (
docs/contracts.md§§3.7, 3.11):API references in the contract: Docker Engine API, Docker logs options, Podman REST API, Podman service API modes, and Podman logs options.
scripts/verify fastoutput:scripts/verify integrationoutput:No push;
progress.mdremains excluded and uncommitted.Independent defensive review started. Branch: stream-params. Base:
ee62153. Review head:46763d7. Scope: stream OPEN log/exec params, stdin EOF, authorization, compatibility. Source remains read-only.Independent defensive review started. Branch: stream-params. Base:
ee62153. Review head:46763d7. Scope: stream OPEN log/exec params, stdin EOF, authorization, compatibility. Source remains read-only.Independent review of
ee62153..46763d7(stream-params)F1 — blocks-merge: EOF can overtake a pending DATA send
Evidence:
packages/stream/src/index.ts:319-345letssend()wait asynchronously for input credit, whileeof()immediately sends an EOF and setslocalEof. When credit later arrives, the waitingsend()rejects at line 336 and never sends its bytes. This violatesdocs/contracts.md§3.11's last-DATA-before-EOF rule and loses caller input. A targeted Bun probe using the unmodified StreamClient and an OPEN_OK with one byte of input credit producedframes=["Open","Eof"]andsendError="Error: channel is closed"after a waitingsend("must-arrive"),eof(), then CREDIT. The probe is/tmp/review-stream-params-eof-race.ts(outside the worktree).Test case: On each real Docker and rootless Podman fixture labelled with a unique
infraternal.testrun id, open non-TTY/bin/cat, constrain client input credit to one byte through a bounded stream test transport, callconst pending = channel.send(marker), thenchannel.eof()before issuing the remaining credit. Assert the complete marker arrives before output EOF,pendingresolves, and CLOSE has exit code zero. Also assert only one client EOF appears on the wire.F2 — blocks-merge:
follow:falsecan remain open indefinitely if an engine ignores the optionEvidence:
crates/agent/src/lib.rs:919-1007passesfollow:falseto the adapter, then loops onstream.next()until it returns None. There is no snapshot deadline, output bound, or other completion path for a compatible engine that ignores the flag. With a live container, the channel can stay open forever and never send the promised output EOF and normal CLOSE fromdocs/contracts.md§3.7. The existingpackages/stream/src/e2e.tscovers compliant engines only.Test case: For separately labelled, bounded Docker and rootless Podman containers, place a failure-injection REST proxy between Bollard and the real engine that forces the engine logs request to
follow=truewhile the client OPEN requestsfollow:false; emit a fixed small history and then keep the container alive without further output. Assert the stream finishes with EOF and normal CLOSE within a defined bound and returns exactly the history. The currentstream.next()wait does not finish.F3 — blocks-merge: audit cannot identify an exec OPEN or record its argv shape
Evidence:
crates/server/src/stream.rs:1067-1081and:1687-1711audit all channel kinds asaction="stream.open"with only a target, outcome and error code.AuditEventandaudit_recordshave no channel-kind or argv-shape field (crates/server/src/storage/mod.rs:217-226, migration0001_auth.sql:89). Thus a successful logs OPEN and a successful exec OPEN on the same container produce indistinguishable audit records apart from request id and time; exec argv count/shape is absent. This fails the job's requirement to record that an exec happened, with its argv shape, while redacting values. The new Debug implementations redact values but do not solve audit attribution.Test case: On each labelled Docker and rootless Podman fixture, OPEN logs and then exec with
argv=["/bin/echo", "secret-marker"],env=["SECRET=secret-marker"], andworkdir="/tmp". Query the durable audit rows by request id. Assert the exec row has a distinct action/kind and a safe argv shape (e.g. argument count and byte lengths), while no row, log or error containssecret-markeror the environment value. The current schema cannot satisfy the kind/shape assertions.Checked without a finding
Proto limits cover argv count, per-entry/total UTF-8 bytes, NUL, env key syntax/uniqueness, workdir length and NUL, tail and since range; the server and infraternal agent call that validation.
ExecOptionsmaps command to Bollardcmd, leavesuser=None, and does not expose container or privilege selection. Existing scopes and the final per-OPEN authorization check still apply; direct argv does not add a capability beyond the prior interactive shell for anai_agenttoken. Omitted params retain the web terminal defaults. OpenAPI and generated TypeScript client add optional fields. Real-engine gate result follows below.Gate output (verbatim)
scripts/verify fast:scripts/verify integration(run once, started with load5 4.39 < 8):Review head:
46763d786fe34fd7d51ecf100d8963e34614f398. Source changes: none; commits: none.Job #29 fix round started.
Branch: stream-params
Base head:
46763d786fI am inspecting the three review findings and will add real Docker and rootless Podman regressions before the fast and integration gates.
Fix-round findings and evidence:
/bin/catfixture checks that no DATA/EOF escapes before added credit, then verifies byte-exact DATA precedes one EOF. The first assertion also saw client CREDIT frames from echoed output; it now permits those while preserving the ordering check.follow:falsesnapshots end on engine completion, the OPEN-time timestamp edge, or a five-second monotonic deadline. The real-engine test forcesfollow:trueagainst labelled, bounded Docker and rootless Podman fixtures; both passed and received clean EOF/CLOSE within the bound.details_jsonidentifies channel kind and, for exec, argument count, program basename, effective TTY, and environment entry count. It omits argument values, environment values, and workdir. Malformed OPEN rows usechannel_kind: unknown. Migration 0012 appends the field. Docker/Podman stream e2e queries persisted rows and checks log/exec distinction and redaction.Test fixtures use per-run labels and bounded logging. The stale-container sweep that could remove another run's containers was removed. Fast and integration both passed once after the main fixes; I am rerunning them after the malformed-OPEN metadata addition.
Fix round complete.
Head:
bb5ab185e79ee08b991b74ae708fdacde31ca931(branchstream-params; not pushed)./bin/catregression verifies the held send, exact bytes, and one EOF.follow:falsestops when the engine ends, a timestamp passes the OPEN-time boundary, or the five-second monotonic deadline fires. Forced-follow real-engine coverage passed on Docker and rootless Podman and checked clean EOF/CLOSE.details_jsonwith channel kind and safe exec shape (argument count, program basename, TTY, environment entry count). Cross-engine coverage distinguishes logs from exec and verifies argv values, environment key/value, and workdir are absent.Fixtures carry per-run labels and bounded logging. Removed the stale-container sweep that could select another run's fixture.
progress.mdremains ignored and uncommitted.scripts/verify fastoutput:scripts/verify integrationoutput:Starting verification on branch
stream-paramsatbb5ab185e79ee08b991b74ae708fdacde31ca931; integratingdevata5fec35ee358542e471fd2ad9e019e84dd458bc0. I will renumber the unmerged stream audit migration to0014_stream_open_audit_details.sql, then verify the upgrade path and review cases on Docker and rootless Podman.Finding:
origin/devalready contains SQLite migrations0012_ssh_hosts.sqland0013_stream_session_creator.sql; the branch-only audit details migration was renamed to0014_stream_open_audit_details.sqlin commite363c9dbefore merging. The merge is in progress; the current remaining content conflict is limited to migration version assertions, which must account for versions 1–14.Post-merge compile check found two SSH stream failure branches in
crates/server/src/stream.rsstill called the audit helper without the mergeddetails_jsonargument. Both now pass the OPEN's safe audit details; generated OpenAPI/client output is still pending because the first generator invocation failed at compile time.Another merged-source compile mismatch was the new dev CLI stream constructor omitting this branch's optional exec fields (
argv,tty,env,workdir). It now usesNonefor each, preserving the existing interactive shell defaults.Migration upgrade verification passes from a database initialized with the exact 13 SQLite migrations on
origin/dev(a5fec35):upgrades_dev_a5fec35_database_with_stream_audit_detailsreported1 passed; 0 failed. After server startup it recorded versions 1–14 and the pre-existing audit row haddetails_json = '{}'. OpenAPI and the TypeScript client were regenerated from the merged proto/server types.Integration commit:
53ccddd(merge oforigin/devata5fec35intostream-params). The migration collision is resolved: dev migrations 0012/0013 remain unchanged and stream audit details is 0014. The migration upgrade test from dev’s exact 1–13 history passes. Merge compile mismatches in SSH audit failures and the CLI exec constructor were fixed; OpenAPI and the API client were regenerated from merged Rust types.Integration finding: real-engine checks passed through resource HTTP, but
scripts/verify integrationfailed incli_and_mcp_use_the_public_http_contractatcrates/infraternal/tests/cli_mcp.rs:2006: expected exit codeSome(0), receivedSome(7). The gate also emitted 10,009 ms slow-command markers for the Docker and Podmanlogscommands; stream exec RTTs were 166 ms (Docker) and 294 ms (Podman). I am tracing the failing command before recording a final status.Root cause for the CLI integration failure: stderr reported
Could not write command output: Broken pipe.read_cli_stdout_exactreturned as soon as it had the expectedcatbytes and dropped the child stdout reader, whileobserve_exec_stdin_eof_behaviorcontinued waiting for EOF behavior. I am changing the test helper to keep draining stdout until the child exits, so it can observe the existing PTY EOF limitation without manufacturing a CLI output failure.The targeted compiled CLI/MCP integration test now passes after commit
9f07684, which keeps the test's stdout reader draining while it observes pipedcatEOF:test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out. It runs against real Docker and rootless Podman. The CLI's PTY-basedexec -ipath reportsSTREAM_EXEC_STDIN_EOF=blocked_by_current_ptyand is interrupted as the test expects; the separate TypeScript stream/bin/cattest proves byte-exact DATA-before-EOF on both engines.Integration finding: a rerun passed Docker/Podman engine adapters and the bounded
follow:falsesnapshot test, then the TypeScript stream E2E's unrelatedtail:5assertion got 4 records on Docker. The fixture currently emits another line every 50 ms, while the finite snapshot excludes records timestamped after OPEN; this can move a boundary-crossing line into the engine's five-record tail and leave four after filtering. I am stabilizing the real-engine fixture by emitting finite history and waiting until the runtime CLI sees it before opening the tail snapshot.The flaky tail fixture is now finite and waits until the real Docker/Podman logs APIs expose all eight initial records before the stream server starts. Targeted rerun passed:
test typescript_client_streams_logs_and_exec_for_docker_and_podman ... ok;test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 4 filtered out; finished in 6.59s. This rerun covers both engines, the credit-blocked/bin/catDATA-before-EOF assertion, and safe logs/exec audit details.Verification result
Final verified head:
db9d877301b8c0f1c2f9b68f93eaedb0a995c005. Merge commit:53ccddda721ef6092b99736c6aaf2595d7fd33ce(integrateddevata5fec35ee358542e471fd2ad9e019e84dd458bc0).The unmerged
0012_stream_open_audit_details.sqlwas renumbered to0014_stream_open_audit_details.sql; dev migrations0012_ssh_hosts.sqland0013_stream_session_creator.sqlremain unchanged. The upgrade test starts from dev's exact versions 1–13, upgrades to 14, and verifies the pre-existing audit row getsdetails_json = '{}'./bin/catstream E2E holds back one byte of credit, verifies no DATA/EOF was sent, grants credit, then checks byte-exact output and DATA-before-one-EOF frame order.follow:falsesnapshot test forcesfollow:trueunderneath on Docker and rootless Podman; it receives the three existing records and clean EOF/CLOSE within the five-second bound plus the test tolerance.echo,tty:false, and one env entry; assertions find no argv/env marker, environment key, or/tmpworkdir in audit text.The browser gate passed against the production release server, including the Ghostty exec terminal. The compiled CLI/MCP test passed against the real server and both engines; its PTY-based
exec -iEOF observer reportsSTREAM_EXEC_STDIN_EOF=blocked_by_current_ptyand interrupts as the test expects. The same test emitted 10,009 ms slow-command markers for Docker and Podmanlogs, while its assertions passed. The SSH contract and SSH product stream tests both passed against isolated realsshd.scripts/verify fastoutputscripts/verify integrationoutputscripts/verify browseroutputNo push; issue remains open.
Orchestrator: step 11d is merged to
devat1337efdand pushed. Gates on the exact merge head: fast 15/15, integration 15/15. Migration renumbered to0014. Follow-up: the CLI'sexec -ishould send the new stdin EOF (15c today observesblocked_by_current_pty).