bug: Prevent remote 32 MiB stdin uploads stalling near 8 MiB #77
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem and evidence
Two real 32 MiB remote stdin uploads closed the infraternal agent tunnel: Docker stopped after 9,158,656 bytes (8.73 MiB), Podman after 8,159,232 bytes (7.78 MiB). Both reached zero input credit and ended as host_offline. Load1 was 28.79 and 29.79 respectively. An earlier independent run also stalled. The client deducts granted credit before sending DATA (crates/infraternal/src/stream.rs:392-405); inbound batches cross the durable authorization fence before enqueue (crates/server/src/storage/mod.rs:2011-2034).
The exact runs are in continuation-results.json. The nearest CPU flamegraph is named-stream-output.svg; it is not a profile of those exact failed uploads.
Recommendation
Trace DATA/CREDIT scheduling and the tunnel-close cause on the remote path. Add a real, byte-exact 32 MiB stdin regression for both Docker and Podman, and keep the signed command, authorization, revocation and admission fences unchanged.
Payoff, effort and risk
Expected gain: complete all 32 MiB without a reconnect; no throughput gain is credible until correctness passes.
Quick win vs deeper change: a deterministic reproduction and event trace is the ≤1-day triage; the root-cause fix is estimated at 1–3 days.
Risk: high, because this crosses tunnel and authorization security surfaces. Independent security review is required.
Priority: Now (≤1 day for triage; correctness blocker).
2026-10-04 owner decisions: Matches the decision to keep the custom mTLS WebSocket plus binary-frame protocol and unify embedded and remote infraternal agent paths. Preserve signed RPC and revocation fences. Related: #60 (stream boundary); #37 covers stdin EOF, a different failure.
Report: perf-audit report
Source: perf-audit 2026-10-04
Starting fix-stream-bugs as the only writer. Branch:
fix-stream-bugs; base/head:87bb5bd2b7410d3a571a903f6cf6a53d1112ca6f. Scope: reproduce #77/#78 with the audit stream driver, fix byte correctness without raising bounds or bypassing authorization, add real Docker/rootless Podman embedded/remote SHA-256 regressions, repeat ten times, run only the requested fast gate. No push or merge.Audit-driver reproduction (audit release
d1eefa9, default 256 KiB window, 16 KiB DATA): remote Docker stdin failed after 7,520,256 bytes; rootless Podman after 8,093,696, inputCredit=0 and typedhost_offline. All four raw-PTY paths returned exit 0 and zero bytes for a 33,554,432-byte NUL payload (SHA-256 empty hash). Embedded Docker stdin completed both 32 and 256 MiB exactly. Root causes under verification: per-DATA input CREDIT emission incrates/agent/src/lib.rs:2322exhausts unchanged tunnel ControlBucket (crates/server/src/tunnel.rs:913); Bollard 0.21.1read.rs:55infers raw/multiplexed mode from byte[0] and consumes NUL payload as empty headers (exec.rs:270), called bycrates/provider-bollard/src/bollard_adapter.rs:622. Patch will coalesce consumed credit and explicitly use a raw TTY transport. Source research: https://docs.rs/bollard/0.21.1/src/bollard/exec.rs.html and https://docs.rs/tokio-util/0.7.19/tokio_util/io/struct.ReaderStream.html . Full before output retained in/tmp/fix-stream-bugs-audit-before.log; final evidence will include the base-head regression.Pristine base
87bb5bddebug reproduction also fails: remote Docker 32 MiB stdin stopped after 10,862,592 bytes; rootless Podman after 11,091,968. Both reporthost_offline. Embedded stdin produced 2,048 CREDITs for 32 MiB and 16,384 for 256 MiB (one per DATA), explaining the tunnel control-rate violation rather than a byte-window exhaustion. Raw TTY all-NUL output returned zero bytes / exit 0; arbitrary binary returned 33,554,424 rather than 33,554,432 bytes (first eight bytes removed), with SHA-256473c7eb5f73f4920d5f1615d5a9118128d04f6a9fd98aee25bdf029674f4a2f5instead ofe09320c5b00b34bb704802136c599a95b3996332ba84d7c7f21112b6231b6bd0. Some immediate cases following a dropped tunnel gotnot_foundduring reconnection; these are recorded as failures, not byte-corruption evidence. The audit-binary driver independently reproduced zero NUL output on all four owners/engines.After the raw transport fix, the compiled CLI regression independently exposed shared terminal file-description flags:
AsyncStdin::newset O_NONBLOCK on inherited stdin, also affecting stdout, which returned EAGAIN (Resource temporarily unavailable), truncating binary output and returning CLI status 7. Reopening the same terminal during setup isolates flags; all 24 PTY cases (two patterns × three clients × two engines × two owners) then passed exact SHA-256 with exit 0. No per-frame work is added to the CLI. Linux semantics: https://man7.org/linux/man-pages/man2/dup.2.html .Targeted final-code security check passed: audited forbidden CLOSE receipt 78.39 / 49.85 / 19.60 / 61.82 ms for embedded Docker/Podman and remote Docker/Podman during active bulk uploads; physical tunnel stop produced audited host_offline in 25.32 ms. This is receipt evidence conservatively below C+850 ms. Final ten-repeat matrix and fast gate are next.
Completed on branch fix-stream-bugs. Head:
e133ee4196306f037284769128c816877d5ea3e5; base:87bb5bd2b7410d3a571a903f6cf6a53d1112ca6f. Two logical commits, no push. Ready for independent review; issues remain open.#77 root cause: base
crates/agent/src/lib.rs:2322returned one tunnel CREDIT per 16 KiB stdin DATA. The unchanged control bucket atcrates/server/src/tunnel.rs:913rejected the resulting control rate and withdrew the physical tunnel. This was control-rate exhaustion, not a larger stdin window requirement. Fix: coalesce consumed native input credit at six-ms intervals (crates/agent/src/lib.rs:2298), arming the timer only while credit is pending. No larger limits, additional per-frame allocation or copying; retain the 256 KiB window, bounded queues, native write timeout and revocation fences.Before: the original audit driver/product reproduced remote Docker/Podman upload stalls at 7,520,256 / 8,093,696 bytes, zero remaining credit and typed host_offline. The new regression against pristine base dev debug also failed, at 10,862,592 / 11,091,968 bytes. Its immediately following 256 MiB opens returned not_found while the tunnel reconnected; those are not byte-loss measurements. Embedded base 32/256 MiB completed but returned 2,048 / 16,384 CREDIT frames. After: all 80 stdin cases passed SHA-256 and exit 0 (32/256 MiB, both owners and engines, ten repetitions each); credit frames 127–315 / 1,026–2,323, unchanged window and tunnel generation.
#78 root cause: base
crates/provider-bollard/src/bollard_adapter.rs:622used Bollard's byte-sniffing exec decoder for raw TTY data. Leading 0/1/2 were mistaken for multiplex headers. Fix: choose framing from TTY mode (bollard_adapter.rs:614), read native raw output with the same 8 KiB capacity and 32-slot pump, preserve response-prefetched bytes in the BufReader (native_exec.rs:65); keep non-TTY demultiplexing. No payload copying or new per-frame work. Also fixed CLI terminal setup atcrates/infraternal/src/cli.rs:1884: reopening terminal stdin isolates O_NONBLOCK from stdout's inherited shared file description. Merely dup'ing it does not isolate file-status flags.Before: the audit driver delivered zero bytes for 32 MiB all-NUL PTY output in all four owner/engine routes, exit 0. The new regression on pristine base failed: embedded NUL output was zero bytes in all clients; arbitrary binary audit/browser output lost the first eight bytes (33,554,424 instead of 33,554,432), and CLI output could truncate with EAGAIN/exit 7. Remote opens immediately after the stdin-induced disconnect could return not_found. After: all 240 PTY cases passed exact byte count, SHA-256 and exit 0 (all-NUL and all-byte patterns, audit/browser/CLI, both owners and engines, ten repetitions each).
Real final-run security evidence: audited typed forbidden CLOSE receipt/queue upper bounds were 59.509 ms embedded Docker, 111.640 ms embedded Podman, 93.026 ms remote Docker and 115.144 ms remote Podman, all below C+850 ms. Stopping the owned physical tunnel produced audited typed host_offline in 25.227 ms. No transfer replaced the tunnel nonce generation. Cleanup passed, and both engines have zero containers carrying infraternal.test=fix-stream-bugs. No demo containers touched.
Evidence:
target/fix-stream-bugs/fix-stream-bugs-1791142648256.jsonlcontains 320 passing transfers, four revocations, one host_offline and final pass/cleanup record. Before logs:/tmp/fix-stream-bugs-audit-before.log,/tmp/fix-stream-bugs-base-before.log. Final gate logs:/tmp/fix-stream-bugs-ten.log,/tmp/fix-stream-bugs-fast.log. Native codec tests passed (2), including prefetched binary bytes/half-close and bounded malformed responses. Added a real-engine native TTY benchmark to verify bench, deliberately ignored in fast; perf-test-VM A/B was not run and these shared-VM completion timings are not performance certification. Used the sccache debug preset, no release build, never verify merge.Primary references: Bollard exec source, Docker exec-start API, ReaderStream, dup file-status flags.
Exact regression gate output (exit 0):
Exact required
INFRATERNAL_VERIFY_RUN_ID=fix-stream-bugs scripts/verify fastoutput (exit 0):Read-only review started: detached worktree review-stream-bugs at
e133ee4196; diff base87bb5bd. Scope: stdin credit bounds/EOF, explicit raw PTY/non-TTY framing, CLI stdin, audited revocation and host_offline. Debug sccache preset only; no commits, push, or verify merge. Independent targeted tests and real-engine binary relay matrix will be recorded.merge
Read-only final review of
87bb5bd...e133ee4, heade133ee4196306f037284769128c816877d5ea3e5, for #77/#78. No blocker or major findings. No introduced minor correctness finding identified. This is the requested scoped review verdict, not a substitute for the orchestrator's merge gate or perf-VM certification.Evidence and reasoning
crates/agent/src/lib.rs:2244–2345: active processing preservesrecv_credit + consumed_input == 262144. DATA is checked against remaining credit before the unchanged bounded provider-input send. Accepted bytes move intoconsumed_input; a flush returns that exact sum once, so neither counter can overflow or grant extra bytes. The timer remains enabled with any positive pending sum, including after EOF; a completed/closed channel retires its input and no longer needs a final CREDIT. No new batch buffer. Native input timeout, payload limit, channel capacities, relay per-channel/connection accounting, and 200/s control bucket are unchanged. Independent real tests covered EOF at 0, 1, 16,385 and 262,145 bytes, plus a one-byte idle credit return, for every Docker/Podman and embedded/remote combination.crates/provider-bollard/src/bollard_adapter.rs:610–650andnative_exec.rs:31–104: TTY create/start both request true and choose a raw byte reader, preserving header-prefetched bytes through BufReader. Non-TTY create/start both request false and retain Bollard's multiplex-header demultiplexer beforeLogOutput::into_bytes. Four independent non-TTY runs each recovered exactly 64 MiB of NUL bytes, 32 MiB written to stdout and 32 MiB to stderr, with SHA-256 and exit 0; no native headers leaked. All 24 new 32 MiB raw PTY runs passed exact count/hash/exit through audit observer, shipped StreamClient in Chromium, and the compiled CLI, for both NUL and arbitrary binary payloads.crates/infraternal/src/cli.rs:1881–1938: terminal stdin is reopened once to isolate O_NONBLOCK from shared stdout. Non-TTY inputs retain duplicate-descriptor reads, flag restoration, regular-file/dev-null fallback, and the existing EOF loop. Sixteen independent real CLI runs passed: nonempty pipe, empty pipe, regular file, and /dev/null, on every engine/owner pair.forbiddenCLOSE; receipt was 66.378–110.379 ms after commit (conservatively proves C+850 ms). Stopping the owned remote infraternal-agent process produced typed, auditedhost_offlineCLOSE in 16.849 ms. Tunnel nonce counts stayed unchanged during successful bulk transfers.Independent verification (debug, sccache preset)
The temporary, ignored driver copies the new real-engine test, changes only import paths/run/output names and the container label, selects one 32 MiB stdin upload per engine/owner (four total, including one remote Docker and one remote Podman), and adds the edge checks described above. Original PTY clients, credit/window assertions, nonce, revocation and host_offline checks remain intact. This is a targeted diagnostic, not a full verify tier;
verify mergewas never run. Initial matrix attempt timed out during the inherited 10-second server-readiness fixture, before transfers; unchanged retry passed. Separate unchanged startup diagnostic passed at 8.705 seconds. An initial CLI unit-test invocation used--libagainst a binary-only package and was corrected to the passing--bin infraternalinvocation above.Logs:
/mnt/hdd/infraternal/jobs/review-stream-bugs/(*-stream-tests.log,native-exec-tests.log,binary-relay-tests-retry.log,debug-build.log). Driver and JSONL evidence:/mnt/hdd/infraternal/wt-lp/review-stream-bugs/target/review-stream-bugs/. Container label wasinfraternal.test=review-stream-bugs; Docker and Podman filtered cleanup checks returned zero IDs. Working tree is clean, HEAD unchanged, no commits or pushes.Primary-source checks