S1 tracer: agent core (embedded agent, shim management, terminal relay) #20
Closed
opened 2026-10-06 11:48:15 +00:00 by kayg
·
35 comments
No Branch/Tag specified
main
j470b-server-mesh-rejoin
mesh-compat-382
j444b-d381-key-wins-flood
j461b-macos-utun-reader-thread
j475b-remove-host-motion
fix-reconnect-new-link
j481-d380
dev
j479-mesh-soak
fix/j476-host-facts
j475-motion-pill-split
j470-mac-node-join
root-upgrade-d376
j474-review-merge
latency-targets-120hz
nightly-suite-shards
gotatun-pin-j469
macos-utun-reader-thread
fix-flood-echo-stall
wip/j416
motion-fast-ease-out
mesh-addhost-j259
takeover-close-terminals-d368
kbd-instant-followups
fix-trust-key-double-click
nightly-suite-partial-report
appwin-icons-j207
rusqlite-slice1
swipe-content-first-d374
mesh-mtu-381
addhost-instant-open
glass-contrast-d365-mock
glass-contrast-d365
signout-followups-442
fix-startup-toast-red
research-gotatun-matrix-j317
themes-sync-j304
fix-load-flakes
fix-nix-zig-patch
fix-compute01-browser-baseline
ratchet-size-baseline-419
predict-auto-d130
revoke-coverage-d367
nightly-suite-queue-timeout
nightly-browser-suite
fix-rustsec-2026-0459
fonts-ioskeley-obel
log-retention-d375
fix-kbd-no-anim-427
fix-dev-red-specs-2
takeover-proof-305-j205
wip/j410-server-pump
ui-consistency-d372
fix-osc52-clipboard
libghostty-vt-1.0
forget-retired-host-d369
j404-merge-426
fix-setup-lock-race
fix-touch-select-426
revoke-access-report-ui
fix-predictive-echo-specs
fix-selinux-agent-exec
wip/j395-checkpoint2
wip/j395
j391-review-scratch
ui-consistency-j195-deferred-rebased
fix-rustc-paths-421
decisions-wording-341
fix-install-suite-red
fix-dev-rtt-specs
fix-pending-removal-align
fix-privileged-skip
fix-install-suite-404
review-j368
mesh-dns-j254
revoke-access-report-d356b
revoke-privilege-round5
fix-removal-confirm
review-j360
gotatun-rotation-367
deps-refresh-340
review-j354
review-J351
review-j350
addhost-takeover-d359
revoke-report-items-347
revoke-end-sessions-d362
review-j348
size-mesh-ssh-346
size-trim-337
revoke-access-diff-d356
j259-pre-rebase-b06c5a6
j305-ended-at-start-spec
addhost-sudo-consent
fix-layout-real-specs-411
fix-shell-open-411
fix-console-hosts-373
fix-shell-exit
fuzz-host-name-353
fix-host-details-410
macos-filter-j197
research-gotatun-tricks
revoke-p3-j216l
stale-state-pill
revoke-followups-j216k
revoke-progress-j216e
backup/macos01-20261106/lume/j197
backup/macos01-20261106/lume/appwin-icons-j207-stash0
backup/macos01-20261106/claude/stash-0
backup/macos01-20261106/claude/stash-1
backup/macos01-20261106/claude/J80-macos-s1
backup/macos01-20261106/claude/J80
backup/macos01-20261106/claude/J80-slice1-20261102
backup/macos01-20261106/claude/J16b-appwin-on-dev
backup/macos01-20261106/claude/appwin-launch-idle-146
backup/macos01-20261106/claude/J80-s1-run
wip/j259
trust-285-p3c
runtime-lock-j216f
mesh-ssh-p3-j175
tun-proof-p3b
trust-285-p3
terminal-polish-j271
tun-proof-p3
ssh-install-flakes-j216i
fuzz-fixes-j264
trust-285-a
trust-285-b
restart-oversized-fixture-j198
backup/macos01-20261106/lume/appwin-icons-j207
restart-bound-j216h
up-preflight-p3-j198
backup/macos01-20261106/lume/j251
up-preflight-j198
wip/up-preflight-j198-j245-p2
wip/up-preflight-j198-j245-p1
agent-fsync-offloop-j196
wip/up-preflight-j198-j245
settings-offscreen-j231
wip/up-preflight-j198-j226-inspection
wip/j103f
themes-j206
wip/up-preflight-j198-j226
flakes-j216c
backup/macos01-20261106/claude/j197
flakes-j216b
backup/macos01-20261106/claude/j197-red
wip/up-preflight-j198-review
flakes-j216
backup/macos01-20261106/claude/appwin-icons-j207
engineering-j208b
appwin-mock-j209
toast-stack-j165
engineering-j208
decisions-d351
decisions-d350
decisions-d349
appearance-presets-mock-j129
decisions-d348
docs-audit-spec
doc-coverage-baseline-raise
decisions-d347
decisions-d346b
decisions-d346
j88h-profile
ui-consistency-j195-deferred
ui-consistency-j195
engineering-practices-j194
preferred-login-ui-j166
signin-pools-263-j174
macos-node-265-s1
decisions-d345
decisions-d344
proxy-cgroup-j164
verify-quick-fuzz-329
proxy-start-io-j168
timing-flakes-j163
up-263-s1
mcp-events-j147
verify-quick-fuzz-j158
proxy-p3-j159
agent-resource-j157-detect
wip/j103d-final
remote-cancel-hang-j155
proxy-traefik-281-j62d
j149-verify-quick
proxy-traefik-281-j62b
j150-drop-unnamed
wip/j79d-red-r4
wip/j103d-d332
rootless-toast-j142
wip/j103d-head
wip/j103d
j125c-mcp
macos-build-j136
backup/macos01-20261106/claude/j136
wip/j79d-red-cli
wip/j79d-red-server
j121-link-liveness
j118c-tracer-failures
wip/j79c-head
release-desktop-j131
j124c-signed-macos
wip/j79c-red-p3
wip/j79c-red-p23
wip/j79c-red-p22
merge-j128-check
perf-deferred-j128
sidebar-toggle-260
j127-shortcut-changed-dot
j125-mcp
j118-tracer-failures
j126-account-summary
settings-sections-mock-j92
j120-pinch-lock
sidebar-292
backup/macos01-20261106/claude/j120
taste-2026-11-03
row-actions-242-215
backup/macos01-20261106/claude/stash-0-untracked
backup/macos01-20261106/claude/J80-265-s1-run
backup/macos01-20261106/claude/J80-macos-s1-20261102
j112-canvas-commands
ui-no-overshoot-scroll
fix-recovery-restart
backup/macos01-20261106/claude/stash-1-untracked
taste-2026-11-02
size-baseline-ratchet
machine-cards-structure-j108
j109-boot-copy-mockups
fix-dev-red
wip/up-263-s1
j95b-settings-rows
proxy-traefik-281
j95-option-rows
docs-human-j91
docs-keys-palette
fix-flood-1013-270-j38d
mesh-ssh-s1
mesh-ssh-s0
addhost-d291-tunnel
test-stability-load
mesh-relay-220
repro-296
shim-thp-off-24
mesh-pins-258-v2
mesh-dns
signin-d311-merged
docs-pages
ui-size-283
fix-ui-batch-specs
wip/j12b-edge-update
gate-on-arm
fix-rename-sync-293
trust-285-j60b
fix-focus-ring-clip
integ-ui-1101
edge-update-v2
signin-d311
taste-2026-11-01
layout-per-tab-250
ui-approved-splitopt-224
ui-approved-splitopt-224-on-dev
icon-text-centre-282
ui-approved-189-212-216
research-reverse-proxy
transfer-budget-320
review-mesh-pins-258-r2
sidebar-pill-259
review-mesh-relay-220
wip/j12-edge-update-v2
nix-package
status-crash-loop-237
fix-dev-test-failures
fix-fixture-socket-path
wip/mesh-relay-220-pre-dev-advance-rebase
fix-canvas-side-clip
research-head-to-head
frosted-real-256
review-mesh-pins-258
wip/mesh-relay-220-pre-final-rebase
review-layout-per-tab-250-r2
broad-suite-triage
nix-devshell
review-status-crash-loop-237-r3
wip/nix-devshell-pre-p3
taste-frosted-real
wip/code-serve-demo-2026-10-30
wip/code-serve-glass-2026-10-30
wip/code-serve-clusters-2026-10-30
wip/code-merge-release94-dev-2026-10-30
wip/rproxy-delta-scratch-2026-10-30
wip/stale-note-257-staged-2026-10-30
wip/d241-size-staged-2026-10-30
review-broad-suite-207
wip/mesh-pins-258-pre-split
wip/mesh-relay-220-pre-rebase
mesh-measure-220
plan-211-update
review-edge-autoupdate-r3
perf24-reconnect
review-stale-note-r2
release-prod-live
d241-shell-marker
review-mesh-pins-server-258
review-revocation-r3
review-setup-token-r2
review-sync-journal-r3
wip/review-mesh-pins-258-prune-review-1995fee46
frosted-scroll-256
sync-journal-r3
revocation-per-account-wip
review-tokio-asyncfd-0386
review-sync-p3s
wip/stash-8by64-stash0-2026-10-30
mesh-drag-r2
setup-token-data-wip
release-9-4
auth-busy-1013
mesh-view-overlap
sync-p3s
review-auth-busy-wake
review-readd-r8-wip
mesh-drag-bounds
audit-auth-93
app-windows-stream-start
review-authority-cap-r3
app-windows-r7-measure
readd-minimal-r6
readd-minimal-r5
app-windows-r7-soak
readd-minimal-r4
app-windows-r7
app-windows-r6
readd-minimal-r3
readd-minimal-r2
readd-minimal
merge-release8-dev
review-busy-page-liveness-r2
review-busy-page-liveness
mesh-view-trim
review-authority-read-retry-r2
review-remove-uninstall-r2
release-retry-reattach
release-retry
review-remove-uninstall
review-authority-read-retry
app-windows-ship
mesh-autoswitch-p2
mesh-ssh
mesh-relay
mesh-macos-node
screen-and-control
session-labels
review-real-logins-r5
review-real-logins-r4
review-real-logins-r3
shim-memory-r2
review-real-logins-r2
shim-memory
shadow-addhost-instant
shadow-real-logins
shadow-same-session-flood
shadow-prod-perf-fix
hosts-from-server
app-windows-real
app-windows-transport
release-prod-6w
mesh-policy-verify
quiet-close-toast
dev-merge-r4
review-mesh-agent-transport
review-app-windows-transport
settings-roundtrip
review-release-pipeline-r4
mesh-agent-transport
ended-strip-fix
review-audit-ca-update-d244
settings-rows
phone-width
signout-clear
fix-restart-tests
dev-merge-rp
signer-hardening
review-audit-ca-update
terminal-fidelity
review-audit-ca-update-wip
mesh-integrate
review-mesh-agent-transport-515d3712
review-mesh-dataplane-feec1988
mesh-dataplane
signout
research-canvas-node
wip/review-renderer-cost
wip/review-terminal-fidelity
padlock-glyph
review-release-pipeline-2e95b689
phone-polish
sidebar-polish
inbox-generic
review-mesh-dataplane-378d4055
prod-batch
ui-inbox
ui-ca-unlock
ui-machine-sections
review-mesh-dataplane-a5f744fb
tls-session-prod
session-bounds-prod
fix-dead-session
session-bounds
tls-session
palette-polish
shortcuts-d222
research-appmeta
phone-gutter
ui-centre-button
wip/ui-centre-button
wip/tls-session
review-addhost-breakit
ui-platform
renderer-cost
remote-agents
mesh-api
ssh-bootstrap
prod-hotfix2
prod-fixes
review-remote-agents
review-mesh-dataplane
session-life
research-browser
review-mesh-api
research-capture
review-mesh-api-round3
review-mesh-api-round1
gate-stability
review-gate-stability
review-relay-direct
review-mesh-api-findings
proto-sessions
font-picker
proto-addhost
proto-windows
mesh-core
latency-gate
research-bsd
research-discovery
keytray-mods
mock-core
ui-tweaks
fix-motion-sel
fix-motion
integrate2
fix-robust
t-web
t-server
vt-ghostty
fix-snap-echo
t-web-auth
review-vt2
t-agent
lane-protocol
lane-shim
lane-auth
s1-base
s1-contracts
s1-workspace
No results found.
No labels
launch-qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set
Reference
kayg/infraternal#20
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Part of #9. Base
dev7174d6b. The agent half:infraternal agent --fd 3speaks the wire protocol over the socketpair, starts shims (contracts/shim.md), reattaches to them after a restart, and relays terminal streams exactly-once with credit flow control and catch-up by snapshot (contracts/wire.md §5–6).Starting the infraternal agent core lane on branch t-agent, base
7174d6b. Reading contracts and the merged shim before implementation; changes stay within the lane and contract proposals will be recorded here.Contract findings before deviations: (1) frozen shim v1 Input has no sequence or PTY-write Ack, and Hello has no input offset (contracts/shim.md §4; crates/infraternal-shim/src/session.rs input queue/write). A crash after socket write but before leg-B Ack cannot distinguish written input from unsent input; a restart-safe exactly-once promise requires shim-owned dedup/ack, negotiated additively via caps. Proposal: a capability for sequenced Input, PTY-write Ack, and input-offset query; retain v1 unchanged. (2) contracts/wire.md §4 and infraternal-protocol Frame have no Exited frame. Proposal: use existing Event with terminal.exited JSON and session_id/status/final_seq. (3) terminal.open JSON schemas are unspecified. Proposal: input uid, cols, rows; output session_id as UUID string. (4) scripts/verify.d/rust.sh still requires agent exit 2 with not implemented, conflicting with the new CLI; request gate ownership/coordination. (5) inherited fd adoption requires an audited syscall boundary; current shim exposes only detach/run. Proposal: a safe shim API to adopt the inherited Unix socket without adding unsafe to the infraternal agent. Working first on pure bounded relay state and tests; no contract changes made.
Follow-up to boundary/gate findings: safe nix send/recv with MSG_DONTWAIT and tokio AsyncFd registration can operate the inherited descriptor without taking ownership or adding unsafe; no shim boundary change is needed for fd adoption. The existing fast CLI check can remain green by preserving the no-fd (remote transport not implemented) stub and implementing agent --fd explicitly. The shim-owned input acknowledgment/dedup gap remains; work can continue within lane ownership for discovery, launch and transport.
The concrete shared-boundary proposal is now reviewable at crates/infraternal-agent/CONTRACT-PROPOSAL.md in t-agent: shim capability bit 0; additive SequencedInput 0x17, PTY-write Ack 0x18, InputStateRequest 0x19 and InputState 0x1A; frozen v1 stays unchanged. It includes the indistinguishable crash cases and why a file/socket-write acknowledgment cannot fix them. Five relay state tests pass, including restored input offsets, byte-range overflow, output overlap/partial credit, 250 ms one-shot starvation, and snapshot lengths that do not advance output offsets. Relay states store no payload buffers (tested size bounds: Output <=64 bytes, Input <=32 bytes). Full fast gate is still doing the initial static builds.
Discovery and launch are committed as
f6eff6cafter a green fast gate. The session receive loop is cancellation-safe (fragmented header and body test), startup handshakes are concurrent and capped at 128, and launch uses a one-shot inotify readiness wait. Security finding: the inherited leg-B descriptor must not reach a target user shell; the launch/probe wrapper closes fd 3 before exec. Test finding: the absolute worktree-based fixture socket path exceeded Linux SUN_LEN; short relative paths within target/ fix the fixture without changing runtime state defaults. Transport socketpair tests now pass for fragmented/coalesced frames, reusable transmit storage, oversized prefixes and truncated EOF. PTY-write Ack/exactly-once input remains blocked on the shared shim capability proposed in CONTRACT-PROPOSAL.md.Real-shim canary finding and fix: the first run received 9,744 bytes but timed out before OPEN_OK because the host login profile enters a multiplexer. Reused crates/infraternal/tests/profile/.zshrc through a child-only wrapper, as the existing shim tests do; no user environment/files changed. The real shim launch/discovery/replay/resize test now passes (1 passed in 0.06s). A FIFO barrier releases output only after the replacement connection sends ReadFrom, avoiding sleep-based replay races. This proves the session helpers, not a runnable --fd daemon or crash-safe input. Remaining CLI relay, agent-process restart/flood tests and requested RSS/p99/throughput measurements depend on resolving the shared shim input capability; no such measurements are claimed.
Blocked handoff for the remaining tracer bullet, head
04c64fd852, branch t-agent (base7174d6b). Six atomic commits, clean worktree. Every commit passed the prescribed fast gate.Implemented within lane ownership: constant-space relay offset/credit state; safe event-driven inherited socket IO; secure, bounded concurrent shim discovery; system/user scope or detached launch; fd-3 closure before child exec; cancellation-safe shim reads; eleven new tests, including real-shim discovery/replay/resize. Evidence: crates/infraternal-agent/src/{relay,sessions,transport}.rs, crates/infraternal-agent/tests/, crates/infraternal/tests/agent_sessions.rs. The real-shim test uses the existing canary profile and a FIFO barrier, not timed replay.
Remaining work is NOT completed: agent --fd 3 wiring and session registry, full daemon relay, agent-process restart/exactly-once input tests, 50M credit-starvation Snapshot/memory test, idle RSS, Input→shim p99 and flood throughput. No daemon performance numbers are claimed. Completing input correctness requires shared shim-owned sequenced input and PTY-write Ack/cursor query outside this lane ownership; awaiting authorization or the shared shim change. Concrete additive layout and crash evidence: crates/infraternal-agent/CONTRACT-PROPOSAL.md. No decisions or shared shim/protocol contracts were changed; nothing pushed/deployed; issue remains open.
Final command: nix shell nixpkgs#binutils nixpkgs#file nixpkgs#gnumake nixpkgs#perl -c scripts/verify fast
Gate output follows verbatim:
Orchestrator decision on CONTRACT-PROPOSAL.md: approved, all of it.
SEQUENCED_INPUT, frames 0x17–0x1A, exactly as proposed. contracts/shim.md §7 allows additive caps, and this closes a real hole: v1 cannot tell an agent restart that happened after PTY write from one that happened before it. That breaks DECISIONS 41 (no lost or doubled keystrokes).terminal.openschema and the uid and size limits: approved.terminal.exited) on the terminal stream: approved.The lane may change
crates/infraternal-shimfor this capability. Do it in separate atomic commits: contract text first, then shim with tests, then the agent use. Caps-zero shims must keep passing their existing tests unchanged.Continuing the approved tracer bullet on branch t-agent, base
04c64fd852. Read CLAUDE.md, DECISIONS.md, docs/ARCHITECTURE.md, the issue thread and CONTRACT-PROPOSAL.md. Implementing contract first, then negotiated shim input, then daemon and real-binary evidence; each logical commit runs the prescribed fast gate.Contract committed as
012b9a0; shim capability committed as the following logical step. Negotiated input checks overflow/gaps/full queue before mutation, trims accepted overlap, and keeps written/accepted cursors across peer replacement. No periodic shim poll remains (signalfd handles child exit). Existing caps-zero compatibility assertions pass; the performance fixture only initializes added state. Evidence: crates/infraternal-shim/tests/unit/sequenced_input.rs and target/issue-20-shim-gate.log. New frames seeded into shim_frame_decode: 4,581,894 executions in 301 seconds, cov=473, ft=1421, no crash (target/issue-20-shim-fuzz.log). Preparing daemon registry and real-binary Ack-delivery barrier tests.The inherited-fd daemon is now wired through the real CLI, with bounded per-session tasks and registry. Real debug binary evidence: open/type/output and terminal.exited status=7 passed; 389,120-byte SIGKILL mid-stream replay passed with exact contiguous output; 1,664-byte crash after shim PTY write but before delivery of its Ack passed with exact input/output. The latter uses a local shim proxy holding InputAck, not a timed sleep. Daemon and CLI each passed their prescribed fast gates (target/issue-20-daemon-gate.log and target/issue-20-cli-gate.log). Release static binary measures are running; debug timing is not claimed as performance evidence.
Release flood oracle finding: a shell-side completion marker does not prove the last kernel PTY bytes have reached shim screen state. First starvation run returned a valid Snapshot at 104850241 with the flood starting at 52429390 (7949 bytes below the expected 50 MiB end). The test incorrectly required the Snapshot itself to include those still-buffered bytes. Updating the oracle to require a near-head Snapshot followed by a contiguous raw suffix through the 50 MiB boundary; this checks no suffix is lost rather than assuming a PTY write and shim read are atomic. Evidence: target/issue-20-tracer-release.log; scripts/verify-agent.py fixture being prepared. Release idle RSS and input Ack timing already satisfy the decision-41 budgets; posting final numbers after the corrected gate.
Startup finding from a repeat release fixture: terminal.open transiently failed. The existing readiness watch observed only IN_CREATE, which Linux emits at bind before listen. If the connection is refused in that interval and the event is consumed, there is no later IN_CREATE and launch waits for its 2-second deadline. Fix: watch IN_ATTRIB too, because the shim chmods the socket to 0600 after listen succeeds. Deterministic regression binds a non-listening socket, consumes creation, confirms ConnectionRefused, then listens/chmods and requires a readiness event. Evidence: crates/infraternal-agent/src/sessions.rs::socket_readiness, tests/unit/sessions.rs::post_listen_chmod_wakes_a_connection_refused_at_bind; both this regression and the real-binary crash oracle pass in target/issue-20-readiness-gate.log. No periodic retry was added.
Starting finish-and-harden work on branch t-agent, base SHA
68ef7556fc. Reviewing inherited tests and fixtures, then reproducing bounded command queue saturation before fixing it. No push or deployment.Reliability bug reproduced deterministically against the base debug binary: scripts/verify-agent-queues.py holds InputState responses across 12 concurrent sessions and sends 65 Input frames per session. Before release of the handshake barriers the daemon exits with: infraternal agent failed: terminal command queue full or closed. Fix replaces try_send with bounded awaiting backpressure and contains terminal-task IO errors to their mux stream. The oracle also compares every echoed payload byte and probes daemon liveness after killing one shim.
Queue hardening committed as
d3ce366; capability/readiness regressions retained asd250690. Both prescribed fast gates pass in target/issue-20-{queues,session-tests}-gate.log. The expanded 24-crash fixture passed debug with seed 20 (4,710,396 payload bytes). Concurrent verification exposed a fixture-only launch error: POSIX /bin/sh rejects closing inherited descriptors above nine (leg-b: Syntax error: Bad fd number). The fixture now uses Bash for arbitrary descriptor numbers and closes stderr on each killed subprocess. Crash tests hold input before forwarding and real PTY-write Acks before delivery, plus random mid-output and immediate-after-send kills; large held-Ack payloads were bounded to avoid the oracle itself blocking unread output.Crash-oracle stall resolved with evidence: target/issue-20-crash-stress-20.log timed out at iteration 22 with 229,376 output bytes received and proxy blocked forwarding an Output frame, while the replacement daemon was sending a large replay. The single-threaded proxy created a duplex transport deadlock; two independently bounded frame pumps remove it without changing daemon/shim code. The corrected release fixture now passes three consecutive 24-iteration runs, seeds 20/21/22: 4,710,396 / 6,222,717 / 5,303,608 payload bytes compared exactly in both directions (72 SIGKILLs total). Logs: target/issue-20-crash-release-{20,21,22}.log. All four crash positions run six times per fixture, including event-controlled pre-forward and real PTY-write-before-Ack windows. Reusable encoder regression retained as
f2f1d56after its green fast gate.Finished finish-and-harden work on branch t-agent. Head SHA:
947d68f02d. Clean worktree. Nothing pushed or deployed; issue stays open.Atomic commits:
d3ce366fix(agent): backpressure full terminal queues and isolate shim failuresd250690test(agent): cover shim capability negotiation and listen readinessf2f1d56test(shim): verify reusable encoding preserves capacity and state947d68ftest(agent): exercise 24 real daemon crashes and release budgetsReliability evidence: the pre-fix daemon failed with "terminal command queue full or closed" under stalled shim handshakes. The bounded eight-slot queue now awaits capacity and pauses leg-B reads. Closed shim IO is confined to its mux stream. The regression sends 12 concurrent sessions x 65 Input frames, compares 2,640,000 payload bytes exactly once, kills one shim, and requires daemon Ping to succeed. Paths: crates/infraternal-agent/src/daemon.rs, crates/infraternal/tests/agent_queues.rs, scripts/verify-agent-queues.py.
Three consecutive release 24-crash fixtures, 72 SIGKILLs total and 16,236,721 payload bytes:
24-iteration crash fixture seed=20: 4710396 payload bytes, all four crash positions x 6: PASS
24-iteration crash fixture seed=21: 6222717 payload bytes, all four crash positions x 6: PASS
24-iteration crash fixture seed=22: 5303608 payload bytes, all four crash positions x 6: PASS
Paths: target/issue-20-crash-release-{20,21,22}.log. Every run exercises four crash positions six times, including barriers before shim forwarding and after actual PTY write before Ack delivery. Replays are compared through an exact output fence with contiguous sequence checks and written-input acknowledgments. Shared proxy fixtures use one bounded frame per direction and shut down companion pumps on disconnect. Concurrent verification also exposed and corrected Bash descriptor handling above fd 9 and a single-threaded fixture proxy duplex deadlock.
Release evidence from target/x86_64-unknown-linux-musl/release/infraternal on this shared development host (not the pinned perf VM); timing includes the Python peer, transport and PTY-write acknowledgment, so it is an upper bound for Input→PTY write. This measurement ran alongside the fast gate. Full output, including an additional 24-crash run, follows verbatim:
Idle daemon RSS: 3,644 KiB (<15 MB); zero CPU ticks during 500 ms idle. Input→PTY-write Ack p99: 276.187 us, n=1000. Credited flood: 52,429,019 bytes / 11.004 s = 4.544 MiB/s. Starved 50 MiB: Snapshot seq 104858502, repaint 27,579 bytes, final cursor 104858502, sampled daemon peak RSS 4,416 KiB (10 ms sampling). Shell-side completion file proves flood progress while starved; contiguous suffix after Snapshot accounts for any final kernel PTY bytes. No throughput budget is claimed beyond these measured numbers.
Every commit passed the prescribed fast command. Per-step logs: target/issue-20-queues-gate.log, target/issue-20-session-tests-gate.log, target/issue-20-shim-tests-gate.log, target/issue-20-crash-tests-gate.log.
Final command: nix shell nixpkgs#binutils nixpkgs#file nixpkgs#gnumake nixpkgs#perl -c scripts/verify fast
Final gate output verbatim:
Independent review started on t-agent, head
947d68f02d, diff base dev7174d6bf6f. Source worktree remains read-only; experiments will use a temporary worktree. Reviewing restart windows, SEQUENCED_INPUT, queue bounds, throughput, idle wakeups and history.Independent review finding R1 (high): daemon.rs:203 awaits a full per-terminal command queue inside the sole leg-B dispatcher. Reproduction on release head
947d68f: hold one shim InputState reply, enqueue 9 Credit frames on that stream, then Ping on stream 0. No Pong after 1 s despite a live daemon; Pong arrives only after the shim barrier is released. One stalled shim can starve control and sibling inputs. Evidence: review-agent/evidence/starvation.log and review_probes.py (report artifacts will be retained under ~/.local/state/codex-jobs/infraternal/review-agent). Use bounded per-stream overload handling that preserves multiplexed dispatch; increasing queue size only moves the failure.Independent review finding R2 (high): daemon.rs:321-329 retains a Snapshot that exceeds credit without arming starvation or detaching. Reproduction: drain initial output credit during a 12 MiB flood until it finishes detached; grant one byte of Credit to trigger catch-up; send another 12 MiB flood. Its completion marker is absent after 3 s; it completes after stream Close releases the shim. Pending Snapshot disables shim reads at line 369 and blocks PTY progress indefinitely if further credit never arrives, contrary to wire section 6 and decisions 75/81. Fix by detaching while credit cannot fund repaint, then requesting a fresh Snapshot once sufficient credit is available. Evidence: review-agent/evidence/snapshot.log and review_probes.py.
Independent review finding R3 (high): daemon.rs:244-247 assumes the first frame after InputStateRequest is InputState. A surviving shim can legally emit InputAck first while previously accepted input reaches the PTY. Real release reproduction: raw/no-echo shell delays reading for 3 s, submit 524288 bytes, abruptly stop infraternal agent after Ack=11827 with submitted=524339, restart, hold StateRequest while retained input drains. Reattachment receives a real InputAck and emits terminal shim disconnected instead of attaching. A separate valid-frame socket oracle also reproduces with InputAck(1), InputState(1,1). Read until State, handling interleaved negotiated Acks; add a retained accepted-but-unwritten restart test. Evidence: review-agent/evidence/real-state-ack.log, real_state_ack_probe.py, and state_ack_probe.py.
Independent review finding R4 (blocker, decision 41): three direct release 50 MiB credited floods measured 3.779, 4.190 and 3.453 MiB/s (50 MiB / 13.233, 11.932 and 14.481 s). Required 10 MB/s is 9.537 MiB/s. perf sampled both shim and infraternal agent: 91.17% CPU in shim, 8.83% in infraternal agent; hottest reset-cell instructions account for about 70% of total. This flood stresses Alacritty row reset/scrolling; the reported ~44 MB/s parser microbenchmark does not establish throughput on this payload. Transport-only optimization cannot meet the budget (even eliminating all sampled daemon cost gives at most 1.10x). Profiling with symbols and additional workloads is in progress; evidence will be retained in review-agent/evidence/perf*.data and flood*.log.
Independent review performance update: symbol-preserving release build (same head
947d68f, optimization unchanged) measured 5.422, 5.662, 5.471 MiB/s for three 50 MiB LF floods after the gate build. Symbolized perf: shim 90.07%, infraternal agent 9.93%; Alacritty Row::reset alone 65.82% of all sampled CPU. Workload probe with ample batched credit: LF 5.712 MiB/s, CRLF 20.423 MiB/s, printable without LF 41.816 MiB/s. This isolates the newline/column-dependent clearing cost and rules out a universal 4 MiB/s relay ceiling. Optimize row dirty spans or batch terminal scroll/reset with equivalent VT/history semantics; a 3x speedup of the 65.82% reset component predicts 1.78x overall, enough to investigate against the 9.537 MiB/s minimum. That is an Amdahl estimate, not a demonstrated fix. 50 MiB daemon+shim syscall trace: 12804 PTY reads (~4095 bytes/read), 25612 sends, 25612 polls, 46807 epoll waits, 63984 receives, 3661 munmaps. Reuse/move output allocations and batch reads/credit as secondary improvements. Evidence will include complete perf reports and syscall tables.Independent review finished. Verdict: merge after fixes, head
947d68f02d(base dev7174d6bf6f).Four required fixes, individually posted above: R1 shared dispatcher stalls behind one full command queue; R2 underfunded Snapshot leaves PTY attached and blocked; R3 legitimate interleaved InputAck breaks retained-input restart reattachment; R4 LF flood falls below decision 41, predominantly Alacritty row reset (65.82% of sampled CPU).
Independent evidence: 24 abrupt restarts, 6,261,482 bytes exactly once; restart during Snapshot and retained 91x31 resize pass; two live daemons deliver 18007 bytes exactly once through replacement; 50 attached idle sessions use 4064 KiB daemon RSS and 50-shim startup discovery takes 13.162 ms; settled 30-second trace has zero returned idle wait syscalls. Symbol-preserving release LF flood runs 5.422/5.662/5.471 MiB/s. Ample-credit workloads LF 5.712, CRLF 20.423, printable 41.816 MiB/s. No universal 4 MiB/s transport ceiling: row clearing is the principal cost. Proposed optimizations and estimated gains, scope limitations, queue inventory, history review and standalone probes are in the report.
Report appended to /home/kayg/.local/state/codex-jobs/infraternal/review-agent/findings.md; evidence retained alongside it under evidence/. Source worktree remains unchanged, no commits/push/deploy, no issue closed. Temporary experiment worktree is being removed. scripts/verify fast exited 0; all Rust and TypeScript checks pass. Full gate output verbatim follows:
Starting review fixes on branch t-agent, base
947d68f02d. Scope: R1–R3 deterministic regressions and corrections; R4 secondary payload-copy, buffer, read and credit batching measurements only. VT engine and row clearing remain for spike-vt. Atomic commits each run the required fast gate.Starting terminal-state spike on branch spike-vt, base
947d68f02d. Evaluating patched alacritty, Ghostty with a custom serializer, and visible-screen state with raw history. Results and reproducible gates will be committed in logical steps under spikes/vt.R3 reproduced with deterministic real-shim FIFO barriers: accepted=524614, written=14150, surviving unwritten=510464 bytes. A real InputAck reaches the replacement before InputStateRequest is released. Reviewed production binary emits Close (target/review-fixes/r3-before.log); corrected handshake verifies 524288 replay bytes exactly once (r3-after.log). Commit
e72589econsumes monotonic Acks with a two-second total State deadline; fast gate exit 0 (r3-gate.log). R1 regression holds State, fills nine command slots and demonstrates no Pong within one second (r1-before.log). R2 regression grants one byte after a detached 12 MiB flood; no Snapshot arrives within the fixture deadline (r2-before.log).R1 corrected in
6265615: eight-slot queues use nonblocking admission; overload retires and cancels only that terminal attachment. Holding a real shim State reply while sending nine commands now produces stream-local Close, sibling terminal output and control Pong before barrier release within one second (target/review-fixes/r1-after.log). Twelve concurrent saturated attachments are retired then reattached: 2640000 payload bytes verified exactly once; stopped shim remains isolated (r1-queues.log). Required fast gate exit 0 (r1-gate.log). Local performance measurements are contended: eight cores, load average 15.47; results will be reported as local comparisons, not performance-VM budget evidence.R2 contract is documented in its own commit
6c4b0c5, wire section 6: one bounded indivisible Snapshot with positive usable credit may borrow; future grants repay debt before raw output or another Snapshot; unfunded pending output detaches after 250 ms without extending the deadline for partial repayment. Test-first accounting regressions fail on the original relay (target/review-fixes/r2-unit-before.log). Corrected real-binary test completes four independent 12 MiB floods, delivers three catch-up Snapshots of 27541–27546 bytes with one byte usable credit, then verifies initial ring-wrap attachment at seq=50317621 (r2-after.log). No stream Close is needed for those four floods.R4 secondary measurements and codec correction:
d24a414moves full owned Output payloads; same-process 16 KiB forwarding benchmark 3899.5 -> 2078.5 ns/frame.6427ab9adds borrowed shim payload decoding and contiguous incremental buffering. Seven release samples (100000 frames each) have medians: owned decode 302.1 ns, borrowed 3.9; new owned feed 906.3 ns, borrowed 209.1 versus original bytewise feed 42051.7 ns (target/review-fixes/codec-{before,after-repeated}.log). Encoder reuse median 326.3 versus fresh 470.4 ns. Every fragmentation boundary/coalescing test passes. Differential owned/borrowed ASan fuzz completed 100000 runs without a finding (codec-fuzz-no-lto.log); fuzz-only LTO disabled to resolve sanitizer-coverage link errors. Both commits have passing required fast gates. Credit batching reduces measured CRLF grants 12895 -> 757 per 50 MiB; contended throughput varies, so no stable throughput claim is made for that change.R4 secondary pump fixes:
cc3a5a9consumes borrowed input into the surviving queue and reuses bounded encoder/stack poll storage. Poll construction benchmark 30.8 -> 1.5 ns/turn; encoder reuse median 470.4 -> 326.3 ns/frame.e8812fedrains immediately-ready PTY bytes into at most 16 KiB (and at most 16 reads), never waiting to fill a batch. Same 50 MiB CRLF workload on only daemon/shim: before drain 164910 traced calls / 25623 poll / 25625 sendto; after 54292 / 6445 / 6447, total -67.1%. PTY read calls increase 12811 -> 15686 while chunks approach 16 KiB. Traced throughput 5.678 -> 7.001 MiB/s is perturbed and contended. Evidence: target/review-fixes/{poll-bench,codec-after-repeated,syscalls-pump,syscalls-after,trace-pump,trace-after}.log. Both commits pass required fast gates; large retained-input restart still verifies 524288 bytes exactly once. VT engine and row clearing remain with spike-vt.Final release results: the new deterministic probes pass with 512512 surviving unwritten bytes and 524288 replay bytes exactly once; held-stream sibling/Pong completion 3.647 ms; four 12 MiB floods plus three 27580-byte one-credit Snapshots and initial ring-wrap attachment (target/review-fixes/r{1,2,3}-release.log). Existing 24-stop oracle seed 21 verifies 6222717 payload bytes, input-to-PTY Ack p99 197.446 us, daemon starved-flood peak RSS 4088 KiB (release-final.log). Three-run local LF median 4.426 MiB/s; explicit 10 MB/s check fails at 4.295 (lf-budget.log). The pinned-VM/VT requirement remains unresolved.
Additional measured memory finding: settled 30-second daemon/shim CPU ticks remain zero, daemon RSS 3852 KiB; shim RSS 5872 KiB. The existing standalone shim-idle <4096 KiB guard fails (shim-idle-final.log). The reviewed baseline binary also measures about 5736 KiB shim RSS in the same open-idle fixture (idle-baseline.log), so this is already present before the secondary fixes; the combined daemon/shim idle check claims zero ticks and daemon budget only. This is recorded without changing VT scope or loosening its guard.
Finished the requested review-fix scope on t-agent, head
6c64ede107, base947d68f02d, in 13 atomic commits. No push or deployment; this issue remains open.R1–R3 regressions are now part of the normal fast gate. Final release evidence: 512512 surviving unwritten input bytes, 524288 replay bytes exactly once, a real interleaved Ack before State; stalled-session sibling output and Pong in 3.647 ms before release; four 12 MiB floods, three 27580-byte Snapshots funded by one usable byte and initial ring-wrap seq=50333127. Twelve saturated streams replay 2640000 bytes exactly once. The full 24-stop fixture seed 21 verifies 6222717 payload bytes; 1000 input-to-PTY Ack samples p99=197.446 us; starved 50 MiB daemon peak RSS=4088 KiB.
R4 secondary evidence: full payload move 3899.5 -> 2078.5 ns/frame; contiguous/borrowed codec, cancellation-safe borrowed Reader, encoder reuse and stack poll storage have measured benefits; 16 KiB readiness draining reduces traced calls 164910 -> 54292 (-67.1%), poll/send calls by about 75%. Credit grants 12895 -> 757 at 64 KiB batching. Owned/borrowed ASan fuzz: 100000 runs, no finding. Production optimization unchanged; fuzz-only LTO override resolves sanitizer coverage linkage.
Remaining budget evidence is explicit: three-run local LF median 4.426 MiB/s; strict 10 MB/s check fails at 4.295 MiB/s. Standalone shim idle guard fails at 5872 KiB versus 4096; reviewed baseline is about 5736 KiB in the same fixture. Thirty settled idle seconds have unchanged daemon/shim CPU ticks, daemon RSS=3852 KiB. VT engine/row clearing remain with spike-vt and pinned-VM budget evidence remains required. The fast gate is green; these two targeted budget checks are not.
Detailed report: docs/agent-review-fixes.md. All local artifact paths and reproduction commands are recorded there under target/review-fixes/. Final required command: nix shell nixpkgs#binutils nixpkgs#file nixpkgs#gnumake nixpkgs#perl -c scripts/verify fast; exit 0. Full output verbatim follows:
Closed by the orchestrator during the 2026-10-28 handover (#254), on the owner's request to close finished issues. Evidence: Agent core merged (
6c64ede,48d7babin dev and prod); review R3 fixed (e72589e), integrated via #34. Reopen if anything is still missing.kayg referenced this issue2026-10-28 17:53:54 +00:00