Dead terminal session: agent respawn loop and a blank live-looking window

kayg commented

2026-10-10 17:35:51 +00:00

Owner

Production bug, 2026-10-10.

Symptom. After the owner closed several terminal windows, one window remained: title "kayg@monitor01 · shell ●" with a green dot, a blank body, and the pill said "1 session".

Server evidence (orchestrator, read from the journal). Since 17:27:03 UTC infraternal agent failed: unknown terminal session about twice a second (256 lines in 2 minutes), each from a new PID. Source: crates/infraternal-agent/src/daemon.rs:405 (registry.get(session_id) misses). Only 2 shims alive on the host.

Work. Branch fix-dead-session from origin/release-prod-2, checked against origin/session-bounds (#151).

Root cause of both halves (why the window points at a dead session; why the attach repeats with the dot green).
Typed, final "session ended" answer from the server, distinct from transient failures; no process per attempt.
Client stops on that answer and shows an honest ended state (existing exited strip in packages/ui / session UI), with one action that opens a new shell on the same host. Transient failures: bounded exponential backoff with jitter.
Tests: server test for the typed result; Playwright spec against the mock (attach count, ended state, reopen).

Refs: D178, D218, #151

Production bug, 2026-10-10. **Symptom.** After the owner closed several terminal windows, one window remained: title "kayg@monitor01 · shell ●" with a green dot, a blank body, and the pill said "1 session". **Server evidence (orchestrator, read from the journal).** Since 17:27:03 UTC `infraternal agent failed: unknown terminal session` about twice a second (256 lines in 2 minutes), each from a new PID. Source: `crates/infraternal-agent/src/daemon.rs:405` (`registry.get(session_id)` misses). Only 2 shims alive on the host. **Work.** Branch `fix-dead-session` from `origin/release-prod-2`, checked against `origin/session-bounds` (#151). - Root cause of both halves (why the window points at a dead session; why the attach repeats with the dot green). - Typed, final "session ended" answer from the server, distinct from transient failures; no process per attempt. - Client stops on that answer and shows an honest ended state (existing exited strip in packages/ui / session UI), with one action that opens a new shell on the same host. Transient failures: bounded exponential backoff with jitter. - Tests: server test for the typed result; Playwright spec against the mock (attach count, ended state, reopen). Refs: D178, D218, #151

kayg commented

2026-10-10 17:35:57 +00:00

Author

Owner

Started. Branch fix-dead-session in .worktrees/fix-dead-session, base origin/release-prod-2 at 9fba1eb2. Overlap check against origin/session-bounds (#151) in progress.

Started. Branch `fix-dead-session` in `.worktrees/fix-dead-session`, base `origin/release-prod-2` at 9fba1eb2. Overlap check against `origin/session-bounds` (#151) in progress.

kayg referenced this issue from a commit

2026-10-10 18:04:22 +00:00

protocol: Close 4404 on a terminal stream means the session is gone, finally (#158)

kayg referenced this issue from a commit

2026-10-10 18:04:22 +00:00

agent: a vanished session closes its stream with 4404, never the channel (#158)

kayg referenced this issue from a commit

2026-10-10 18:04:22 +00:00

server: back off when the infraternal agent link keeps dying young (#158)

kayg referenced this issue from a commit

2026-10-10 18:04:22 +00:00

server: tell the browser a session is gone with Close 4404, keep the socket (#158)

kayg referenced this issue from a commit

2026-10-10 18:04:22 +00:00

tracer: a terminal stops for good on Close 4404, and can start a new shell (#158)

kayg referenced this issue from a commit

2026-10-10 18:04:22 +00:00

tracer: reconnect with jittered backoff that resets only after a stable socket (#158)

kayg referenced this issue from a commit

2026-10-10 18:04:22 +00:00

canvas: a window whose session ended says so and offers a new shell (#158)

kayg referenced this issue from a commit

2026-10-10 18:04:22 +00:00

tracer: spec and capture for a dead session on the product canvas (#158)

kayg commented

2026-10-10 18:14:57 +00:00

Author

Owner

Finding: root causes (code on origin/release-prod-2 9fba1eb2).

The respawn loop is the server's, not the browser's.

crates/infraternal-agent/src/daemon.rs:405: an AttachTerm whose id is missing from the registry returns Err from run_channel, so the whole embedded infraternal agent exits (embedded_cli.rs:58 logs "infraternal agent failed: unknown terminal session").
crates/infraternal-server/src/embedded.rs:1489 connection_lost keeps every attachment. run_actor (embedded.rs:585-600) spawns a new agent and reattach_all (embedded.rs:1042) re-sends the same dead AttachTerm. The backoff is reset to 200 ms on every successful connect and only doubled on loss, so the cycle is about 450 ms: about 2 per second with a new PID each time, matching 256 lines in 2 minutes. Each restart also cuts every other terminal stream on the host.
The registry (daemon.rs:310-314, :370) is filled only by startup discovery and this process's own opens, and nothing ever removes entries. It is a cache, not the truth.

Why the window stays green and blank.

apps/tracer/src/terminal.ts:825-845 tryAttach sets attached = true, clears the status and fires onAttached (the live dot) as soon as it sends AttachTerm. The server answers Ack before the agent has checked anything (websocket.rs:1214). After that the agent dies, so nothing else arrives: the body stays blank and the dot stays green.
A stream Close on a shared socket only sets a status line (terminal.ts:789-793), and no close code means "final".
The tight client loop exists on the other path. An attach to a session whose exit was recorded, or that the account never had, fails owned_writer (websocket.rs:1145), which closes the whole WebSocket. connection.ts:379 resets the retry counter on every Welcome, so the page reconnects and attaches again every ~50 ms.

Why a window points at a dead id (closing windows did not end it).

Closing a window never ends a session on this line. lifecycle.ts:146-155 only detaches (D178). No branch implements D218 (git grep D218 finds nothing in apps/ or crates/ on any remote branch).
The browser's session list is its own memory: live-inventory.ts (localStorage) plus the canvas document. Nothing reconciles it with the host. A shell's exit is recorded only when a browser is attached at that moment (terminal_exit.rs, b00f0a06). A window restored from the document, or reopened from the sidebar, palette or manager, can therefore name a session whose shim is gone, and nothing ever tells the page. On this line a shim disappears only through ExitAck (uninstall), its own I/O error, or being killed. The journal excerpt does not show which happened. The loop starts when a window attaches to that stale id.
origin/session-bounds (#151) makes this more frequent: it retires dead shims and refreshes the registry, but keeps registry.get(...)? failing the channel. An attach to any retired session would hit the same loop. This fix should land with or before it.

Finding: root causes (code on `origin/release-prod-2` 9fba1eb2). **The respawn loop is the server's, not the browser's.** - `crates/infraternal-agent/src/daemon.rs:405`: an `AttachTerm` whose id is missing from the registry returns `Err` from `run_channel`, so the whole embedded infraternal agent exits (`embedded_cli.rs:58` logs "infraternal agent failed: unknown terminal session"). - `crates/infraternal-server/src/embedded.rs:1489` `connection_lost` keeps every attachment. `run_actor` (`embedded.rs:585-600`) spawns a new agent and `reattach_all` (`embedded.rs:1042`) re-sends the same dead `AttachTerm`. The backoff is reset to 200 ms on every successful connect and only doubled on loss, so the cycle is about 450 ms: about 2 per second with a new PID each time, matching 256 lines in 2 minutes. Each restart also cuts every other terminal stream on the host. - The registry (`daemon.rs:310-314`, `:370`) is filled only by startup discovery and this process's own opens, and nothing ever removes entries. It is a cache, not the truth. **Why the window stays green and blank.** - `apps/tracer/src/terminal.ts:825-845` `tryAttach` sets `attached = true`, clears the status and fires `onAttached` (the live dot) as soon as it *sends* `AttachTerm`. The server answers `Ack` before the agent has checked anything (`websocket.rs:1214`). After that the agent dies, so nothing else arrives: the body stays blank and the dot stays green. - A stream `Close` on a shared socket only sets a status line (`terminal.ts:789-793`), and no close code means "final". - The tight *client* loop exists on the other path. An attach to a session whose exit was recorded, or that the account never had, fails `owned_writer` (`websocket.rs:1145`), which closes the whole WebSocket. `connection.ts:379` resets the retry counter on every Welcome, so the page reconnects and attaches again every ~50 ms. **Why a window points at a dead id (closing windows did not end it).** - Closing a window never ends a session on this line. `lifecycle.ts:146-155` only detaches (D178). No branch implements D218 (`git grep D218` finds nothing in apps/ or crates/ on any remote branch). - The browser's session list is its own memory: `live-inventory.ts` (localStorage) plus the canvas document. Nothing reconciles it with the host. A shell's exit is recorded only when a browser is attached at that moment (`terminal_exit.rs`, b00f0a06). A window restored from the document, or reopened from the sidebar, palette or manager, can therefore name a session whose shim is gone, and nothing ever tells the page. On this line a shim disappears only through ExitAck (uninstall), its own I/O error, or being killed. The journal excerpt does not show which happened. The loop starts when a window attaches to that stale id. - `origin/session-bounds` (#151) makes this more frequent: it retires dead shims and refreshes the registry, but keeps `registry.get(...)?` failing the channel. An attach to any retired session would hit the same loop. This fix should land with or before it.

kayg referenced this issue from a commit

2026-10-10 18:48:02 +00:00

tracer: reconnect jitter only lengthens a step, never under 50 ms (#158)

kayg referenced this issue from a commit

2026-10-10 18:48:02 +00:00

tracer: the dead-session spec retries a hung-up test-double request (#158)

kayg referenced this issue

2026-10-10 19:24:53 +00:00

Terminal opens refused at 128 sessions; dead sessions linger; background sessions invisible #151

kayg referenced this issue from a commit

2026-10-10 19:24:59 +00:00

protocol: reuse final session-ended Close from #158

kayg referenced this issue from a commit

2026-10-10 19:37:46 +00:00

tests: satisfy clippy in the dead-session regressions (#158)

kayg referenced this issue

2026-10-10 19:52:24 +00:00

Terminal opens refused at 128 sessions; dead sessions linger; background sessions invisible #151

kayg commented

2026-10-10 19:52:25 +00:00

Author

Owner

Coordination from session-bounds round 2 (#151): reused your unchanged protocol Close 4404/session ended constants from 0cd816bd as a93942d6 on session-bounds. Retained exited shims will retire at 600 s and preserve a private fixed-size Exited runtime record. On reconnect after that bound: ordered existing terminal.exited Event with status/final_seq, then your final Close 4404. Missing sessions: same final Close, no duplicate ended type. Implementation/tests in progress; no UI changes in this lane.

Coordination from session-bounds round 2 (#151): reused your unchanged protocol Close 4404/session ended constants from 0cd816bd as a93942d6 on session-bounds. Retained exited shims will retire at 600 s and preserve a private fixed-size Exited runtime record. On reconnect after that bound: ordered existing terminal.exited Event with status/final_seq, then your final Close 4404. Missing sessions: same final Close, no duplicate ended type. Implementation/tests in progress; no UI changes in this lane.

kayg referenced this issue

2026-10-10 19:57:30 +00:00

Terminal opens refused at 128 sessions; dead sessions linger; background sessions invisible #151

kayg commented

2026-10-10 19:57:38 +00:00

Author

Owner

Finished on branch fix-dead-session, head 523daf6c (pushed), base origin/release-prod-2 9fba1eb2. Not deployed.

scripts/verify fast (verbatim tail):

PT_INTERP: absent; size: 19904416 bytes (baseline 16142672 + allowance 262144 = max 16404816)
binary exceeds the 16404816-byte size budget: 19904416 bytes

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+
EXIT 1

Static binary: 19,904,416 B on this branch against 19,896,960 B on release-prod-2, built from the same target dir: +7,456 B. The size row fails as it did before this branch; the limit is unchanged.
Tracer: bun run check passes. The canvas, recovery and exit unit tests pass (148). The protocol tests pass (36; the Rust/TS cross test takes 30-32 s under load, against a 30 s timeout).
dead-session.spec.ts passes 3 of 3 runs (TRACER_TEST_PORT=5558). sessions-canvas and terminal specs: every failure on this branch also fails on release-prod-2 in the same run conditions (9 failures on base, 6 on this branch). snapshot-resume flakes 3 of 8 times on base too.
Open: the transfer budget's deferred row. It was already within 35-70 B brotli of its 45,056 B limit on base (44,987-45,022 B; the hashed chunk names make it vary from build to build). This branch adds about 60 B net: 45,038-45,110 B, so it fails about two runs in three. I did not raise the limit. The orchestrator and the owner need to choose: raise the deferred margin, or move existing deferred code on demand.
Merge with origin/session-bounds (#151): release-prod-2 and session-bounds already conflict in 8 files. This branch adds only trivial conflicts (two test-module registrations in daemon.rs, adjacent doc comments in sessions.rs). The changed AttachTerm line merges cleanly and compiles against its lifecycle::Registry::get. session-bounds retires dead shims, so without this fix an attach to any retired session would hit the same respawn loop. Land this with it or first.

Finished on branch `fix-dead-session`, head `523daf6c` (pushed), base `origin/release-prod-2` 9fba1eb2. Not deployed. `scripts/verify fast` (verbatim tail): ``` PT_INTERP: absent; size: 19904416 bytes (baseline 16142672 + allowance 262144 = max 16404816) binary exceeds the 16404816-byte size budget: 19904416 bytes +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ EXIT 1 ``` - Static binary: 19,904,416 B on this branch against 19,896,960 B on release-prod-2, built from the same target dir: **+7,456 B**. The size row fails as it did before this branch; the limit is unchanged. - Tracer: `bun run check` passes. The canvas, recovery and exit unit tests pass (148). The protocol tests pass (36; the Rust/TS cross test takes 30-32 s under load, against a 30 s timeout). - `dead-session.spec.ts` passes 3 of 3 runs (TRACER_TEST_PORT=5558). sessions-canvas and terminal specs: every failure on this branch also fails on release-prod-2 in the same run conditions (9 failures on base, 6 on this branch). `snapshot-resume` flakes 3 of 8 times on base too. - **Open: the transfer budget's deferred row.** It was already within 35-70 B brotli of its 45,056 B limit on base (44,987-45,022 B; the hashed chunk names make it vary from build to build). This branch adds about 60 B net: 45,038-45,110 B, so it fails about two runs in three. I did not raise the limit. The orchestrator and the owner need to choose: raise the deferred margin, or move existing deferred code on demand. - Merge with `origin/session-bounds` (#151): release-prod-2 and session-bounds already conflict in 8 files. This branch adds only trivial conflicts (two test-module registrations in daemon.rs, adjacent doc comments in sessions.rs). The changed AttachTerm line merges cleanly and compiles against its `lifecycle::Registry::get`. session-bounds retires dead shims, so without this fix an attach to any retired session would hit the same respawn loop. Land this with it or first.

kayg commented

2026-10-10 20:01:34 +00:00

Author

Owner

Independent review started on review-fix-dead-session at target 523daf6cef; merge base with dev 2eaaaa36a2. Read #158 and all four comments and related #151 history (41 comments). Author result.md is absent; using completion claims on #158. Full merge-base diff includes 617 files from earlier production branches; focusing new behavioral probes on eleven #158 commits since 9fba1eb2 while checking touched contracts. Evidence commits local; no push/deploy per reviewer output brief.

Independent review started on review-fix-dead-session at target 523daf6cefc5b2f9b11ac25e94c8b5a68ec973ec; merge base with dev 2eaaaa36a2dcdda12582150d29c14d660e6bcb2f. Read #158 and all four comments and related #151 history (41 comments). Author result.md is absent; using completion claims on #158. Full merge-base diff includes 617 files from earlier production branches; focusing new behavioral probes on eleven #158 commits since 9fba1eb2 while checking touched contracts. Evidence commits local; no push/deploy per reviewer output brief.

kayg commented

2026-10-10 20:01:49 +00:00

Author

Owner

Follow-up (coordinator): the deferred transfer row now passes with margin, and no D188 budget was raised. a552bb2e moves the logs-window body to log-body.on-demand.ts. No product path opens a logs window, so the product pays nothing. Deferred brotli over 5 consecutive builds on the commit: 44,706 / 44,725 / 44,762 / 44,713 / 44,727 B against 45,056 B (294-350 B under). clusters-logs (2), dead-session and detached-output specs pass; tsc and 114 canvas unit tests pass. No Rust or protocol change.

Follow-up (coordinator): the deferred transfer row now passes with margin, and no D188 budget was raised. `a552bb2e` moves the logs-window body to `log-body.on-demand.ts`. No product path opens a logs window, so the product pays nothing. Deferred brotli over 5 consecutive builds on the commit: 44,706 / 44,725 / 44,762 / 44,713 / 44,727 B against 45,056 B (294-350 B under). clusters-logs (2), dead-session and detached-output specs pass; tsc and 114 canvas unit tests pass. No Rust or protocol change.

kayg referenced this issue from a commit

2026-10-10 20:01:49 +00:00

canvas: the logs window body loads on demand (D188, #158)

kayg commented

2026-10-10 20:08:49 +00:00

Author

Owner

P2 DS-01 confirmed: live-inventory.ts attach changes a known gone record from exited to open while gone=true. Selecting it again can paint live/remove the ended strip until the redundant attach is refused. Normal Bun unit test selecting a known gone session keeps its final exited state: 0 pass / 1 fail, expected exited / received open, 5.29 ms. Evidence committed on review-fix-dead-session (local), implementation unchanged. Native QUIC final-Close probe is prepared; exact-head Rust checks still compiling. Existing production dead-session.spec passes 1/1 in 48.4 s at port5683. Three scripts/verify fast attempts stopped at RTT self-test under load35.53/26.56/23.57; no gate failure waived.

P2 DS-01 confirmed: live-inventory.ts attach changes a known gone record from exited to open while gone=true. Selecting it again can paint live/remove the ended strip until the redundant attach is refused. Normal Bun unit test `selecting a known gone session keeps its final exited state`: 0 pass / 1 fail, expected exited / received open, 5.29 ms. Evidence committed on review-fix-dead-session (local), implementation unchanged. Native QUIC final-Close probe is prepared; exact-head Rust checks still compiling. Existing production dead-session.spec passes 1/1 in 48.4 s at port5683. Three `scripts/verify fast` attempts stopped at RTT self-test under load35.53/26.56/23.57; no gate failure waived.

kayg commented

2026-10-10 20:26:42 +00:00

Author

Owner

Completed independent transfer comparison, local test commit on review-fix-dead-session. P2 DS-02: production deferred Brotli45073/45056 B (+17), base9fba1eb2 passes45013/45056 B: observed +60 B. Confirms the author's intermittent near-boundary overflow; preview bundle results excluded. Normal Bun test dead-session production canvas fits the deferred Brotli budget: 0 pass/1 fail, 13.85 ms. Critical production transfer passes289073/307200 gzip and240145/256000 Brotli. Bun1.4.2, local load26.36/29.65/29.29. Original spec passes1/1; original canvas unit run110 PASS plus known DS-01 and mesh layout latency47.187ms/25ms under host load. Weighted exact-head fast gate now passed RTT selftest312.18ms p50/317.08ms p99 and is waiting on compilation. Native QUIC proof compiling; no implementation changes/push.

Completed independent transfer comparison, local test commit on review-fix-dead-session. P2 DS-02: production deferred Brotli45073/45056 B (+17), base9fba1eb2 passes45013/45056 B: observed +60 B. Confirms the author's intermittent near-boundary overflow; preview bundle results excluded. Normal Bun test `dead-session production canvas fits the deferred Brotli budget`: 0 pass/1 fail, 13.85 ms. Critical production transfer passes289073/307200 gzip and240145/256000 Brotli. Bun1.4.2, local load26.36/29.65/29.29. Original spec passes1/1; original canvas unit run110 PASS plus known DS-01 and mesh layout latency47.187ms/25ms under host load. Weighted exact-head fast gate now passed RTT selftest312.18ms p50/317.08ms p99 and is waiting on compilation. Native QUIC proof compiling; no implementation changes/push.

kayg commented

2026-10-10 20:49:58 +00:00

Author

Owner

P1 DS-03 confirmed. HOLD (P0=0/P1=1/P2=2/P3=0), merge after fixes.
Location daemon.rs run_bound_terminal:536-540 and relink:602-604. On an absent shim, relink queues Message::Ended then terminal returns Ok; the enclosing select returns, cancels writer and retires the bound native QUIC stream before Close4404 is sent. Remote windows receive no final ended answer.
Real quinn/capability-bound test vanished_native_session_delivers_final_close_before_stream_retirement: relay completes Ok but peer receives UnexpectedEof before its final answer; 0 pass/1 fail in0.08s. Initial fixture-parent setup failure excluded and parent creation corrected. Failing test committed locally on review-fix-dead-session; no implementation repair/push. Evidence ~/.cache/codex-scratch/fix-dead-session-review/native-gone-runtime.log. Exact-head fast gate has passed RTT/fmt/VT fmt and is compiling clippy; final gate report follows.

P1 DS-03 confirmed. HOLD (P0=0/P1=1/P2=2/P3=0), merge after fixes. Location daemon.rs run_bound_terminal:536-540 and relink:602-604. On an absent shim, relink queues Message::Ended then terminal returns Ok; the enclosing select returns, cancels writer and retires the bound native QUIC stream before Close4404 is sent. Remote windows receive no final ended answer. Real quinn/capability-bound test `vanished_native_session_delivers_final_close_before_stream_retirement`: relay completes Ok but peer receives UnexpectedEof before its final answer; 0 pass/1 fail in0.08s. Initial fixture-parent setup failure excluded and parent creation corrected. Failing test committed locally on review-fix-dead-session; no implementation repair/push. Evidence ~/.cache/codex-scratch/fix-dead-session-review/native-gone-runtime.log. Exact-head fast gate has passed RTT/fmt/VT fmt and is compiling clippy; final gate report follows.

kayg commented

2026-10-10 20:51:42 +00:00

Author

Owner

Completed independent scale diagnostic on review-fix-dead-session: 200 sequential vanished-session ids on one real inherited Unix channel, all Close4404 replies plus all 200 late-Input/Close/Ping fences PASS without channel replacement. CPUWeight10000, load22.88/29.11/34.59, p50 193us/p99 4519us/max6486us, 0.12s. This is the changed refusal path, not 200 simultaneous live PTYs or a typing-latency claim. Rust normal ignored benchmark test committed separately from the P1 failing native test967e7fec. Protocol36/36 and34462 assertions PASS in2.74s on weighted warm rerun; earlier30s timeout retained in evidence. HOLD remains P1=1/P2=2. Exact-head clippy still compiling.

kayg commented

2026-10-10 20:55:23 +00:00

Author

Owner

Completed durable review evidence step ca2bcce0 on local review-fix-dead-session. Report docs/reviews/dead-session-158.md plus five committed raw proof excerpts. Tests: DS-01 d20f8537 (P2 final inventory revival), DS-02 feae879e (P2 deferred transfer overflow), DS-03 967e7fec (P1 native QUIC loses final Close); passing sequential200-id diagnostic8ae46775. Verdict HOLD/merge after fixes, P0=0/P1=1/P2=2/P3=0. No implementation changes or pushes.
Exact-head archived fast gate and latency gate remain running locally under CPUWeight10000; fmt/VT fmt/default RTT have passed, Rust clippy/release dependencies still compiling. Final gates will be a separate evidence commit/comment with verbatim output, not inferred from the author report.

Completed durable review evidence step ca2bcce0 on local review-fix-dead-session. Report docs/reviews/dead-session-158.md plus five committed raw proof excerpts. Tests: DS-01 d20f8537 (P2 final inventory revival), DS-02 feae879e (P2 deferred transfer overflow), DS-03 967e7fec (P1 native QUIC loses final Close); passing sequential200-id diagnostic8ae46775. Verdict HOLD/merge after fixes, P0=0/P1=1/P2=2/P3=0. No implementation changes or pushes. Exact-head archived fast gate and latency gate remain running locally under CPUWeight10000; fmt/VT fmt/default RTT have passed, Rust clippy/release dependencies still compiling. Final gates will be a separate evidence commit/comment with verbatim output, not inferred from the author report.

kayg commented

2026-10-10 21:36:23 +00:00

Author

Owner

DS-04 P2, b892d4d5: production browser injects a transient Close1013 while the same session and shared socket remain live. No reattach in10s: attach_count1→1, socket opens1→1/closes0→0; 1failed,31.3s. Test test/transient-session-review.spec.ts. terminal.ts:798–800 only force-closes owned sockets; canvas shares its connection. This is an existing-path gap against the brief’s bounded transient retry claim; paired production-base probe is running. HOLD remains (P0=0,P1=1,P2=3,P3=0). Implementation unchanged, no push.

DS-04 P2, b892d4d5: production browser injects a transient Close1013 while the same session and shared socket remain live. No reattach in10s: attach_count1→1, socket opens1→1/closes0→0; 1failed,31.3s. Test test/transient-session-review.spec.ts. terminal.ts:798–800 only force-closes owned sockets; canvas shares its connection. This is an existing-path gap against the brief’s bounded transient retry claim; paired production-base probe is running. HOLD remains (P0=0,P1=1,P2=3,P3=0). Implementation unchanged, no push.

kayg commented

2026-10-10 21:39:30 +00:00

Author

Owner

Exact archived job head523daf6c, no review tests: scripts/verify fast completed. 574 Rust tests passed,0 failed; RTT312.18/317.08ms. Static stripped PIE19904416B fails historical16404816B limit; independently built production base9fba1eb2=19896992B, delta+7424B (D205 report, no budget edits). Size failure prevents the TS dispatcher; direct TS compiler and exact-head111 canvas units passed separately. Native latency fails same-session flood on both head (PTY p99383.47ms) and production base (498.69ms), budget334ms; no causal regression claimed. Full latency browser phase remains running.

Exact archived job head523daf6c, no review tests: scripts/verify fast completed. 574 Rust tests passed,0 failed; RTT312.18/317.08ms. Static stripped PIE19904416B fails historical16404816B limit; independently built production base9fba1eb2=19896992B, delta+7424B (D205 report, no budget edits). Size failure prevents the TS dispatcher; direct TS compiler and exact-head111 canvas units passed separately. Native latency fails same-session flood on both head (PTY p99383.47ms) and production base (498.69ms), budget334ms; no causal regression claimed. Full latency browser phase remains running. Verbatim fast Rust summary: +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+

kayg commented

2026-10-10 21:43:37 +00:00

Author

Owner

Evidence step committed65735b43 (local review-fix-dead-session). Exact-head fast summary/paired size and target/base transient retry proofs now durable in docs/reviews/dead-session-158.md and evidence/dead-session-158/. Review remote all-targets clippy, Rustfmt, TypeScript and diff checks PASS. DS03 final source rerun: 0pass/1fail, UnexpectedEof before Close4404,0.03s. DS04 base confirmation uses test-only head mock instruments; first run lacked counters and is excluded. No implementation changes or push. Full exact-head browser phase at131/506; native latency already red on head and base.

kayg commented

2026-10-10 21:57:37 +00:00

Author

Owner

Finished independent review: HOLD — merge after fixes (P0=0, P1=1, P2=3, P3=0).
Target fix-dead-session523daf6c; local review-fix-dead-session head355a53e81938e886db0fc023e803f43a8af40faf. Implementation unchanged; no push/deploy/issue closure.

P1 DS-03: real bound native QUIC retires before final Close4404; UnexpectedEof,0pass/1fail,0.03s. Test native_gone_review_tests.rs,967e7fec.
P2 DS-01: selecting known-gone inventory changes exited→open (d20f8537).
P2 DS-02: default production deferred Brotli45073 initially/45083 fresh vs45056B budget; base45013B (feae879e).
P2 DS-04: transient Close1013 never reattaches a live shared stream for10s; attach1→1/socket opens1→1,closes0→0. Reproduces on production base with instrumented mock (b892d4d5).

Exact-head fast:574 Rust testsPASS/0FAIL; static stripped19904416B vs production base19896992B,delta+7424B (D205). Size row stops TS dispatcher; independent111canvas units/36protocol tests and TS compilerPASS. Review Rustfmt,remote all-targets clippy,diff checksPASS.
Required native latency100samples/scenario: same-session PTY echo p99head383.47ms/base498.69ms vs334ms; growth52.63/172.37ms vs16.67ms. Both fail; no causal regression claimed or failure waived. Auth/recovery browser6scenariosPASS. Full506browser phase stopped after unrelated grid timeouts:130pass/14fail/1interrupted/12skipped/349notrun,29.7m; this is partial, untriaged coverage. Independent changed dead-session flowPASS48.4s.

Report: docs/reviews/dead-session-158.md; proofs docs/reviews/evidence/dead-session-158/; findings appended in the assigned job findings.md. Owned14GBtarget and temporary head/base archives removed; raw logs and paired stripped artifacts retained in job scratch.

scripts/verify fast on unmodified job head (verbatim tail):

PT_INTERP: absent; size: 19904416 bytes (baseline 16142672 + allowance 262144 = max 16404816)
binary exceeds the 16404816-byte size budget: 19904416 bytes

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Finished independent review: **HOLD — merge after fixes (P0=0, P1=1, P2=3, P3=0).** Target fix-dead-session523daf6c; local review-fix-dead-session head355a53e81938e886db0fc023e803f43a8af40faf. Implementation unchanged; no push/deploy/issue closure. P1 DS-03: real bound native QUIC retires before final Close4404; UnexpectedEof,0pass/1fail,0.03s. Test native_gone_review_tests.rs,967e7fec. P2 DS-01: selecting known-gone inventory changes exited→open (d20f8537). P2 DS-02: default production deferred Brotli45073 initially/45083 fresh vs45056B budget; base45013B (feae879e). P2 DS-04: transient Close1013 never reattaches a live shared stream for10s; attach1→1/socket opens1→1,closes0→0. Reproduces on production base with instrumented mock (b892d4d5). Exact-head fast:574 Rust testsPASS/0FAIL; static stripped19904416B vs production base19896992B,delta+7424B (D205). Size row stops TS dispatcher; independent111canvas units/36protocol tests and TS compilerPASS. Review Rustfmt,remote all-targets clippy,diff checksPASS. Required native latency100samples/scenario: same-session PTY echo p99head383.47ms/base498.69ms vs334ms; growth52.63/172.37ms vs16.67ms. Both fail; no causal regression claimed or failure waived. Auth/recovery browser6scenariosPASS. Full506browser phase stopped after unrelated grid timeouts:130pass/14fail/1interrupted/12skipped/349notrun,29.7m; this is partial, untriaged coverage. Independent changed dead-session flowPASS48.4s. Report: docs/reviews/dead-session-158.md; proofs docs/reviews/evidence/dead-session-158/; findings appended in the assigned job findings.md. Owned14GBtarget and temporary head/base archives removed; raw logs and paired stripped artifacts retained in job scratch. scripts/verify fast on unmodified job head (verbatim tail): ``` PT_INTERP: absent; size: 19904416 bytes (baseline 16142672 + allowance 262144 = max 16404816) binary exceeds the 16404816-byte size budget: 19904416 bytes +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ```

kayg commented

2026-10-10 22:02:57 +00:00

Author

Owner

Round 2 started: branch fix-dead-session, base/head a552bb2ee3. Read #158 and related #151 history; review report at local 355a53e8 (review branch is not published on origin). Bringing in the five exact review test commits, then repairing DS-03, DS-01, DS-04 and checking five fresh D188 production builds. Budgets unchanged; no deployment.

Round 2 started: branch fix-dead-session, base/head a552bb2ee3b99538d52805954a8f46be9749d3f1. Read #158 and related #151 history; review report at local 355a53e8 (review branch is not published on origin). Bringing in the five exact review test commits, then repairing DS-03, DS-01, DS-04 and checking five fresh D188 production builds. Budgets unchanged; no deployment.

kayg referenced this issue from a commit

2026-10-10 22:03:01 +00:00

test: retain final ended inventory state on selection (#158)

kayg referenced this issue from a commit

2026-10-10 22:03:01 +00:00

test: require the final ended answer on native QUIC streams (#158)

kayg referenced this issue from a commit

2026-10-10 22:03:03 +00:00

test: expose the deferred production transfer overflow (#158)

kayg referenced this issue from a commit

2026-10-10 22:03:04 +00:00

test: measure 200 final replies without replacing the channel (#158)

kayg referenced this issue from a commit

2026-10-10 22:03:06 +00:00

test: retain a transient shared terminal close for review (#158)

kayg commented

2026-10-10 22:10:44 +00:00

Author

Owner

Review tests imported unchanged and pushed individually: 445d4abb (DS-01), 48b07c5d (DS-03), 796b4f11 (DS-02), add06701 (200 replies), 8cac789a (DS-04). Intentional failing-test commits per the approved brief. DS-01 reproduced: exited→open, 0 pass/1 fail, 81.53 ms. DS-04 reproduced on production build/port5791: attach1→1, socket opens1→1/closes0→0 over the 10-second retry window, 1 fail. Native build initially stopped because make is absent; rerunning in the existing documented Nix make/perl shell. Logs: ~/.cache/codex-scratch/fix-dead-session-round2/{inventory-before,transient-before,native-before}.log.

Review tests imported unchanged and pushed individually: 445d4abb (DS-01), 48b07c5d (DS-03), 796b4f11 (DS-02), add06701 (200 replies), 8cac789a (DS-04). Intentional failing-test commits per the approved brief. DS-01 reproduced: exited→open, 0 pass/1 fail, 81.53 ms. DS-04 reproduced on production build/port5791: attach1→1, socket opens1→1/closes0→0 over the 10-second retry window, 1 fail. Native build initially stopped because make is absent; rerunning in the existing documented Nix make/perl shell. Logs: ~/.cache/codex-scratch/fix-dead-session-round2/{inventory-before,transient-before,native-before}.log.

kayg referenced this issue from a commit

2026-10-10 22:26:02 +00:00

test: bound native final reply draining for a stuck peer (#158)

kayg commented

2026-10-10 22:37:06 +00:00

Author

Owner

Candidate fixes: daemon.rs drains native FIFO after every successful terminal return, whole-drain timeout1s (also covers unsupported/refused shim Close paths); shared mux writer was already independent. live-inventory.ts preserves gone/exited on selection. terminal.ts retries shared Close on the SAME stream id (retains durable input writer/cursors), reusing connection.ts jitter/stability policy; cancels retry on socket lifecycle, end, retarget and dispose. Tracer check PASS; source units122PASS/0FAIL; production browser dead-session/detached-output/transient-review3PASS,46.3s, port5791/default300msRTT. Transient proof: attach1→2, socket opens1→1/closes0→0, same session. Five fresh production deferred Brotli builds:44693,44771,44759,44772,44697 B, all below45056 (284–363 B margin). No budget changes. Native preserved-before repro/full fast are queued behind the ongoing OpenSSL build.

kayg commented

2026-10-10 23:14:37 +00:00

Author

Owner

DS-03 candidate now passes the unchanged real bound QUIC review regression (1PASS,0.12s): final Close4404 arrives before stream retirement. Companion stuck-peer/drain test also passes; filtered two-test run2PASS/0FAIL in1.01s. Native logs originally named native-before/native-bounds-before became candidate runs because source was repaired while C dependencies were building; they are NOT failing-before evidence. Exact pre-fix cbabf63e archive is now compiling its own two-test reproduction in native-preserved-before.log. Full fast has passed RTT/fmt/VT fmt and is waiting for that native build lock; static baseline a552bb2e build is progressing independently. No fix commit yet.

DS-03 candidate now passes the unchanged real bound QUIC review regression (1PASS,0.12s): final Close4404 arrives before stream retirement. Companion stuck-peer/drain test also passes; filtered two-test run2PASS/0FAIL in1.01s. Native logs originally named native-before/native-bounds-before became candidate runs because source was repaired while C dependencies were building; they are NOT failing-before evidence. Exact pre-fix cbabf63e archive is now compiling its own two-test reproduction in native-preserved-before.log. Full fast has passed RTT/fmt/VT fmt and is waiting for that native build lock; static baseline a552bb2e build is progressing independently. No fix commit yet.

kayg commented

2026-10-10 23:17:01 +00:00

Author

Owner

Exact pre-fix cbabf63e archive reproduced both DS-03 probes:0PASS/2FAIL in0.02s. Real native QUIC: UnexpectedEof before final Close4404. Stuck peer: returned Ok immediately instead of attempting a bounded drain. Candidate source: unchanged real QUIC test1PASS in0.12s; both probes2PASS/0FAIL in1.01s. Logs native-preserved-before.log and native-bounds-before.log under job scratch. Clippy/full fast now progressing after cold C dependencies completed. Additional production input regression PASS12.6s (22.6s including build):11 exact bytes, same stream id/session, no socket replacement; source test and mock-only stream-id metadata will be a separate test commit.

Exact pre-fix cbabf63e archive reproduced both DS-03 probes:0PASS/2FAIL in0.02s. Real native QUIC: UnexpectedEof before final Close4404. Stuck peer: returned Ok immediately instead of attempting a bounded drain. Candidate source: unchanged real QUIC test1PASS in0.12s; both probes2PASS/0FAIL in1.01s. Logs native-preserved-before.log and native-bounds-before.log under job scratch. Clippy/full fast now progressing after cold C dependencies completed. Additional production input regression PASS12.6s (22.6s including build):11 exact bytes, same stream id/session, no socket replacement; source test and mock-only stream-id metadata will be a separate test commit.

kayg commented

2026-10-10 23:56:22 +00:00

Author

Owner

Full gate findings, not waived: clippy caught doc_markdown in the new stuck-peer fixture (AttachTerm needed backticks), corrected in source. Workspace tests stopped at vt_reconnect/snapshot_limits_never_close_live_reconnect: wrapped retention/approximation/legacy case PASS; next fixture shell open returned terminal open refused,27.83s. Isolated unchanged-budget rerun in progress. Queued full rerun stopped at RTT self-test p50=323.17/p99=360.31ms; new full gate required after own compilation settles. GNU gate reuses this job’s already tested locked vendored OpenSSL3.6.3 archive through target-prefixed openssl-sys build variables; no library/version/manifest/compiler flags changed. Musl remains default vendored recipe. Exact a552bb2e x86 static baseline completed:19,904,384B, stripped/static PIE/no PT_INTERP. Logs fast-final.log,fast-corrected.log,reconnect-isolated.log,static-base.log.

Full gate findings, not waived: clippy caught doc_markdown in the new stuck-peer fixture (AttachTerm needed backticks), corrected in source. Workspace tests stopped at vt_reconnect/snapshot_limits_never_close_live_reconnect: wrapped retention/approximation/legacy case PASS; next fixture shell open returned terminal open refused,27.83s. Isolated unchanged-budget rerun in progress. Queued full rerun stopped at RTT self-test p50=323.17/p99=360.31ms; new full gate required after own compilation settles. GNU gate reuses this job’s already tested locked vendored OpenSSL3.6.3 archive through target-prefixed openssl-sys build variables; no library/version/manifest/compiler flags changed. Musl remains default vendored recipe. Exact a552bb2e x86 static baseline completed:19,904,384B, stripped/static PIE/no PT_INTERP. Logs fast-final.log,fast-corrected.log,reconnect-isolated.log,static-base.log.

kayg commented

2026-10-11 00:09:34 +00:00

Author

Owner

Unchanged vt_reconnect test passed in isolation: snapshot_limits_never_close_live_reconnect,1 PASS/0 FAIL,19.05s (reconnect-isolated.log). Corrected full scripts/verify fast rerun is progressing; fmt,VT native fmt,strict clippy PASS. RTT self-test p50=315.33/p99=326.12ms PASS. No failed row has been waived. Five default production builds passed deferred Brotli:44,693/44,771/44,759/44,772/44,697B,all below45,056B. Exact a552bb2e static baseline19,904,384B; candidate size pending. Logs under $HOME/.cache/codex-scratch/fix-dead-session-round2/.

Unchanged vt_reconnect test passed in isolation: snapshot_limits_never_close_live_reconnect,1 PASS/0 FAIL,19.05s (reconnect-isolated.log). Corrected full scripts/verify fast rerun is progressing; fmt,VT native fmt,strict clippy PASS. RTT self-test p50=315.33/p99=326.12ms PASS. No failed row has been waived. Five default production builds passed deferred Brotli:44,693/44,771/44,759/44,772/44,697B,all below45,056B. Exact a552bb2e static baseline19,904,384B; candidate size pending. Logs under $HOME/.cache/codex-scratch/fix-dead-session-round2/.

kayg referenced this issue from a commit

2026-10-11 00:23:57 +00:00

test: mark the native attach type as code in fixture docs (#158)

kayg referenced this issue from a commit

2026-10-11 00:25:27 +00:00

native terminal: drain final frames before retiring the stream (#158)

kayg commented

2026-10-11 00:26:11 +00:00

Author

Owner

DS-03 complete and pushed:d215b3c0 (fixture doc lint correction:3b55079c). Successful native terminal completion drains queued final frames before retirement,with1s stuck-peer timeout. Real QUIC final-close and blocked-peer tests:before0/2 PASS,after2/2 PASS; all other immediate final-send/Ok returns use this same completion path; shared writer is independent. scripts/verify fast:fmt/VT native fmt/clippy/test/cargo-deny PASS; static builds FAIL(1),known overrun19,904,800B versus exact a552bb2e19,904,384B (+416B). TS fast PASS separately (static failure stops combined dispatcher); protocol36PASS/34,462 assertions. Review diagnostic200 sequential vanished-session refusals on one real Unix channel PASS: p50=210us,p99=4526us,max=4777us,late Input/Close/Ping fences retained. Local infraternal-dev,8 CPUs,CPUWeight10000,initial/final load8.14/13.87/21.11. Raw logs fast-verified.log,ts-fast.log,gone-200.log in job scratch. No failed test waived; unchanged reconnect/shim reruns and full workspace suite passed.

kayg referenced this issue from a commit

2026-10-11 00:26:36 +00:00

sessions: preserve final gone state when another window selects it (#158)

kayg commented

2026-10-11 00:27:01 +00:00

Author

Owner

DS-01 complete and pushed:003126ad. Selection leaves a known gone session exited,without changing frecency or saving it again. Unchanged gone-review test:before0PASS/1FAIL (exited->open),after1PASS/0FAIL,3 assertions. Tracer check and full src suite122PASS/0FAIL,10,862 assertions; shared inventory/canvas tests passed TS fast. D206 final UUID stays final; New shell uses a new UUID.

kayg referenced this issue from a commit

2026-10-11 00:27:22 +00:00

connection: share the existing jittered retry policy (#158)

kayg referenced this issue from a commit

2026-10-11 00:27:51 +00:00

terminal: retry transient shared-stream closes without dropping siblings (#158)

kayg referenced this issue from a commit

2026-10-11 00:28:38 +00:00

test: expose attached stream ids in the terminal fixture (#158)

kayg referenced this issue from a commit

2026-10-11 00:28:59 +00:00

test: preserve acknowledged input across a shared terminal retry (#158)

kayg commented

2026-10-11 00:29:26 +00:00

Author

Owner

DS-04 complete and pushed:be84bd8f shares existing socket backoff;14f0fdca retries shared streams on transient Close while keeping the durable stream/writer id and cursors;16dc989c adds test-only stream-id metadata;fbc28c18 adds nonzero-input continuity regression. No timer for live idle terminals; retries cancel on lifecycle transitions and final4404. Production review transient test:before attach1->1 FAIL,after1->2 PASS,same UUID,socket opens1->1/closes0->0. New continuity spec PASS,echo beforeafter exactly once,input cursor11,same stream id. Production dead-session/detached-output/review transient suite3/3 PASS; full tracer check/src122/122 PASS; transfer gates passed all five default builds. Final SHA builds/browser replay pending to include build metadata.

kayg commented

2026-10-11 00:32:04 +00:00

Author

Owner

DS-02 complete at code head fbc28c18:five fresh default production builds deferred Brotli44,704/44,708/44,720/44,709/44,679B,all below unchanged D188 limit45,056B (minimum headroom336B). Each transfer-budget and independent review transfer test PASS. No further lazy-loading change needed beyond a552bb2e logs-window on-demand body. Final bun check and bun test src:122PASS/0FAIL,10,862 assertions. Logs final-build-{1..5}.log,final-transfer-{1..5}.log,final-tracer.log under job scratch. Browser production replay underway on5791/5792.

DS-02 complete at code head fbc28c18:five fresh default production builds deferred Brotli44,704/44,708/44,720/44,709/44,679B,all below unchanged D188 limit45,056B (minimum headroom336B). Each transfer-budget and independent review transfer test PASS. No further lazy-loading change needed beyond a552bb2e logs-window on-demand body. Final bun check and bun test src:122PASS/0FAIL,10,862 assertions. Logs final-build-{1..5}.log,final-transfer-{1..5}.log,final-tracer.log under job scratch. Browser production replay underway on5791/5792.

kayg commented

2026-10-11 00:34:23 +00:00

Author

Owner

Final production browser replay at fbc28c18:4/4 PASS,58.5s,on TRACER_TEST_PORT5791/upstream5792 (300ms RTT profile). Specs:dead-session,detached-output,transient-session-review,transient-session-input. Production screenshot attached by dead-session spec:apps/tracer/captures/dead-session/ended-1440x900-ink.png; no visual-quality verdict. Author checklist:unchanged wire/auth contracts and imported assertions; audited immediate final-frame/Ok paths; bounded native drain/retirement proven; shared socket and durable stream id/input cursors retained; retry cancellation/final4404 guards reviewed; idle terminals add no retry timer; existing UI reused; D188 budgets unchanged; no dependencies/engine choice. All four HOLD items addressed. Only known static-size overrun remains (+416B versus a552bb2e). Final full gate replay at pushed code head in progress; earlier full workspace gate passed every other row.

Final production browser replay at fbc28c18:4/4 PASS,58.5s,on TRACER_TEST_PORT5791/upstream5792 (300ms RTT profile). Specs:dead-session,detached-output,transient-session-review,transient-session-input. Production screenshot attached by dead-session spec:apps/tracer/captures/dead-session/ended-1440x900-ink.png; no visual-quality verdict. Author checklist:unchanged wire/auth contracts and imported assertions; audited immediate final-frame/Ok paths; bounded native drain/retirement proven; shared socket and durable stream id/input cursors retained; retry cancellation/final4404 guards reviewed; idle terminals add no retry timer; existing UI reused; D188 budgets unchanged; no dependencies/engine choice. All four HOLD items addressed. Only known static-size overrun remains (+416B versus a552bb2e). Final full gate replay at pushed code head in progress; earlier full workspace gate passed every other row.

kayg referenced this issue from a commit

2026-10-11 00:44:50 +00:00

docs: record the dead-session HOLD repairs and gate evidence (#158)

kayg commented

2026-10-11 00:45:54 +00:00

Author

Owner

Round2 finished and pushed. Head:80d6da0a96fb04dd7bf4b2227a37c0ae7c8d269f; code head:fbc28c18; result.md records atomic SHAs and evidence. DS-03/01/04/02 addressed; every imported review assertion preserved and PASS, including real QUIC final close, sticky inventory, shared transient retry and isolated200-refusal diagnostic.

scripts/verify fast (final pushed code-head replay),verbatim summary:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Known pre-existing static-size overrun only:19,904,800B vs exact a552bb2e19,904,384B (+416B),unchanged maximum16,404,816B. Default musl release remains stripped/static PIE/no PT_INTERP; GNU checks reused this job's locked vendored archive. No failed test waived. TypeScript fast PASS separately because static failure stops dispatcher.

Final tracer check/src:122PASS/0FAIL,10,862 assertions. Protocol:36PASS,34,462 assertions. Production browser:4PASS/58.5s,on5791/5792 (dead-session,detached-output,review transient,nonzero-input continuity). Five fresh default production builds at fbc28c18,deferred Brotli:44,704/44,708/44,720/44,709/44,679B; all transfer and review gates PASS under unchanged45,056B limit,min margin336B.

200 sequential real-channel vanished replies PASS:p50=210us,p99=4526us,max=4777us,late Input/Close/Ping fences retained. Local infraternal-dev,8 CPUs,CPUWeight10000; initial/final load8.14/13.87/21.11. This measures sequential refusals,not200live PTYs. Screenshot:apps/tracer/captures/dead-session/ended-1440x900-ink.png. Author checklist recorded in previous comment. Logs/artifacts retained under $HOME/.cache/codex-scratch/fix-dead-session-round2/ (fast-code-head.log,ts-fast.log,final-tracer.log,final-browser.log,final-transfer-{1..5}.log,gone-200.log,static-base.log,static-final.log). Build targets and temporary source checkouts removed. Working tree clean; origin matches head. Issue left open.

Round2 finished and pushed. Head:80d6da0a96fb04dd7bf4b2227a37c0ae7c8d269f; code head:fbc28c18; result.md records atomic SHAs and evidence. DS-03/01/04/02 addressed; every imported review assertion preserved and PASS, including real QUIC final close, sticky inventory, shared transient retry and isolated200-refusal diagnostic. scripts/verify fast (final pushed code-head replay),verbatim summary: ``` +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` Known pre-existing static-size overrun only:19,904,800B vs exact a552bb2e19,904,384B (+416B),unchanged maximum16,404,816B. Default musl release remains stripped/static PIE/no PT_INTERP; GNU checks reused this job's locked vendored archive. No failed test waived. TypeScript fast PASS separately because static failure stops dispatcher. Final tracer check/src:122PASS/0FAIL,10,862 assertions. Protocol:36PASS,34,462 assertions. Production browser:4PASS/58.5s,on5791/5792 (dead-session,detached-output,review transient,nonzero-input continuity). Five fresh default production builds at fbc28c18,deferred Brotli:44,704/44,708/44,720/44,709/44,679B; all transfer and review gates PASS under unchanged45,056B limit,min margin336B. 200 sequential real-channel vanished replies PASS:p50=210us,p99=4526us,max=4777us,late Input/Close/Ping fences retained. Local infraternal-dev,8 CPUs,CPUWeight10000; initial/final load8.14/13.87/21.11. This measures sequential refusals,not200live PTYs. Screenshot:apps/tracer/captures/dead-session/ended-1440x900-ink.png. Author checklist recorded in previous comment. Logs/artifacts retained under $HOME/.cache/codex-scratch/fix-dead-session-round2/ (fast-code-head.log,ts-fast.log,final-tracer.log,final-browser.log,final-transfer-{1..5}.log,gone-200.log,static-base.log,static-final.log). Build targets and temporary source checkouts removed. Working tree clean; origin matches head. Issue left open.

kayg referenced this issue

2026-10-11 00:52:43 +00:00

Terminal opens refused at 128 sessions; dead sessions linger; background sessions invisible #151

kayg commented

2026-10-11 00:52:43 +00:00

Author

Owner

Started production integration on session-bounds-prod, base b373cc4c66 (release-prod-2 plus #158 head 80d6da0a). Read all #151/#158 comments; no parent linked. Merge origin/session-bounds 8b0f74f4, preserve SB-01/DS-01..04 tests and use ordered terminal.exited status followed by the existing final Close 4404. Gates use local CPUWeight10000 scopes and browser ports5795/5796. No deployment.

Started production integration on session-bounds-prod, base b373cc4c6661571ff41ef56eb8dee59380ea6304 (release-prod-2 plus #158 head 80d6da0a). Read all #151/#158 comments; no parent linked. Merge origin/session-bounds 8b0f74f4, preserve SB-01/DS-01..04 tests and use ordered terminal.exited status followed by the existing final Close 4404. Gates use local CPUWeight10000 scopes and browser ports5795/5796. No deployment.

kayg commented

2026-10-11 00:54:45 +00:00

Author

Owner

Delta independent verification started: review-fix-dead-session at target 80d6da0a96, merge base with dev 2eaaaa36. Scope 523daf6c..80d6da0a only plus DS-01..04. Read all 29 #158 comments and linked #151 history; no parent issue linked. Prior local review branch preserved as review-fix-dead-session-prior-355a53e8. All five earlier test commits already imported unchanged by author; no duplicate cherry-picks needed. Evidence local, no push/deploy per final reviewer brief.

Delta independent verification started: review-fix-dead-session at target 80d6da0a96fb04dd7bf4b2227a37c0ae7c8d269f, merge base with dev 2eaaaa36. Scope 523daf6c..80d6da0a only plus DS-01..04. Read all 29 #158 comments and linked #151 history; no parent issue linked. Prior local review branch preserved as review-fix-dead-session-prior-355a53e8. All five earlier test commits already imported unchanged by author; no duplicate cherry-picks needed. Evidence local, no push/deploy per final reviewer brief.

kayg commented

2026-10-11 01:06:30 +00:00

Author

Owner

Delta finding DS-05 P2: newly on-demand logs body at apps/tracer/src/canvas/views/body.ts:58 handles success only. Refusing its production chunk leaves a visible logs window empty with no retry action (onDemand documents retry on rejection). Normal test a refused logs-body download offers a retry in its window: 0 PASS/1 FAIL; retry element absent after 5000ms. Test committed locally on review-fix-dead-session. Existing two logs browser specs PASS. No current product path opens logs, so P2/nonblocking under D206. Prior DS-01/02/04 tests PASS; DS-03 Rust gate still compiling. Weighted tracer src rerun122PASS/0FAIL; initial two unchanged Mesh timeouts retained. Five default transfer builds44699/44743/44729/44720/44796B, all below45056.

kayg commented

2026-10-11 01:17:17 +00:00

Author

Owner

Completed delta behavioral evidence on local review-fix-dead-session9a2a789c: DS-01 unit, DS-02 production transfer, DS-04 production retry PASS; original assertions unchanged (native real QUIC function also byte-identical, execution pending source build). Browser6/6PASS in1.5m; weighted tracer122/122PASS in4.92s after first120/122 run with two unchanged Mesh timeouts. Build graph/precompression5/5PASS; final transfer critical gzip289267/307200,Brotli240435/256000; deferred gzip50770/51200,Brotli44757/45056. DS-05 P2 failing test committed9a2a789c (download refused, no retry after5s). Paired retained artifacts real PTY100samples: same-session flood final468.49ms/prior424.22ms p99, both above334ms; baseline/sibling flood pass both. Load sampled during run24.83/32.76/30.10,CPUWeight10000; no causal regression established under concurrent host builds. Fresh clippy PASS19m50s; native/source/full fast still compiling. No push/deploy.

kayg referenced this issue

2026-10-11 01:20:41 +00:00

Terminal opens refused at 128 sessions; dead sessions linger; background sessions invisible #151

kayg commented

2026-10-11 01:20:45 +00:00

Author

Owner

Evidence step committed locally22df46fa: docs/reviews/dead-session-158-delta.md, production logs screenshots, five raw proof excerpts. DS-05 failing test9a2a789c remains unchanged; P0=0/P1=0/P2=1/P3=0 newly confirmed findings. Earlier DS-03 source rerun remains pending, so final verdict not issued yet. Fresh clippy took19m50s; native test compilation holds the workspace gate lock. Review exceeds the approximate30minute target to finish required source/gate verification. No implementation fixes,push,deployment or issue closure.

kayg commented

2026-10-11 01:31:34 +00:00

Author

Owner

Fresh gate restarted after stopping only this review's redundant GNU OpenSSL build and waiting initial gate. Reuses this review's already-built exact locked3.6.3 static GNU archives (SHA256 recorded); target-prefixed build environment only, default musl recipe unchanged. Initial gate remains incomplete evidence. Full restart: RTT295.34/322.17ms PASS; fmt/VTfmt/clippy PASS (clippy37.09s with cache). Workspace source tests compiling. Evidence HEAD22df46fa; DS-05 test9a2a789c. No budgets/dependencies/implementation changed.

kayg commented

2026-10-11 01:36:18 +00:00

Author

Owner

Read-only integration review found two P1 races; failing-first proof now recorded. Native retiring-handshake plus16 in-flight Credit: command receiver closure cancels saved answer (InvalidData/native terminal command queue unavailable). Saved-status native stuck peer also exceeds3s instead of reaching#158's1s drain; focused native run3PASS/2FAIL in3.00s. Candidate: closed command sink yields to terminal completion; retired events use ordered FIFO without live ownership-delivery ack, so existing whole-native drain covers BOTH saved event and4404. Live exits retain ack-before-owner-release.
Client drop between retired Event and4404: production spec FAIL20s, no ended strip; retired pendingExit incorrectly blocks reattach although it never completes through paint. Candidate: preserve saved status but allow socket/shared retries until4404; live pending exits still block input/retry until paint. Existing final4404 remains sole final transition. Logs native-races-before.log, retired-disconnect-before.log; unchanged-after tests in progress.
Initial full test gate also found fixture setup FileNotFoundError target/ac, not behavioral failure. Linked ignored target to this job scratch build directory (as prior#151 run); corrected full replay required, no failure waived.

Read-only integration review found two P1 races; failing-first proof now recorded. Native retiring-handshake plus16 in-flight Credit: command receiver closure cancels saved answer (InvalidData/native terminal command queue unavailable). Saved-status native stuck peer also exceeds3s instead of reaching#158's1s drain; focused native run3PASS/2FAIL in3.00s. Candidate: closed command sink yields to terminal completion; retired events use ordered FIFO without live ownership-delivery ack, so existing whole-native drain covers BOTH saved event and4404. Live exits retain ack-before-owner-release. Client drop between retired Event and4404: production spec FAIL20s, no ended strip; retired pendingExit incorrectly blocks reattach although it never completes through paint. Candidate: preserve saved status but allow socket/shared retries until4404; live pending exits still block input/retry until paint. Existing final4404 remains sole final transition. Logs native-races-before.log, retired-disconnect-before.log; unchanged-after tests in progress. Initial full test gate also found fixture setup FileNotFoundError target/ac<PID>, not behavioral failure. Linked ignored target to this job scratch build directory (as prior#151 run); corrected full replay required, no failure waived.

kayg commented

2026-10-11 01:41:10 +00:00

Author

Owner

Completed fresh-source native verification, evidence commit25e70bc7 on review-fix-dead-session. DS-03 unchanged real QUIC final-Close test and stuck-peer bound2/2PASS in1.00s. All earlier DS-01..04 now PASS; new DS-05 P2 remains (test9a2a789c). No P0/P1 remain. Fresh200 sequential vanished replies on one real inherited channel plus all late Input/Close/Ping fences PASS,p50=216us/p99=1474us/max=2226us,0.11s; CPUWeight10000,infraternal-dev8 CPUs,initial/final load26.62/40.26/35.38. Not200 simultaneous live PTYs. Workspace fast gate progressing through real reconnect tests; static build/output still pending. Logs native-final.log/scale-final.log in delta scratch; no push/deploy/fix.

kayg commented

2026-10-11 01:49:31 +00:00

Author

Owner

Fresh target-code workspace tests completed without failures; dependency checks PASS. Native2/2 and200-refusal diagnostic independently PASS; standard separate TypeScript fast gate PASS including36 protocol tests/34462 assertions in4.73s. Full scripts/verify fast is now building the default musl static release; final summary pending. Evidence branch25e70bc7, all earlier DS-01..04 resolved, new DS-05 P2 test9a2a789c remains. No P0/P1. Final verdict will include actual static output; no implementation changes,push/deploy or timeout waiver.

kayg commented

2026-10-11 01:57:28 +00:00

Author

Owner

Fresh source Rust workspace gate:576PASS/0FAIL/62ignored; fmt/VTfmt/clippy/cargo-deny PASS, no Rust test waiver. Separate standard TypeScript fast gate PASS, including36protocol tests/34462 assertions in4.73s. Default musl release waits only for its vendored OpenSSL build before linking. All four prior findings pass, new DS-05 P2 remains. Evidence25e70bc7; verdict MERGE(noP0/P1) will be posted with the actual completed static gate output. Scratch logs preserved; no implementation changes or push/deploy.

kayg commented

2026-10-11 02:06:17 +00:00

Author

Owner

Independent delta review finished. Verdict: merge — MERGE (no P0/P1). Findings P0=0/P1=0/P2=1/P3=0.
Target 80d6da0a; scope 523daf6c..80d6da0a only. Local review-fix-dead-session HEAD d801b7e3bb. All earlier DS-01..04 tests PASS; original assertions preserved. New DS-05 P2: refused on-demand logs download leaves a blank window without retry; no current product entry point. Failing production test 9a2a789c, 0 PASS/1 FAIL. Implementation unchanged.

Fresh source fast: 576 Rust PASS/0 FAIL/62 ignored; native QUIC/stuck-peer 2/2 PASS. 200 sequential refusals/fences on one real channel PASS, p50 216us/p99 1474us/max 2226us; CPUWeight=10000,8 CPUs,load26.62/40.26/35.38. Not 200 live PTYs. Standard separate TS fast PASS,36 protocol tests/34462 assertions. Browser6/6 PASS,1.5m,ports5863/5864; DS-05 at5865/5866. Five default deferred Brotli builds44699..44796B below45056.

Fresh stripped static x86:19904832B,no PT_INTERP; historical16404816B maximum fails(D205). Observed +416B vs retained523daf6c,+32B vs author final; cross-build diagnostics. Default musl recipe unchanged; owned locked GNU OpenSSL cache reuse disclosed. No Rust test waiver. Native same-session flood fails both prior/final artifacts:424.22/468.49ms p99 vs334ms; baseline/sibling flood pass. No causal delta regression established under shared load; no green latency gate claimed.

Evidence docs/reviews/dead-session-158-delta.md; commits9a2a789c/22df46fa/25e70bc7/d801b7e3. Owned build target removed; binary/logs retained in ~/.cache/codex-scratch/fix-dead-session-review/delta/. No push/deploy/issue closure.

Completed scripts/verify fast on target code, verbatim:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Independent delta review finished. Verdict: **merge — MERGE (no P0/P1)**. Findings P0=0/P1=0/P2=1/P3=0. Target 80d6da0a; scope 523daf6c..80d6da0a only. Local review-fix-dead-session HEAD d801b7e3bb4f49ac5a42bba31ff2555a329627f0. All earlier DS-01..04 tests PASS; original assertions preserved. New DS-05 P2: refused on-demand logs download leaves a blank window without retry; no current product entry point. Failing production test 9a2a789c, 0 PASS/1 FAIL. Implementation unchanged. Fresh source fast: 576 Rust PASS/0 FAIL/62 ignored; native QUIC/stuck-peer 2/2 PASS. 200 sequential refusals/fences on one real channel PASS, p50 216us/p99 1474us/max 2226us; CPUWeight=10000,8 CPUs,load26.62/40.26/35.38. Not 200 live PTYs. Standard separate TS fast PASS,36 protocol tests/34462 assertions. Browser6/6 PASS,1.5m,ports5863/5864; DS-05 at5865/5866. Five default deferred Brotli builds44699..44796B below45056. Fresh stripped static x86:19904832B,no PT_INTERP; historical16404816B maximum fails(D205). Observed +416B vs retained523daf6c,+32B vs author final; cross-build diagnostics. Default musl recipe unchanged; owned locked GNU OpenSSL cache reuse disclosed. No Rust test waiver. Native same-session flood fails both prior/final artifacts:424.22/468.49ms p99 vs334ms; baseline/sibling flood pass. No causal delta regression established under shared load; no green latency gate claimed. Evidence docs/reviews/dead-session-158-delta.md; commits9a2a789c/22df46fa/25e70bc7/d801b7e3. Owned build target removed; binary/logs retained in ~/.cache/codex-scratch/fix-dead-session-review/delta/. No push/deploy/issue closure. Completed scripts/verify fast on target code, verbatim: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ```

kayg referenced this issue from a commit

2026-10-11 02:15:11 +00:00

Merge fix-dead-session into release-prod-2: ended sessions stop the reconnect loop (#158)

kayg referenced this issue from a commit

2026-10-11 02:58:20 +00:00

Merge session-bounds into the production ended-session path (#151)

kayg commented

2026-10-11 03:05:00 +00:00

Author

Owner

Finished and pushed session-bounds-prod HEAD 04cdc52a01. Commits: 026e471c (merge/conflict resolutions and single authenticated saved status -> final Close 4404 path), f2d84235 (detached-output fixture attachment barrier), 04cdc52a (short result.md). Both test suites retained; contract choice and every reproduced integration finding recorded above. No deployment; issues remain open.

Final scripts/verify fast output (fast-last.log), verbatim:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| clippy                    | PASS         |
| test                      | PASS         |
| cargo-deny                | PASS         |
| static builds             | FAIL (1)     |
+---------------------------+--------------+

Rust 601 passed/0 failed/63 ignored, including SB-01 and DS-01..DS-04. No load failure waived; dev mesh-headroom baseline passed, final full target retry also passed that test and bootstrap confirmation. Tracer check/src 142 passed (10,901 assertions); protocol 36 passed (34,462 assertions), TS fast passed separately because static failure stops combined dispatcher. Production browser 7 passed in 1.8 min; real-PTY switch 7 passed in 1.0 min. Native gone_review 5 passed in 1.05 s. Three default builds deferred Brotli 44,773/44,705/44,734 B <= unchanged 45,056 B.

Known static exception only: final 19,992,480 B, exact b373cc4c 19,904,832 B, delta +87,648 B, no PT_INTERP; frozen limit 16,404,816 B. Real 200-session scale: 200 recovered/201 admitted/200 after reap; PSS 341,851 KiB empty (1,709.26/session), 3,710,906 KiB filled (18,554.53/session), 200 simultaneous owners add no measured PSS; facts 101 p50 4.589 ms/max 44.399 ms. infraternal-dev/8 CPUs/CPUWeight=10000, load 25.68/27.04/25.28 -> 33.93/28.81/25.88. Separate 200 sequential gone replies p50 232 µs/p99 700 µs/max 1,912 µs, load 46.25/36.84/29.80.

Author checklist completed; read-only code re-review found no remaining concrete issue and checked production exit 0/7 captures (status/actions visible, no clipping/overlap). Captures: apps/tracer/captures/dead-session/retired-exit-0-1440x900-ink.png and retired-exit-7-1440x900-ink.png. Evidence report: result.md. Raw logs/binaries/captures retained under $HOME/.cache/codex-scratch/session-bounds-prod/; owned targets/baseline checkouts/runtime removed. Code and test content unchanged after final gates; final documentation quotes that evidence.

Finished and pushed session-bounds-prod HEAD 04cdc52a0147e2a281b5545caafca014096dec9e. Commits: 026e471c (merge/conflict resolutions and single authenticated saved status -> final Close 4404 path), f2d84235 (detached-output fixture attachment barrier), 04cdc52a (short result.md). Both test suites retained; contract choice and every reproduced integration finding recorded above. No deployment; issues remain open. Final scripts/verify fast output (fast-last.log), verbatim: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | clippy | PASS | | test | PASS | | cargo-deny | PASS | | static builds | FAIL (1) | +---------------------------+--------------+ ``` Rust 601 passed/0 failed/63 ignored, including SB-01 and DS-01..DS-04. No load failure waived; dev mesh-headroom baseline passed, final full target retry also passed that test and bootstrap confirmation. Tracer check/src 142 passed (10,901 assertions); protocol 36 passed (34,462 assertions), TS fast passed separately because static failure stops combined dispatcher. Production browser 7 passed in 1.8 min; real-PTY switch 7 passed in 1.0 min. Native gone_review 5 passed in 1.05 s. Three default builds deferred Brotli 44,773/44,705/44,734 B <= unchanged 45,056 B. Known static exception only: final 19,992,480 B, exact b373cc4c 19,904,832 B, delta +87,648 B, no PT_INTERP; frozen limit 16,404,816 B. Real 200-session scale: 200 recovered/201 admitted/200 after reap; PSS 341,851 KiB empty (1,709.26/session), 3,710,906 KiB filled (18,554.53/session), 200 simultaneous owners add no measured PSS; facts 101 p50 4.589 ms/max 44.399 ms. infraternal-dev/8 CPUs/CPUWeight=10000, load 25.68/27.04/25.28 -> 33.93/28.81/25.88. Separate 200 sequential gone replies p50 232 µs/p99 700 µs/max 1,912 µs, load 46.25/36.84/29.80. Author checklist completed; read-only code re-review found no remaining concrete issue and checked production exit 0/7 captures (status/actions visible, no clipping/overlap). Captures: apps/tracer/captures/dead-session/retired-exit-0-1440x900-ink.png and retired-exit-7-1440x900-ink.png. Evidence report: result.md. Raw logs/binaries/captures retained under $HOME/.cache/codex-scratch/session-bounds-prod/; owned targets/baseline checkouts/runtime removed. Code and test content unchanged after final gates; final documentation quotes that evidence.

kayg referenced this issue

2026-10-11 03:10:27 +00:00

Standalone TLS: /auth/session returns 401 right after passkey registration #149

kayg commented

2026-10-11 03:28:26 +00:00

Author

Owner

Deployed to production: release-prod-2 at 1c08c4b7 (merge of fix-dead-session 80d6da0a; delta review MERGE, no P0/P1). Backup /var/backups/infraternal/20261011-052732-1c08c4b7.

Pre-deploy on the production line: tsc PASS; bun test src 137/0; transfer budget deferred brotli 44590/45056; Playwright dead-session, detached-output, palette-polish, shortcuts-d222, phone-gutter: 39 passed (6.1m).

monitor01 journal: the "unknown terminal session" loop logged 312 lines and stopped at 2026-10-10 17:29 UTC (window closed), so there was no live loop to observe; 0 such lines since the restart, no warnings. Open P2 from the delta review (logs-body download has no retry) is fixed forward later.

Deployed to production: release-prod-2 at 1c08c4b7 (merge of fix-dead-session 80d6da0a; delta review MERGE, no P0/P1). Backup /var/backups/infraternal/20261011-052732-1c08c4b7. Pre-deploy on the production line: tsc PASS; bun test src 137/0; transfer budget deferred brotli 44590/45056; Playwright dead-session, detached-output, palette-polish, shortcuts-d222, phone-gutter: 39 passed (6.1m). monitor01 journal: the "unknown terminal session" loop logged 312 lines and stopped at 2026-10-10 17:29 UTC (window closed), so there was no live loop to observe; 0 such lines since the restart, no warnings. Open P2 from the delta review (logs-body download has no retry) is fixed forward later.

kayg referenced this issue from a commit

2026-10-11 04:17:44 +00:00

Merge TLS session cookie fix onto production (#149; D206)

kayg referenced this issue

2026-10-11 04:18:25 +00:00

Standalone TLS: /auth/session returns 401 right after passkey registration #149

kayg referenced this issue

2026-10-11 05:14:29 +00:00

Standalone TLS: /auth/session returns 401 right after passkey registration #149

kayg referenced this issue from a commit

2026-10-11 10:57:07 +00:00

Merge prod-batch into release-prod-2: deployed 707a4572 (#151 #149 #154, machine sections, CA unlock, inbox)