Gate: idle canary fails on both baseline and vt-ghostty

kayg commented

2026-10-06 15:42:29 +00:00

Owner

The historical idle canary in the fast gate fails on both the t-agent baseline (e72589e) and the vt-ghostty head. The vt-ghostty author reported this, so it is not caused by the engine switch.

To do:

Decide whether the canary itself is broken (a stale threshold, or measuring the wrong process) or a real idle-wakeup regression crept into t-agent.
Fix the gate or the regression.

DECISIONS 41 and 81 require zero idle wakeups.

The historical idle canary in the fast gate **fails on both** the t-agent baseline (e72589e) and the vt-ghostty head. The vt-ghostty author reported this, so it is not caused by the engine switch. To do: - Decide whether the canary itself is broken (a stale threshold, or measuring the wrong process) or a real idle-wakeup regression crept into t-agent. - Fix the gate or the regression. DECISIONS 41 and 81 require zero idle wakeups.

kayg referenced this issue

2026-10-06 15:42:51 +00:00

Status board: work in flight (kept current) #24

kayg referenced this issue

2026-10-06 15:43:28 +00:00

Status board: work in flight (kept current) #24

kayg commented

2026-10-06 15:45:15 +00:00

Author

Owner

Starting investigation on branch t-agent at base 6c64ede107. Read issue #27 and parent #5 history, CLAUDE.md, DECISIONS.md, and docs/ARCHITECTURE.md.

Starting investigation on branch t-agent at base 6c64ede10745b690c27ece4e035a7455fedd5a0a. Read issue #27 and parent #5 history, CLAUDE.md, DECISIONS.md, and docs/ARCHITECTURE.md.

kayg commented

2026-10-06 16:00:24 +00:00

Author

Owner

Finding: scripts/verify-agent.py:26-32 measures only the spawned infraternal agent's RSS and utime/stime equality across 500 ms, before terminal.open. crates/infraternal/tests/agent_daemon.rs:8-15 is its fast-gate caller. It does not count scheduled wakeups and covers neither shim nor server; currently initializes auth then exits 2 (). e72589e and d3c7194 have the same canary; their only idle-path difference is VT terminal construction/resize (). Agent waits in tokio::select! (daemon.rs:147, 387); shim uses poll(..., None) (session.rs:316). A controlled 1 Hz signal sleeper showed 4 schedstat slices over 3.1 s, while the 500 ms CPU-tick check is phase- and CPU-time-dependent. The current release process-tree 60 s sample is in progress.

Finding: scripts/verify-agent.py:26-32 measures only the spawned infraternal agent's RSS and utime/stime equality across 500 ms, before terminal.open. crates/infraternal/tests/agent_daemon.rs:8-15 is its fast-gate caller. It does not count scheduled wakeups and covers neither shim nor server; currently initializes auth then exits 2 (). e72589e and d3c7194 have the same canary; their only idle-path difference is VT terminal construction/resize (). Agent waits in tokio::select! (daemon.rs:147, 387); shim uses poll(..., None) (session.rs:316). A controlled 1 Hz signal sleeper showed 4 schedstat slices over 3.1 s, while the 500 ms CPU-tick check is phase- and CPU-time-dependent. The current release process-tree 60 s sample is in progress.

kayg commented

2026-10-06 16:00:45 +00:00

Author

Owner

Correction to the previous finding comment: crates/infraternal/src/main.rs:50-76 shows that the server command initializes auth and exits with status 2. crates/infraternal-shim/src/session.rs is the only idle-path file changed between e72589e and d3c7194, and that diff only adds error propagation to VT construction and resize.

Correction to the previous finding comment: crates/infraternal/src/main.rs:50-76 shows that the server command initializes auth and exits with status 2. crates/infraternal-shim/src/session.rs is the only idle-path file changed between e72589e and d3c7194, and that diff only adds error propagation to VT construction and resize.

kayg commented

2026-10-06 16:03:32 +00:00

Author

Owner

Measurement before canary correction: release x86_64-unknown-linux-musl binary from 6c64ede, idle shell prompt held open, 60.000 s on an 8-logical-CPU i9-9900K; load average 4.02 at start and 3.33 at end. Per-thread schedstat timeslices and /proc sched context switches were unchanged: infraternal agent PID 1697493 = 0 slices (0.000 scheduled runs/min), session shim PID 1697496 = 0 (0.000/min), zsh PID 1697504 = 0 (0.000/min). The monitor sent no requests during the window. No real idle wakeup was observed. Server is not measurable as an idle process: its command exits 2 after printing not implemented.

Measurement before canary correction: release x86_64-unknown-linux-musl binary from 6c64ede, idle shell prompt held open, 60.000 s on an 8-logical-CPU i9-9900K; load average 4.02 at start and 3.33 at end. Per-thread schedstat timeslices and /proc sched context switches were unchanged: infraternal agent PID 1697493 = 0 slices (0.000 scheduled runs/min), session shim PID 1697496 = 0 (0.000/min), zsh PID 1697504 = 0 (0.000/min). The monitor sent no requests during the window. No real idle wakeup was observed. Server is not measurable as an idle process: its command exits 2 after printing not implemented.

kayg commented

2026-10-06 16:10:02 +00:00

Author

Owner

Regression evidence: scripts/test-idle-canary.py injects a real SIGALRM process with a 1.5 s first delay and 1.0 s period. The historical 500 ms CPU-tick observation stays unchanged, but the new procfs observer counts 2 schedstat slices in 2.501 s. The test is registered in crates/infraternal/tests/idle_canary.rs for the fast Rust test gate.

kayg commented

2026-10-06 16:12:28 +00:00

Author

Owner

Historical process measurements: exact e72589e static release binary, 60.000 s on the same 8-CPU i9-9900K, load average 2.31 to 2.22: agent 0 schedstat slices, shim 0, zsh 0; each 0.000 scheduled runs/min. The read-only VT worktree had advanced beyond the requested d3c7194; its existing static binary was built at 32f6ebe, before later test-only commits. Over 60.000 s on the same machine, load average 0.98 to 1.11: agent 0, shim 0, zsh 0 slices, each 0.000/min. Both measurements used an open idle shell prompt and no traffic. These observations support a broken/underspecified canary rather than an idle runtime wakeup. The server remains N/A because its subcommand exits during startup.

Historical process measurements: exact e72589e static release binary, 60.000 s on the same 8-CPU i9-9900K, load average 2.31 to 2.22: agent 0 schedstat slices, shim 0, zsh 0; each 0.000 scheduled runs/min. The read-only VT worktree had advanced beyond the requested d3c7194; its existing static binary was built at 32f6ebe, before later test-only commits. Over 60.000 s on the same machine, load average 0.98 to 1.11: agent 0, shim 0, zsh 0 slices, each 0.000/min. Both measurements used an open idle shell prompt and no traffic. These observations support a broken/underspecified canary rather than an idle runtime wakeup. The server remains N/A because its subcommand exits during startup.

kayg commented

2026-10-06 16:14:00 +00:00

Author

Owner

Canary correction implemented in the worktree at 6c64ede: scripts/idle_canary.py counts per-thread schedstat schedule-ins over 2.2 s and reads procfs without contacting the target; verify-agent now checks the idle agent before opening a shell and checks both agent and shim at an idle prompt. The fast-gate Rust test injects a 1 Hz timer and verifies 2 slices in 2.500 s after the old 500 ms window reports unchanged CPU ticks. Focused Rust test passed; real static functional fixture reported agent 0/2.200 s, attached agent 0/2.201 s, shim 0/2.201 s, RSS 3640 KiB, and 24/24 exact restart replays across 4,710,396 bytes.

Canary correction implemented in the worktree at 6c64ede: scripts/idle_canary.py counts per-thread schedstat schedule-ins over 2.2 s and reads procfs without contacting the target; verify-agent now checks the idle agent before opening a shell and checks both agent and shim at an idle prompt. The fast-gate Rust test injects a 1 Hz timer and verifies 2 slices in 2.500 s after the old 500 ms window reports unchanged CPU ticks. Focused Rust test passed; real static functional fixture reported agent 0/2.200 s, attached agent 0/2.201 s, shim 0/2.201 s, RSS 3640 KiB, and 24/24 exact restart replays across 4,710,396 bytes.

kayg commented

2026-10-06 16:23:09 +00:00

Author

Owner

Post-change 60.000 s scheduler sample, after the canary source changes and full gate build: current t-agent x86_64 static binary at 6c64ede, load average 2.69 to 3.30, agent 0 slices (0.000/min), shim 0 (0.000/min), zsh 0 (0.000/min). The read-only VT binary sample from 32f6ebe in the same window recorded agent 0, shim 0, zsh 0 slices (all 0.000/min). Context-switch deltas were also zero. Runtime code is unchanged by this canary correction.

Post-change 60.000 s scheduler sample, after the canary source changes and full gate build: current t-agent x86_64 static binary at 6c64ede, load average 2.69 to 3.30, agent 0 slices (0.000/min), shim 0 (0.000/min), zsh 0 (0.000/min). The read-only VT binary sample from 32f6ebe in the same window recorded agent 0, shim 0, zsh 0 slices (all 0.000/min). Context-switch deltas were also zero. Runtime code is unchanged by this canary correction.

kayg commented

2026-10-06 16:25:35 +00:00

Author

Owner

Finished at head 48d7babc52 on t-agent.

TypeScript protocol summary (verbatim):
==> TypeScript protocol tests
bun test v1.4.2 (744846f84)
stream differential: 134 streams, 11439 schedules, zero divergences
7 pass
0 fail
34390 expect() calls
Ran 7 tests across 2 files. [1.80s]

kayg commented

2026-10-06 16:27:26 +00:00

Author

Owner

Scale arithmetic: DECISIONS 18 specifies one infraternal agent per host; sessions.rs:32 caps a host at 128 sessions. The measured per-unit rates yield 0/min x 1 agent per host = 0/min and 0/min x 128 idle shims = 0/min projected aggregate. This is multiplication of the measured zero per-unit result; a 128-session saturation run was not performed.

kayg referenced this issue from a commit

2026-10-10 09:20:33 +00:00

test(agent): count idle scheduler activations for #27

kayg referenced this issue from a commit

2026-10-10 09:20:34 +00:00

Merge t-agent: infraternal agent core, shim management and terminal relay (#20, #27)

kayg referenced this issue

2026-10-12 12:13:55 +00:00

Idle RSS over budget on dev #187

kayg referenced this issue

2026-10-12 12:14:07 +00:00

Idle RSS over budget on dev #187

kayg commented

2026-10-28 17:52:17 +00:00

Author

Owner

Closed by the orchestrator during the 2026-10-28 handover (#254), on the owner's request to close finished issues. Evidence: Idle canary fixed at 48d7bab (in dev and prod); zero idle wakeups measured, gate PASS. Reopen if anything is still missing.

Closed by the orchestrator during the 2026-10-28 handover (#254), on the owner's request to close finished issues. Evidence: Idle canary fixed at 48d7bab (in dev and prod); zero idle wakeups measured, gate PASS. Reopen if anything is still missing.

kayg closed this issue

2026-10-28 17:52:17 +00:00

~~kayg referenced this issue 2026-10-28 17:53:54 +00:00~~

Orchestrator handover 2026-10-28: moving to a new machine #254

kayg referenced this issue

2026-10-28 18:00:45 +00:00

Orchestrator handover 2026-10-28: moving to a new machine #254

Rows
Columns

Gate: idle canary fails on both baseline and vt-ghostty #27