kayg/infraternal

Watch

Fork 0

J155: cancelling_remote_dispatch_retires_the_borrowed_authenticated_channel hangs on dev #332

New issue

Closed

opened 2026-11-03 20:59:25 +00:00 by kayg · 4 comments

kayg commented

2026-11-03 20:59:25 +00:00

Owner

daemon::remote_tests::cancelling_remote_dispatch_retires_the_borrowed_authenticated_channel (crates/infraternal-agent) hangs on unmodified origin/dev, both alone and as its module (J152 on #281, comment 77190: dev 3/3 hung alone, 3/3 as module, compute01). In some test orders it takes down the remote_client run and the gate's test row.

J155 job: reproduce, find the root cause (deadlock / lost wakeup / cancelled future holding the channel / test bug / product cancellation bug), fix it where it is wrong, prove 30/30 alone and as module plus 5 interleaved full-lib orders, and say whether it explains the other agent_daemon failures (#324).

Plan (estimate ~2 h):

Branch remote-cancel-hang-j155 from origin/dev e8c7acd44, reproduce on compute01 through build-slot with a backtrace of the hung threads.
Root cause from the backtrace + code, fix, atomic commit + push.
30/30 loops alone and as module, 5 full-lib orders, before/after numbers here.
scripts/verify fast verbatim with head SHA.

`daemon::remote_tests::cancelling_remote_dispatch_retires_the_borrowed_authenticated_channel` (`crates/infraternal-agent`) hangs on unmodified origin/dev, both alone and as its module (J152 on #281, comment 77190: dev 3/3 hung alone, 3/3 as module, compute01). In some test orders it takes down the `remote_client` run and the gate's `test` row. J155 job: reproduce, find the root cause (deadlock / lost wakeup / cancelled future holding the channel / test bug / product cancellation bug), fix it where it is wrong, prove 30/30 alone and as module plus 5 interleaved full-lib orders, and say whether it explains the other `agent_daemon` failures (#324). Plan (estimate ~2 h): 1. Branch `remote-cancel-hang-j155` from origin/dev `e8c7acd44`, reproduce on compute01 through build-slot with a backtrace of the hung threads. 2. Root cause from the backtrace + code, fix, atomic commit + push. 3. 30/30 loops alone and as module, 5 full-lib orders, before/after numbers here. 4. `scripts/verify fast` verbatim with head SHA.

kayg commented

2026-11-03 21:28:44 +00:00

Author

Owner

J155 start + root cause. Branch remote-cancel-hang-j155, base origin/dev e8c7acd44. Work on compute01 (~/worktrees/j155, target ~/.cache/codex-scratch/j155/target).

It is not order- or proxy-dependent: it is the AF_UNIX path bound

Unmodified dev with the default state root: alone 30/30 pass, module 30/30 pass (60 s cap each). As root: 3/3 pass. Also J62b head ad2148fb9 alone: pass.
Full-lib head runs with a dumper armed: state-head1 root passed, state-headr1 root hung. gdb on the hung binary: only 2 threads, libtest main + the test's current-thread runtime parked in epoll_wait. So a pending future, not a lock deadlock.
With the test changed to also watch the daemon task, the same run reports:
dispatcher ended before its greeting: Ok(Err(Error { kind: InvalidInput, message: "path must be shorter than SUN_LEN" }))

Mechanism.

The fixture's state dir is <root>/daemon-<36-char uuid>, and the dispatcher binds <state>/plugins.sock. Linux allows 108 bytes including the NUL. A state root of 51 characters or more (no trailing slash) overflows.
run_authenticated then fails before it greets. The test holds a second Arc (spare) to the host channel, so the TLS socket never closes and the peer never sees EOF. The test waits on Receive::next(&control) forever, without ever looking at the daemon task.

This explains J152's split exactly: ~/.cache/codex-scratch/j152/state-head-N is 51 characters (hangs) and state-dev-N is 50 (fits). J152's "dev hangs alone 3/3" must have used a longer root. There's no log of those runs in j152-evidence/.

Product. No cancellation bug. Abort retires the borrowed channel as specified, and the SUN_LEN refusal is correct. (Side note, not changed: an error before RetireChannel exists, e.g. a failing Service::start, does not retire the borrowed channel. The caller still owns it then. Retiring wouldn't send EOF anyway.)

Fix (test-only, 2 commits):

77e397222 the test selects on greeting vs dispatcher result, so it fails with the real error and can't hang. No timeout added.
132464d02 the fixture dir becomes d-<8 hex> (the scheme daemon_readiness_tests uses), so state roots up to 80 bytes fit.

Proof runs (30/30, 5 orders, verify fast) follow.

**J155 start + root cause.** Branch `remote-cancel-hang-j155`, base origin/dev `e8c7acd44`. Work on compute01 (`~/worktrees/j155`, target `~/.cache/codex-scratch/j155/target`). ### It is not order- or proxy-dependent: it is the AF_UNIX path bound - Unmodified dev with the default state root: **alone 30/30 pass, module 30/30 pass** (60 s cap each). As root: 3/3 pass. Also J62b head `ad2148fb9` alone: pass. - Full-lib head runs with a dumper armed: `state-head1` root passed, `state-headr1` root **hung**. gdb on the hung binary: only 2 threads, libtest main + the test's current-thread runtime parked in `epoll_wait`. So a pending future, not a lock deadlock. - With the test changed to also watch the daemon task, the same run reports: `dispatcher ended before its greeting: Ok(Err(Error { kind: InvalidInput, message: "path must be shorter than SUN_LEN" }))` **Mechanism.** 1. The fixture's state dir is `<root>/daemon-<36-char uuid>`, and the dispatcher binds `<state>/plugins.sock`. Linux allows 108 bytes including the NUL. A state root of 51 characters or more (no trailing slash) overflows. 2. `run_authenticated` then fails before it greets. The test holds a second Arc (`spare`) to the host channel, so the TLS socket never closes and the peer never sees EOF. The test waits on `Receive::next(&control)` forever, without ever looking at the daemon task. This explains J152's split exactly: `~/.cache/codex-scratch/j152/state-head-N` is 51 characters (hangs) and `state-dev-N` is 50 (fits). J152's "dev hangs alone 3/3" must have used a longer root. There's no log of those runs in `j152-evidence/`. **Product.** No cancellation bug. Abort retires the borrowed channel as specified, and the SUN_LEN refusal is correct. (Side note, not changed: an error *before* `RetireChannel` exists, e.g. a failing `Service::start`, does not retire the borrowed channel. The caller still owns it then. Retiring wouldn't send EOF anyway.) **Fix (test-only, 2 commits):** - `77e397222` the test selects on greeting vs dispatcher result, so it fails with the real error and can't hang. No timeout added. - `132464d02` the fixture dir becomes `d-<8 hex>` (the scheme `daemon_readiness_tests` uses), so state roots up to 80 bytes fit. Proof runs (30/30, 5 orders, verify fast) follow.

kayg commented

2026-11-03 22:02:03 +00:00

Author

Owner

J155 proof (compute01, claude). Head f1ac848db (2 commits: 695e80893 test fails instead of hanging; f1ac848db fixture dir d-<8 hex>). The 695e80893 SHA replaces 77e397222, which I fixed up because it did not compile (temporary borrow); verify quick caught it. Features native-streams,mesh,plugin-host-facts. Caps: 20 s alone, 30 s module.

Before: unmodified dev `e8c7acd44`

state root	alone	module	full lib
default (`~/.cache/codex-scratch/remote-agents`)	30/30	30/30	—
51-character root (`…/j155/state-long12`)	0/5 (all rc 124, hung)	0/5 (all rc 124)	hung, gdb dump taken, killed at 98 s
J62b head `ad2148fb9`, `state-headr1` root (51 characters)	—	—	hung (`gdb-headr1.txt`)

After: `f1ac848db`

state root	alone	module
default	30/30	30/30
51 characters	30/30	30/30
78 characters	30/30	30/30

Full infraternal-agent --lib, default root (the gate's environment), 5 orders:

d1-threads1 rc=0 secs=90 test result: ok. 222 passed; 0 failed; 22 ignored; 0 measured; 0 filtered out; finished in 89.51s hung=0
d2-threads2 rc=0 secs=55 test result: ok. 222 passed; 0 failed; 22 ignored; 0 measured; 0 filtered out; finished in 55.03s hung=0
d3-threads4 rc=0 secs=35 test result: ok. 222 passed; 0 failed; 22 ignored; 0 measured; 0 filtered out; finished in 30.79s hung=0
d4-default  rc=0 secs=61 test result: ok. 222 passed; 0 failed; 22 ignored; 0 measured; 0 filtered out; finished in 58.55s hung=0
d5-shuffle  rc=0 secs=25 test result: ok. 222 passed; 0 failed; 22 ignored; 0 measured; 0 filtered out; finished in 24.92s hung=0   (RUSTC_BOOTSTRAP=1 -Z unstable-options --shuffle-seed 332)

The same 5 orders at the 51-character root: no hang in any order. 13 other tests fail fast with the same path must be shorter than SUN_LEN / socket.as_os_str().len() <= 107 bound (liveness_tests, readiness*, replay_fence, census_retry, remote_primed, native_credit). They overflow the same way, but only this test turned the overflow into a hang. I left them alone: the gate does not set a long root, and a general test_state_root() fallback (like scripts/agent_fixture.py's) is a wider change. That is the orchestrator's call.

Runner note: tests that default to crate-relative ../../target fail with NotFound when CARGO_TARGET_DIR is outside the worktree, unless <worktree>/target exists. That accounted for the 14–17 "environment" failures in J152's and my early runs.

Does this explain today's other `agent_daemon` failures (#324)?

No. The gate's test row sets no INFRATERNAL_TEST_STATE_ROOT. Its default root gives a 104-byte socket path, which fits. So this hang cannot fire in a plain verify fast as claude or root. It only fires in harnesses with a state root of 51 characters or more (J152's). resource_unavailable and "terminal unavailable" are a different cause. I didn't chase them.

verify fast follows.

**J155 proof (compute01, claude).** Head `f1ac848db` (2 commits: `695e80893` test fails instead of hanging; `f1ac848db` fixture dir `d-<8 hex>`). The `695e80893` SHA replaces `77e397222`, which I fixed up because it did not compile (temporary borrow); `verify quick` caught it. Features `native-streams,mesh,plugin-host-facts`. Caps: 20 s alone, 30 s module. ### Before: unmodified dev `e8c7acd44` | state root | alone | module | full lib | |---|---|---|---| | default (`~/.cache/codex-scratch/remote-agents`) | 30/30 | 30/30 | — | | 51-character root (`…/j155/state-long12`) | **0/5 (all rc 124, hung)** | **0/5 (all rc 124)** | **hung**, gdb dump taken, killed at 98 s | | J62b head `ad2148fb9`, `state-headr1` root (51 characters) | — | — | **hung** (`gdb-headr1.txt`) | ### After: `f1ac848db` | state root | alone | module | |---|---|---| | default | **30/30** | **30/30** | | 51 characters | **30/30** | **30/30** | | 78 characters | **30/30** | **30/30** | Full `infraternal-agent --lib`, default root (the gate's environment), 5 orders: ```text d1-threads1 rc=0 secs=90 test result: ok. 222 passed; 0 failed; 22 ignored; 0 measured; 0 filtered out; finished in 89.51s hung=0 d2-threads2 rc=0 secs=55 test result: ok. 222 passed; 0 failed; 22 ignored; 0 measured; 0 filtered out; finished in 55.03s hung=0 d3-threads4 rc=0 secs=35 test result: ok. 222 passed; 0 failed; 22 ignored; 0 measured; 0 filtered out; finished in 30.79s hung=0 d4-default rc=0 secs=61 test result: ok. 222 passed; 0 failed; 22 ignored; 0 measured; 0 filtered out; finished in 58.55s hung=0 d5-shuffle rc=0 secs=25 test result: ok. 222 passed; 0 failed; 22 ignored; 0 measured; 0 filtered out; finished in 24.92s hung=0 (RUSTC_BOOTSTRAP=1 -Z unstable-options --shuffle-seed 332) ``` The same 5 orders at the 51-character root: no hang in any order. 13 *other* tests fail fast with the same `path must be shorter than SUN_LEN` / `socket.as_os_str().len() <= 107` bound (`liveness_tests`, `readiness*`, `replay_fence`, `census_retry`, `remote_primed`, `native_credit`). They overflow the same way, but only this test turned the overflow into a hang. I left them alone: the gate does not set a long root, and a general `test_state_root()` fallback (like `scripts/agent_fixture.py`'s) is a wider change. That is the orchestrator's call. Runner note: tests that default to crate-relative `../../target` fail with NotFound when `CARGO_TARGET_DIR` is outside the worktree, unless `<worktree>/target` exists. That accounted for the 14–17 "environment" failures in J152's and my early runs. ### Does this explain today's other `agent_daemon` failures (#324)? **No.** The gate's `test` row sets no `INFRATERNAL_TEST_STATE_ROOT`. Its default root gives a 104-byte socket path, which fits. So this hang cannot fire in a plain `verify fast` as `claude` or root. It only fires in harnesses with a state root of 51 characters or more (J152's). `resource_unavailable` and "terminal unavailable" are a different cause. I didn't chase them. `verify fast` follows.

kayg commented

2026-11-03 23:00:13 +00:00

Author

Owner

J155 finish. Branch remote-cancel-hang-j155, head f1ac848dbc39a15c1b87aea9453dc5ba6fb678fd (2 commits on origin/dev e8c7acd44). Not merged, not deployed.

scripts/verify fast on compute01 (nix develop, through build-slot), verbatim:

+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| fmt                       | PASS         |
| VT native fmt             | PASS         |
| CLI plugin compile-out    | PASS         |
| clippy                    | PASS         |
| SSH runtime test helper   | PASS         |
| real-binary fixture paths | PASS         |
| test                      | PASS         |
| SSH inode reuse fixture   | PASS         |
| shim process-per-user     | PASS         |
| cargo-deny                | PASS         |
| gate architecture         | PASS         |
| static builds             | PASS         |
+---------------------------+--------------+

==> Compiler metadata generator checks
PT_INTERP: absent; size: 22479152 bytes; baseline 22418256; delta +60896 bytes (report only, D243 amended 2026-11-01)
PT_INTERP: absent; size: 20495336 bytes; baseline 20453280; delta +42056 bytes (report only, D243 amended 2026-11-01)
x86_64-unknown-linux-musl: +102400 bytes accepted with delta; zero/negative deltas reported; missing/wrong/dynamic/PT_INTERP/inspection defects rejected PASS
aarch64-unknown-linux-musl: +102400 bytes accepted with delta; zero/negative deltas reported; missing/wrong/dynamic/PT_INTERP/inspection defects rejected PASS

 48 pass
 0 fail
 77248 expect() calls
Ran 48 tests across 12 files. [14.57s]
VERIFY_EXIT=0

Size: both commits change only #[cfg(test)] code, so the shipped binary does not change. The deltas shown are against the gate's stored baseline, which reflects drift on dev, not this branch.

Real environment: the hang is an unprivileged test-fixture path, and root runs of the test pass both before and after (3/3, /root/j155, removed). No test-server flow applies. Cleanup: scratch targets, worktrees, state dirs and /root/j155 are deleted. Logs and the gdb dump are kept in compute01:~/.cache/codex-scratch/j155/*.log|*.txt.

**J155 finish.** Branch `remote-cancel-hang-j155`, head `f1ac848dbc39a15c1b87aea9453dc5ba6fb678fd` (2 commits on origin/dev `e8c7acd44`). Not merged, not deployed. `scripts/verify fast` on compute01 (`nix develop`, through build-slot), verbatim: ```text +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | fmt | PASS | | VT native fmt | PASS | | CLI plugin compile-out | PASS | | clippy | PASS | | SSH runtime test helper | PASS | | real-binary fixture paths | PASS | | test | PASS | | SSH inode reuse fixture | PASS | | shim process-per-user | PASS | | cargo-deny | PASS | | gate architecture | PASS | | static builds | PASS | +---------------------------+--------------+ ==> Compiler metadata generator checks PT_INTERP: absent; size: 22479152 bytes; baseline 22418256; delta +60896 bytes (report only, D243 amended 2026-11-01) PT_INTERP: absent; size: 20495336 bytes; baseline 20453280; delta +42056 bytes (report only, D243 amended 2026-11-01) x86_64-unknown-linux-musl: +102400 bytes accepted with delta; zero/negative deltas reported; missing/wrong/dynamic/PT_INTERP/inspection defects rejected PASS aarch64-unknown-linux-musl: +102400 bytes accepted with delta; zero/negative deltas reported; missing/wrong/dynamic/PT_INTERP/inspection defects rejected PASS 48 pass 0 fail 77248 expect() calls Ran 48 tests across 12 files. [14.57s] VERIFY_EXIT=0 ``` Size: both commits change only `#[cfg(test)]` code, so the shipped binary does not change. The deltas shown are against the gate's stored baseline, which reflects drift on dev, not this branch. Real environment: the hang is an unprivileged test-fixture path, and root runs of the test pass both before and after (3/3, `/root/j155`, removed). No test-server flow applies. Cleanup: scratch targets, worktrees, state dirs and `/root/j155` are deleted. Logs and the gdb dump are kept in `compute01:~/.cache/codex-scratch/j155/*.log|*.txt`.

kayg commented

2026-11-04 17:04:49 +00:00

Author

Owner

J210 owner-authorized housekeeping: closing this completed scope because its work is merged on dev in b0eb5237c. Verified the merge commit is an ancestor of current origin/dev. The implementation reports and evidence remain in this issue.

J210 owner-authorized housekeeping: closing this completed scope because its work is merged on dev in [b0eb5237c](https://git.kayg.org/kayg/infraternal/commit/b0eb5237c417fc332a16717544649175e4e34921). Verified the merge commit is an ancestor of current origin/dev. The implementation reports and evidence remain in this issue.

kayg closed this issue

2026-11-04 17:04:50 +00:00

kayg referenced this issue

2026-11-04 17:04:55 +00:00

Test hang: cancelling_remote_dispatch_retires_the_borrowed_authenticated_channel hangs on dev (order-dependent) #333

kayg referenced this issue

2026-11-04 17:07:06 +00:00

J210: overnight housekeeping, 2026-11-04 #350

kayg referenced this issue

2026-11-07 10:43:32 +00:00

static binary +5.4 MB since 2026-11-02: attribution and trims #413